Search Results for author:
Found 7 papers, 3 papers with code
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear.
The Role of Temporal Hierarchy in Spiking Neural Networks
The temporal dynamics such as time constants of the synapses and neurons and delays have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network and increase the accuracy of the SNNs in solving temporal tasks.
Two types of pyramidal cells and their role in temporal processing
Recent work has provided new insights into the temporal specialization of Intratelencephalic (IT) and Pyramidal tract neurons (PT).
Bio-Inspired, Task-Free Continual Learning through Activity Regularization
We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation.
Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process.
Credit Assignment in Neural Networks through Deep Feedback Control
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output.
Tailoring Artificial Neural Networks for Optimal Learning
As one of the most important paradigms of recurrent neural networks, the echo state network (ESN) has been applied to a wide range of fields, from robotics to medicine, finance, and language processing.
