Search Results for author:
Found 30 papers, 24 papers with code
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
The costly self-attention layers in modern Transformers require memory and compute quadratic in sequence length.
Automating Continual Learning
General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments.
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions
Recent studies of the computational power of recurrent neural networks (RNNs) reveal a hierarchy of RNN architectures, given real-time and finite-precision assumptions.
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Unlike prior work that compares MoEs with dense baselines under the compute-equal condition, our evaluation condition is parameter-equal, which is crucial to properly evaluate LMs.
Exploring the Promise and Limits of Real-Time Recurrent Learning
To scale to such challenging tasks, we focus on certain well-known neural architectures with element-wise recurrence, allowing for tractable RTRL without approximation.
Mindstorms in Natural Language-Based Societies of Mind
What should be the social structure of an NLSOM?
Contrastive Training of Complex-Valued Autoencoders for Object Discovery
Current state-of-the-art object-centric models use slots and attention-based routing for binding.
Accelerating Neural Self-Improvement via Bootstrapping
Few-shot learning with sequence-processing neural networks (NNs) has recently attracted a new wave of attention in the context of large language models.
Topological Neural Discrete Representation Learning à la Kohonen
Unsupervised learning of discrete representations from continuous ones in neural networks (NNs) is the cornerstone of several applications today.
Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks
Short-term memory in standard, general-purpose, sequence-processing recurrent neural networks (RNNs) is stored as activations of nodes or "neurons."
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations
While the original CTL is used to test length generalization or productivity, CTL++ is designed to test systematicity of NNs, that is, their capability to generalize to unseen compositions of known functions.
Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules
Work on fast weight programmers has demonstrated the effectiveness of key/value outer product-based learning rules for sequentially generating a weight matrix (WM) of a neural net (NN) by another NN or itself.
Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed.
Unsupervised Learning of Temporal Abstractions with Slot-based Transformers
The discovery of reusable sub-routines simplifies decision-making and planning in complex reinforcement learning problems.
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the entire training experience.
A Modern Self-Referential Weight Matrix That Learns to Modify Itself
The weight matrix (WM) of a neural network (NN) is its program.
Improving Baselines in the Wild
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts.
Training and Generating Neural Networks in Compressed Weight Space
The inputs and/or outputs of some neural nets are weight matrices of other neural nets.
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization.
Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization.
Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization.
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS.
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s.
Linear Transformers Are Secretly Fast Weight Programmers
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values).
The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment
We present a complete training pipeline to build a state-of-the-art hybrid HMM-based ASR system on the 2nd release of the TED-LIUM corpus.
Language Modeling with Deep Transformers
We explore deep autoregressive Transformer models in language modeling for speech recognition.
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation
To the best knowledge of the authors, the results obtained when training on the full LibriSpeech training set, are the best published currently, both for the hybrid DNN/HMM and the attention-based systems.
Ranked #24 on
Speech Recognition
on LibriSpeech test-other
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model.
Ranked #42 on
Speech Recognition
on LibriSpeech test-clean
Language Modelling
Sequence-To-Sequence Speech Recognition
+1
Improved training of end-to-end attention models for speech recognition
Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition.
Ranked #43 on
Speech Recognition
on LibriSpeech test-clean
(using extra training data)
