Search Results for author:
Found 21 papers, 15 papers with code
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities.
Deriving Activation Functions Using Integration
Our work proposes a novel approach to designing activation functions by focusing on their gradients and deriving the corresponding activation functions using integration.
Understanding and Minimising Outlier Features in Neural Network Training
They are well known to emerge during standard transformer training and have the undesirable effect of hindering quantisation in afflicted models.
The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments
In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages.
On the Effect of (Near) Duplicate Subwords in Language Modelling
In this paper, we study the impact of near duplicate subwords on LM training efficiency.
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
This leads to the notion of a `compute-optimal' model, i. e. a model that allocates a given level of compute during training optimally to maximize performance.
The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute
We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours.
Mindstorms in Natural Language-Based Societies of Mind
What should be the social structure of an NLSOM?
Large Language Model Programs
In recent years, large pre-trained language models (LLMs) have demonstrated the ability to follow instructions and perform novel tasks from a few examples.
Solving Quantitative Reasoning Problems with Language Models
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding.
Ranked #21 on
Math Word Problem Solving
on MATH
Block-Recurrent Transformers
It is merely a transformer layer: it uses self-attention and cross-attention to efficiently compute a recurrent function over a large set of state vectors and tokens.
A Modern Self-Referential Weight Matrix That Learns to Modify Itself
The weight matrix (WM) of a neural network (NN) is its program.
Improving Baselines in the Wild
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts.
Augmenting Classic Algorithms with Neural Components for Strong Generalisation on Ambiguous and High-Dimensional Data
We augment classic algorithms with learned components to adapt them to domains currently dominated by deep learning models.
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s.
Linear Transformers Are Secretly Fast Weight Programmers
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values).
Learning Associative Inference Using Fast Weight Memory
Humans can quickly associate stimuli to solve problems in novel contexts.
Ranked #1 on
Question Answering
on catbAbI LM-mode
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure.
Ranked #1 on
Question Answering
on Mathematics Dataset
Learning to Reason with Third Order Tensor Products
We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data.
Learning to Reason with Third-Order Tensor Products
We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data.
GATED FAST WEIGHTS FOR ASSOCIATIVE RETRIEVAL
We improve previous end-to-end differentiable neural networks (NNs) with fast weight memories.
