Search Results for author:
Found 52 papers, 21 papers with code
Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network
Inflectional morphology has since long been a useful testing ground for broader questions about generalisation in language and the viability of neural network models as cognitive models of language.
Can Transformers Process Recursive Nested Constructions, Like Humans?
A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions – a prototypical example of recursion in natural language.
MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
MultiLoKo can be used to study a variety of questions regarding the multilinguality of LLMs as well as meta-questions about multilingual benchmark creation.
Compute Optimal Scaling of Skills: Knowledge vs Reasoning
Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions.
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
The explosion of high-performing conversational language models (LMs) has spurred a shift from classic natural language processing (NLP) benchmarks to expensive, time-consuming and noisy human evaluations - yet the relationship between these two evaluation strategies remains hazy.
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
Our MLGym framework makes it easy to add new tasks, integrate and evaluate models or agents, generate synthetic data at scale, as well as develop new learning algorithms for training agents on AI research tasks.
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models
In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks.
Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
We propose a novel analysis method called ConTAM, and show with a large scale survey of existing and novel n-gram based contamination metrics across 13 benchmarks and 7 models from 2 different families that ConTAM can be used to better understand evaluation data contamination and its effects.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
Lastly, our research rediscovers the importance of using alignment metrics beyond simple percent alignment, showing that judges with high percent agreement can still assign vastly different scores.
Quantifying Variance in Evaluation Benchmarks
Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities.
Interpretability of Language Models via Task Spaces
To this end, we construct 'linguistic task spaces' -- representations of an LM's language conceptualisation -- that shed light on the connections LMs draw between language phenomena.
From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
The staggering pace with which the capabilities of large language models (LLMs) are increasing, as measured by a range of commonly used natural language understanding (NLU) benchmarks, raises many questions regarding what "understanding" means for a language model and how it compares to human understanding.
The ICL Consistency Test
Just like the previous generation of task-tuned models, large language models (LLMs) that are adapted to tasks via prompt-based methods like in-context-learning (ICL) perform well in some setups but not in others.
WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
We propose WorldSense, a benchmark designed to assess the extent to which LLMs are consistently able to sustain tacit world models, by testing how they draw simple inferences from descriptions of simple arrangements of entities.
Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation
When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others.
The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks
NLP models have progressed drastically in recent years, according to numerous datasets proposed to evaluate performance.
Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning
Finding the best way of adapting pre-trained language models to a task is a big challenge in current NLP.
Curriculum Learning with Adam: The Devil Is in the Wrong Details
We present a number of different case studies with different common hand-crafted and automated CL approaches to illustrate this phenomenon, and we find that none of them outperforms optimisation with only Adam with well-chosen hyperparameters.
Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses
At the staggering pace with which the capabilities of large language models (LLMs) are increasing, creating future-proof evaluation sets to assess their understanding becomes more and more challenging.
The Curious Case of Absolute Position Embeddings
Transformer language models encode the notion of word order using positional information.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Text Characterization Toolkit
In NLP, models are usually evaluated by reporting single-number performance scores on a number of readily available benchmarks, without much deeper analysis.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Towards Interactive Language Modeling
As a first contribution we present a road map in which we detail the steps that need to be taken towards interactive language modeling.
Sparse Interventions in Language Models with Differentiable Masking
Typically, interpretation methods i) do not guarantee that the model actually uses the encoded information, and ii) do not discover small subsets of neurons responsible for a considered phenomenon.
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions -- a prototypical example of recursion in natural language.
How BPE Affects Memorization in Transformers
In this work, we demonstrate that the size of the subword vocabulary learned by Byte-Pair Encoding (BPE) greatly affects both ability and tendency of standard Transformer models to memorize training data, even when we control for the number of learned parameters.
The paradox of the compositionality of natural language: a neural machine translation case study
Obtaining human-like performance in NLP is often argued to require compositional generalisation.
Language Models Use Monotonicity to Assess NPI Licensing
We investigate the semantic knowledge of language models (LMs), focusing on (1) whether these LMs create categories of linguistic environments based on their semantic monotonicity properties, and (2) whether these categories play a similar role in LMs as in human language understanding, using negative polarity item licensing as a case study.
Attention vs non-attention for a Shapley-based explanation method
In this work, we consider Contextual Decomposition (CD) -- a Shapley-based input feature attribution method that has been shown to work well for recurrent NLP models -- and we test the extent to which it is useful for models that contain attention operations.
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines.
Language Modelling as a Multi-Task Problem
In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability.
The Grammar of Emergent Languages
In this paper, we consider the syntactic properties of languages emerged in referential games, using unsupervised grammar induction (UGI) techniques originally designed to analyse natural language.
Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans
We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e. g., capturing the correct number agreement between subject and verb when they are separated by other phrases).
Internal and external pressures on language emergence: least effort, object constancy and frequency
In previous work, artificial agents were shown to achieve almost perfect accuracy in referential games where they have to communicate to identify images.
Co-evolution of language and agents in referential games
However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners.
Location Attention for Extrapolation to Longer Sequences
We hypothesize that models with a separate content- and location-based attention are more likely to extrapolate than those with common attention mechanisms.
Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment
Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena.
Compositionality decomposed: how do neural networks generalise?
Despite a multitude of empirical studies, little consensus exists on whether neural networks are able to generalise compositionally, a controversy that, in part, stems from a lack of agreement about what it means for a neural model to be compositional.
Assessing incrementality in sequence-to-sequence models
Since their inception, encoder-decoder models have successfully been applied to a wide array of problems in computational linguistics.
Transcoding compositionally: using attention to find more generalizable solutions
While sequence-to-sequence models have shown remarkable generalization power across several natural language tasks, their construct of solutions are argued to be less compositional than human-like generalization.
On the Realization of Compositionality in Neural Networks
We present a detailed comparison of two types of sequence to sequence models trained to conduct a compositional task.
The emergence of number and syntax units in LSTM language models
Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure.
Formal models of Structure Building in Music, Language and Animal Songs
Human language, music and a variety of animal vocalisations constitute ways of sonic communication that exhibit remarkable structural complexity.
The Fast and the Flexible: training neural networks to learn to follow instructions from small data
Learning to follow human instructions is a long-pursued goal in artificial intelligence.
Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items
In this paper, we attempt to link the inner workings of a neural language model to linguistic theory, focusing on a complex phenomenon well discussed in formal linguis- tics: (negative) polarity items.
Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue
We investigate how encoder-decoder models trained on a synthetic dataset of task-oriented dialogues process disfluencies, such as hesitations and self-corrections.
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
We show that `diagnostic classifiers', trained to predict number from the internal states of a language model, provide a detailed understanding of how, when, and where this information is represented.
Learning compositionally through attentive guidance
While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure.
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure
To develop an understanding of what the recurrent network encodes, visualisation techniques alone do not suffice.
POS-tagging of Historical Dutch
We present a study of the adequacy of current methods that are used for POS-tagging historical Dutch texts, as well as an exploration of the influence of employing different techniques to improve upon the current practice.
