Search Results for author:
Found 58 papers, 32 papers with code
High probability or low information? The probability–quality paradox in language generation
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality.
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages
This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features.
A surprisal–duration trade-off across and within the world’s languages
We thus conclude that there is strong evidence of a surprisal–duration trade-off in operation, both across and within the world’s languages.
Tokenisation is NP-Complete
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most $\delta$ symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).
Towards a Similarity-adjusted Surprisal Theory
Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified as surprisal.
Local and Global Decoding in Text Generation
Traditional methods, such as top-$k$ and top-$\pi$, apply local normalisation to the model's output distribution, which can distort it.
Investigating Critical Period Effects in Language Acquisition through Neural Language Models
We vary the age of exposure by training LMs on language pairs in various experimental conditions, and find that LMs, which lack any direct analog to innate maturational stages, do not show CP effects when the age of exposure of L2 is delayed.
How to Compute the Probability of a Word
Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics research.
Causal Estimation of Memorisation Profiles
Understanding memorisation in language models has practical and societal implications, e. g., studying models' training dynamics or preventing copyright infringements.
The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments
In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages.
On the Effect of (Near) Duplicate Subwords in Language Modelling
In this paper, we study the impact of near duplicate subwords on LM training efficiency.
Revisiting the Optimality of Word Lengths
Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio.
Quantifying the redundancy between prosody and text
Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings.
An Exploration of Left-Corner Transformations
The left-corner transformation (Rosenkrantz and Lewis, 1970) is used to remove left recursion from context-free grammars, which is an important step towards making the grammar parsable top-down with simple techniques.
On the Efficacy of Sampling Adapters
While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution.
Testing the Predictions of Surprisal Theory in 11 Languages
We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically.
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B.
A Measure-Theoretic Characterization of Tight Language Models
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings.
A Natural Bias for Language Generation Models
After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens.
On the Effect of Anticipation on Reading Times
We assess the effect of anticipation on reading by comparing how well surprisal and contextual entropy predict reading times on four naturalistic reading datasets: two self-paced and two eye-tracking.
The Architectural Bottleneck Principle
This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
On the Intersection of Context-Free and Regular Languages
It shows, by a simple construction, that the intersection of a context-free language and a regular language is itself context-free.
Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
As reinforcement learning (RL) naturally fits this objective -- maximizing an user's reward per session -- it has become an emerging topic in recommender systems.
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
As we show, however, this is not a tight approximation -- in either theory or practice.
Naturalistic Causal Probing for Morpho-Syntax
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing.
UniMorph 4.0: Universal Morphology
The project comprises two major thrusts: a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Probing for the Usage of Grammatical Number
We also find that BERT uses a separate encoding of grammatical number for nouns and verbs.
On the probability-quality paradox in language generation
Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings.
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Numerous analyses of reading time (RT) data have been implemented -- all in an effort to better understand the cognitive processes driving reading comprehension.
Locally Typical Sampling
Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.
A surprisal--duration trade-off across and within the world's languages
We thus conclude that there is strong evidence of a surprisal--duration trade-off in operation, both across and within the world's languages.
On Homophony and Rényi Entropy
Homophony's widespread presence in natural languages is a controversial topic.
Revisiting the Uniform Information Density Hypothesis
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal.
A Bayesian Framework for Information-Theoretic Probing
Pimentel et al. (2020) recently analysed probing from an information-theoretic perspective.
Modeling the Unigram Distribution
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus.
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge -- e. g. syntactic structure -- in contextual representations.
How (Non-)Optimal is the Lexicon?
Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon's optimality and to explore the relative costs of major constraints on natural codes.
Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models
Recent research has demonstrated that large pre-trained language models reflect societal biases expressed in natural language.
Finding Concept-specific Biases in Form--Meaning Associations
It is not a new idea that there are small, cross-linguistic associations between the forms and meanings of words.
Disambiguatory Signals are Stronger in Word-initial Positions
Psycholinguistic studies of human word processing and lexical access provide ample evidence of the preferred nature of word-initial versus word-final segments, e. g., in terms of attention paid by listeners (greater) or the likelihood of reduction by speakers (lower).
Pareto Probing: Trading Off Accuracy for Complexity
In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.
Speakers Fill Lexical Semantic Gaps with Context
For a language to be clear and efficiently encoded, we posit that the lexical ambiguity of a word type should correlate with how much information context provides about it, on average.
Metaphor Detection using Context and Concreteness
We report the results of our system on the Metaphor Detection Shared Task at the Second Workshop on Figurative Language Processing 2020.
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Systems were developed using data from 45 languages and just 5 language families, fine-tuned with data from an additional 45 languages and 10 language families (13 in total), and evaluated on all 90 languages.
A Corpus for Large-Scale Phonetic Typology
A major hurdle in data-driven research on typology is having sufficient data in many languages to draw meaningful conclusions.
Phonotactic Complexity and its Trade-offs
We present methods for calculating a measure of phonotactic complexity---bits per phoneme---that permits a straightforward cross-linguistic comparison.
A Tale of a Probe and a Parser
One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations.
Predicting Declension Class from Form and Meaning
We find for two Indo-European languages (Czech and German) that form and meaning respectively share significant amounts of information with class (and contribute additional information above and beyond gender).
Assessing the Reliability of Visual Explanations of Deep Models with Adversarial Perturbations
Thus, in this work we propose an objective measure to evaluate the reliability of explanations of deep models.
Information-Theoretic Probing for Linguistic Structure
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language.
Rethinking Phonotactic Complexity
In this work, we propose the use of phone-level language models to estimate phonotactic complexity{---}measured in bits per phoneme{---}which makes cross-linguistic comparison straightforward.
Meaning to Form: Measuring Systematicity as Information
A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade?
UaiNets: From Unsupervised to Active Deep Anomaly Detection
This work presents a method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection.
Deep Active Learning for Anomaly Detection
Anomalies are intuitively easy for human experts to understand, but they are hard to define mathematically.
Fast Node Embeddings: Learning Ego-Centric Representations
Representation learning is one of the foundations of Deep Learning and allowed important improvements on several Machine Learning tasks, such as Neural Machine Translation, Question Answering and Speech Recognition.
Detection of bimanual gestures everywhere: why it matters, what we need and what is missing
Bimanual gestures are of the utmost importance for the study of motor coordination in humans and in everyday activities.
