Search Results for author:
Found 242 papers, 119 papers with code
Conditional Poisson Stochastic Beams
Beam search is the default decoding strategy for many sequence generation tasks in NLP.
On the Machine Learning of Ethical Judgments from Natural Language
Ethics is one of the longest standing intellectual endeavors of humanity.
SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection
The 2022 SIGMORPHON–UniMorph shared task on large scale morphological inflection generation included a wide range of typologically diverse languages: 33 languages from 11 top-level language families: Arabic (Modern Standard), Assamese, Braj, Chukchi, Eastern Armenian, Evenki, Georgian, Gothic, Gujarati, Hebrew, Hungarian, Itelmen, Karelian, Kazakh, Ket, Khalkha Mongolian, Kholosi, Korean, Lamahalot, Low German, Ludic, Magahi, Middle Low German, Old English, Old High German, Old Norse, Polish, Pomak, Slovak, Turkish, Upper Sorbian, Veps, and Xibe.
High probability or low information? The probability–quality paradox in language generation
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality.
A surprisal–duration trade-off across and within the world’s languages
We thus conclude that there is strong evidence of a surprisal–duration trade-off in operation, both across and within the world’s languages.
Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions
Of particular interest, languages on the same branch of our phylogenetic tree are notably similar, whereas languages from separate branches are no more similar than chance.
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages
This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features.
The SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes
This study describes the structure and the results of the SIGTYP 2022 shared task on the prediction of cognate reflexes from multilingual wordlists.
Efficient Sampling of Dependency Structure
In this paper, we adapt two spanning tree sampling algorithms to faithfully sample dependency trees from a graph subject to the root constraint.
Better Estimation of the KL Divergence Between Language Models
In this paper, we introduce a Rao--Blackwellized estimator that is also unbiased and provably has variance less than or equal to that of the standard Monte Carlo estimator.
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models.
Unique Hard Attention: A Tale of Two Sides
We show that this no longer holds with only leftmost-hard attention -- in that case, they correspond to a \emph{strictly weaker} fragment of LTL.
BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
We also call for papers outside the competition in any relevant areas.
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
No submissions outperformed the baselines in the multimodal track.
From Language Models over Tokens to Language Models over Characters
Modern language models are internally -- and mathematically -- distributions over token strings rather than \emph{character} strings, posing numerous challenges for programmers building user applications on top of them.
Likelihood as a Performance Gauge for Retrieval-Augmented Generation
Based on these findings, we propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
Controllable Context Sensitivity and the Knob Behind It
In this paper, we search for a knob which controls this sensitivity, determining whether language models answer from the context or their prior knowledge.
Training Neural Networks as Recognizers of Formal Languages
We provide results on a variety of languages across the Chomsky hierarchy for three neural architectures: a simple RNN, an LSTM, and a causally-masked transformer.
Gumbel Counterfactual Generation From Language Models
Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as a structural equation model using the Gumbel-max trick, which we called Gumbel counterfactual generation.
An $\mathbf{L^*}$ Algorithm for Deterministic Weighted Regular Languages
Extracting finite state automata (FSAs) from black-box models offers a powerful approach to gaining interpretable insights into complex model behaviors.
Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse
While these fluctuations can be viewed as theoretically uninteresting noise on top of a uniform target, another explanation is that UID is not the only functional pressure regulating information content in a language.
Efficiently Computing Susceptibility to Context in Language Models
However, exactly computing susceptibility is difficult and, thus, Du et al. (2024) falls back on a Monte Carlo approximation.
Reverse-Engineering the Reader
Numerous previous studies have sought to determine to what extent language models, pretrained on natural language text, can serve as useful models of human cognition.
Activation Scaling for Steering and Interpreting Language Models
Given the prompt "Rome is in", can we steer a language model to flip its prediction of an incorrect token "France" to a correct token "Italy" by only multiplying a few relevant activation vectors with scalars?
Can Transformers Learn $n$-gram Language Models?
Much theoretical work has described the ability of transformers to represent formal languages.
On the Proper Treatment of Tokenization in Psycholinguistics
The paper argues that token-level language models should be (approximately) marginalized into character-level language models before they are used in psycholinguistic studies to compute the surprisal of a region of interest; then, the marginalized character-level language model can be used to compute the surprisal of an arbitrary character substring, which we term a focal area, that the experimenter may wish to use as a predictor.
Generalized Measures of Anticipation and Responsivity in Online Language Processing
We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts.
On the Role of Context in Reading Time Prediction
Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency.
Investigating Critical Period Effects in Language Acquisition through Neural Language Models
We vary the age of exposure by training LMs on language pairs in various experimental conditions, and find that LMs, which lack any direct analog to innate maturational stages, do not show CP effects when the age of exposure of L2 is delayed.
The Foundations of Tokenization: Statistical and Computational Concerns
The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
Variational Best-of-N Alignment
To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on a controlled generation task suggest that while variational BoN is not as effective as BoN in aligning language models, it is close to BoN performance as vBoN appears more often on the Pareto frontier of reward and KL divergence compared to models trained with KL-constrained RL objective.
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning
We present several results on the representational capacity of recurrent and transformer LMs with CoT reasoning, showing that they can represent the same family of distributions over strings as probabilistic Turing machines.
A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
The relationship between the quality of a string, as judged by a human reader, and its probability, $p(\boldsymbol{y})$ under a language model undergirds the development of better language models.
Correlation Does Not Imply Compensation: Complexity and Irregularity in the Lexicon
It has been claimed that within a language, morphologically irregular words are more likely to be phonotactically simple and morphologically regular words are more likely to be phonotactically complex.
What Do Language Models Learn in Context? The Structured Task Hypothesis
Large language models (LLMs) exhibit an intriguing ability to learn a novel task from in-context examples presented in a demonstration, termed in-context learning (ICL).
What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages
We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers.
On Affine Homotopy between Language Encoders
In this spirit, we study the properties of \emph{affine} alignment of language encoders and its implications on extrinsic similarity.
Lower Bounds on the Expressivity of Recurrent Neural Language Models
The recent successes and spread of large neural language models (LMs) call for a thorough understanding of their computational ability.
Joint Lemmatization and Morphological Tagging with LEMMING
We present LEMMING, a modular log-linear model that jointly models lemmatization and tagging and supports the integration of arbitrary global features.
Transformers Can Represent $n$-gram Language Models
This provides a first step towards understanding the mechanisms that transformer LMs can use to represent probability distributions over strings.
Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
Low-resource named entity recognition is still an open problem in NLP.
Low Resource Named Entity Recognition
named-entity-recognition
+2
Labeled Morphological Segmentation with Semi-Markov Models
We present labeled morphological segmentation, an alternative view of morphological processing that unifies several tasks.
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
Context versus Prior Knowledge in Language Models
To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context.
Towards Explainability in Legal Outcome Prediction Models
Current legal outcome prediction models - a staple of legal NLP - do not explain their reasoning.
The Role of $n$-gram Smoothing in the Age of Neural Networks
For nearly three decades, language models derived from the $n$-gram assumption held the state of the art on the task.
On the Challenges and Opportunities in Generative AI
The field of deep generative modeling has grown rapidly and consistently over the years.
On Efficiently Representing Regular Languages as RNNs
However, a closer inspection of Hewitt et al.'s (2020) construction shows that it is not inherently limited to hierarchical structures.
A Practical Method for Generating String Counterfactuals
Interventions targeting the representation space of language models (LMs) have emerged as an effective means to influence model behavior.
Direct Preference Optimization with an Offset
DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response.
Representation Surgery: Theory and Practice of Affine Steering
In the case of neural language models, an encoding of the undesirable behavior is often present in the model's representations.
Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?
We construct tests for each one in order to understand whether current LLMs display the same cognitive biases as children in these steps.
Principled Gradient-based Markov Chain Monte Carlo for Text Generation
Recent papers have demonstrated the possibility of energy-based text generation by adapting gradient-based sampling algorithms, a paradigm of MCMC algorithms that promises fast convergence.
Revisiting the Optimality of Word Lengths
Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio.
The Ethics of Automating Legal Actors
The introduction of large public legal datasets has brought about a renaissance in legal NLP.
The Causal Influence of Grammatical Gender on Distributional Semantics
However, when we control for the meaning of the noun, the relationship between grammatical gender and adjective choice is near zero and insignificant.
Quantifying the redundancy between prosody and text
Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings.
An Exploration of Left-Corner Transformations
The left-corner transformation (Rosenkrantz and Lewis, 1970) is used to remove left recursion from context-free grammars, which is an important step towards making the grammar parsable top-down with simple techniques.
Formal Aspects of Language Modeling
Large language models have become one of the most commonly deployed NLP inventions.
Efficient Algorithms for Recognizing Weighted Tree-Adjoining Languages
From these, we also immediately obtain stringsum and allsum algorithms for TAG, LIG, PAA, and EPDA.
On the Representational Capacity of Recurrent Neural Language Models
We extend the Turing completeness result to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions.
Recurrent Neural Language Models as Probabilistic Finite-state Automata
These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.
An Analysis of On-the-fly Determinization of Finite-state Automata
In this paper we establish an abstraction of on-the-fly determinization of finite-state automata using transition monoids and demonstrate how it can be applied to bound the asymptotics.
A Geometric Notion of Causal Probing
The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace.
Testing the Predictions of Surprisal Theory in 11 Languages
We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families.
On the Efficacy of Sampling Adapters
While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution.
Efficient Semiring-Weighted Earley Parsing
This paper provides a reference description, in the form of a deduction system, of Earley's (1970) context-free parsing algorithm with various speed-ups.
Generalizing Backpropagation for Gradient-Based Interpretability
Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model's output with respect to its inputs.
A Formal Perspective on Byte-Pair Encoding
Via submodular functions, we prove that the iterative greedy version is a $\frac{1}{{\sigma(\boldsymbol{\mu}^\star)}}(1-e^{-{\sigma(\boldsymbol{\mu}^\star)}})$-approximation of an optimal merge sequence, where ${\sigma(\boldsymbol{\mu}^\star)}$ is the total backward curvature with respect to the optimal merge sequence $\boldsymbol{\mu}^\star$.
Hexatagging: Projective Dependency Parsing as Tagging
We introduce a novel dependency parser, the hexatagger, that constructs dependency trees by tagging the words in a sentence with elements from a finite set of possible tags.
LEACE: Perfect linear concept erasure in closed form
Concept erasure aims to remove specified features from an embedding.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically.
Convergence and Diversity in the Control Hierarchy
Furthermore, using an even stricter notion of equivalence called d-strong equivalence, we make precise the intuition that a CFG controlling a CFG is a TAG, a PDA controlling a PDA is an embedded PDA, and a PDA controlling a CFG is a LIG.
Structured Voronoi Sampling
In this paper, we take an important step toward building a principled approach for sampling from language models with gradient-based methods.
Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective
We show that these exhaustive comparisons can be avoided, and, moreover, the complexity of such tasks can be reduced to linear by casting the relation between tokens as a partial order over the string.
All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations
Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models.
RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text
In addition to producing AI-generated content (AIGC), we also demonstrate the possibility of using RecurrentGPT as an interactive fiction that directly interacts with consumers.
Efficient Prompting via Dynamic In-Context Learning
To achieve this, we train a meta controller that predicts the number of in-context examples suitable for the generalist model to make a good prediction based on the performance-efficiency trade-off for a specific input.
Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus
Several recent papers claim human parity at sentence-level Machine Translation (MT), especially in high-resource languages.
Controlled Text Generation with Natural Language Instructions
Large language models generate fluent texts and can follow natural language instructions to solve a wide range of tasks without task-specific training.
Discriminative Class Tokens for Text-to-Image Diffusion Models
This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs
The pathsum in ordinary acyclic WFSAs is efficiently computed by the backward algorithm in time $O(|E|)$, where $E$ is the set of transitions.
A Measure-Theoretic Characterization of Tight Language Models
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings.
The Ordered Matrix Dirichlet for State-Space Models
For discrete data, SSMs commonly do so through a state-to-action emission matrix and a state-to-state transition matrix.
On the Effect of Anticipation on Reading Times
We assess the effect of anticipation on reading by comparing how well surprisal and contextual entropy predict reading times on four naturalistic reading datasets: two self-paced and two eye-tracking.
Schrödinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition
Recent work has shown that despite their impressive capabilities, text-to-image diffusion models such as DALL-E 2 (Ramesh et al., 2022) can display strange behaviours when a prompt contains a word with multiple possible meanings, often generating images containing both senses of the word (Rassin et al., 2022).
On Parsing as Tagging
There have been many proposals to reduce constituency parsing to tagging in the literature.
The Architectural Bottleneck Principle
This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component.
Autoregressive Structured Prediction with Language Models
Recent years have seen a paradigm shift in NLP towards using pretrained language models ({PLM}) for a wide range of tasks.
Ranked #1 on
Relation Extraction
on CoNLL04
(NER Micro F1 metric)
A Bilingual Parallel Corpus with Discourse Annotations
The BWB corpus consists of Chinese novels translated by experts into English, and the annotated test set is designed to probe the ability of machine translation systems to model various discourse phenomena.
Investigating the Role of Centering Theory in the Context of Neural Coreference Resolution Systems
Our analysis further shows that contextualized embeddings contain much of the coherence information, which helps explain why CT can only provide little gains to modern neural coreference resolvers which make use of pretrained representations.
Mutual Information Alleviates Hallucinations in Abstractive Summarization
Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i. e., output content not supported by the source document.
Log-linear Guardedness and its Implications
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful.
Algorithms for Weighted Pushdown Automata
Weighted pushdown automata (WPDAs) are at the core of many natural language processing tasks, like syntax-based statistical machine translation and transition-based dependency parsing.
An Ordinal Latent Variable Model of Conflict Intensity
It is based only on the action category ("what") and disregards the subject ("who") and object ("to whom") of an event, as well as contextual information, like associated casualty count, that should contribute to the perception of an event's "intensity".
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Equivariant Transduction through Invariant Alignment
The ability to generalize compositionally is key to understanding the potentially infinite number of sentences that can be constructed in a human language from only a finite number of words.
On the Intersection of Context-Free and Regular Languages
It shows, by a simple construction, that the intersection of a context-free language and a regular language is itself context-free.
Visual Comparison of Language Model Adaptation
The simplicity of adapter training and composition comes along with new challenges, such as maintaining an overview of adapter properties and effectively comparing their produced embedding spaces.
Benchmarking Compositionality with Formal Languages
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability.
On the Role of Negative Precedent in Legal Outcome Prediction
In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction.
Probing via Prompting
We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
The SIGMORPHON 2022 Shared Task on Morpheme Segmentation
The SIGMORPHON 2022 shared task on morpheme segmentation challenged systems to decompose a word into a sequence of morphemes and covered most types of morphology: compounds, derivations, and inflections.
Ranked #8 on
Morpheme Segmentaiton
on UniMorph 4.0
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
As we show, however, this is not a tight approximation -- in either theory or practice.
Naturalistic Causal Probing for Morpho-Syntax
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing.
A Structured Span Selector
Many natural language processing tasks, e. g., coreference resolution and semantic role labeling, require selecting text spans and making decisions about them.
UniMorph 4.0: Universal Morphology
The project comprises two major thrusts: a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision.
Exact Paired-Permutation Testing for Structured Test Statistics
However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm.
Probing for the Usage of Grammatical Number
We also find that BERT uses a separate encoding of grammatical number for nouns and verbs.
Estimating the Entropy of Linguistic Distributions
Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language.
On the probability-quality paradox in language generation
Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings.
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Numerous analyses of reading time (RT) data have been implemented -- all in an effort to better understand the cognitive processes driving reading comprehension.
On Decoding Strategies for Neural Text Generators
For example, the nature of the diversity-quality trade-off in language generation is very task-specific; the length bias often attributed to beam search is not constant across tasks.
Locally Typical Sampling
Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.
Linear Adversarial Concept Erasure
Modern neural models trained on textual data rely on pre-trained representations that emerge without direct supervision.
Kernelized Concept Erasure
One prominent approach for the identification of concepts in neural representations is searching for a linear subspace whose erasure prevents the prediction of the concept from the representations.
A Latent-Variable Model for Intrinsic Probing
The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information.
A Word on Machine Ethics: A Response to Jiang et al. (2021)
Ethics is one of the longest standing intellectual endeavors of humanity.
Probing as Quantifying Inductive Bias
Such performance improvements have motivated researchers to quantify and understand the linguistic information encoded in these representations.
A surprisal--duration trade-off across and within the world's languages
We thus conclude that there is strong evidence of a surprisal--duration trade-off in operation, both across and within the world's languages.
On Homophony and Rényi Entropy
Homophony's widespread presence in natural languages is a controversial topic.
Classifying Dyads for Militarized Conflict Analysis
We do this by devising a set of textual and graph-based features which represent each of the causes.
Revisiting the Uniform Information Density Hypothesis
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal.
Conditional Poisson Stochastic Beam Search
In this work, we propose a new method for turning beam search into a stochastic process: Conditional Poisson stochastic beam search.
A Plug-and-Play Method for Controlled Text Generation
Large pre-trained language models have repeatedly shown their ability to produce fluent text.
Efficient Sampling of Dependency Structures
Colbourn (1996)'s sampling algorithm has a running time of $\mathcal{O}(N^3)$, which is often greater than the mean hitting time of a directed graph.
Searching for More Efficient Dynamic Programs
To this end, we describe a set of program transformations, a simple metric for assessing the efficiency of a transformed program, and a heuristic search procedure to improve this metric.
A Bayesian Framework for Information-Theoretic Probing
Pimentel et al. (2020) recently analysed probing from an information-theoretic perspective.
Differentiable Subset Pruning of Transformer Heads
Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer.
Towards Zero-shot Language Modeling
Motivated by this question, we aim at constructing an informative prior over neural weights, in order to adapt quickly to held-out languages in the task of character-level language modeling.
On Finding the K-best Non-projective Dependency Trees
Furthermore, we present a novel extension of the algorithm for decoding the K-best dependency trees of a graph which are subject to a root constraint.
SIGTYP 2021 Shared Task: Robust Spoken Language Identification
While language identification is a fundamental speech and language processing task, for many languages and language families it remains a challenging task.
Modeling the Unigram Distribution
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus.
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing
One method of doing so, which is frequently cited to support the claim that models like BERT encode syntax, is called probing; probes are small supervised models trained to extract linguistic information from another model's output.
Is Sparse Attention more Interpretable?
Sparse attention has been claimed to increase model interpretability under the assumption that it highlights influential inputs.
Examining the Inductive Bias of Neural Language Models with Artificial Languages
Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages.
On Finding the $K$-best Non-projective Dependency Trees
Furthermore, we present a novel extension of the algorithm for decoding the $K$-best dependency trees of a graph which are subject to a root constraint.
Higher-order Derivatives of Weighted Finite-state Machines
In the case of second-order derivatives, our scheme runs in the optimal $\mathcal{O}(A^2 N^4)$ time where $A$ is the alphabet size and $N$ is the number of states.
Language Model Evaluation Beyond Perplexity
As concrete examples, text generated under the nucleus sampling scheme adheres more closely to the type--token relationship of natural language than text produced using standard ancestral sampling; text from LSTMs reflects the natural language distributions over length, stopwords, and symbols surprisingly well.
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge -- e. g. syntactic structure -- in contextual representations.
A Cognitive Regularizer for Language Modeling
The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanation for certain syntactic, morphological, and prosodic choices.
How (Non-)Optimal is the Lexicon?
Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon's optimality and to explore the relative costs of major constraints on natural codes.
Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models
Recent research has demonstrated that large pre-trained language models reflect societal biases expressed in natural language.
Finding Concept-specific Biases in Form--Meaning Associations
It is not a new idea that there are small, cross-linguistic associations between the forms and meanings of words.
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation
Standard automatic metrics, e. g. BLEU, are not reliable for document-level MT evaluation.
Searching for Search Errors in Neural Morphological Inflection
Yet, on word-level tasks, exact inference of these models reveals the empty string is often the global optimum.
Differentiable Generative Phonology
As the main contribution of our work, we implement the phonological generative system as a neural model differentiable end-to-end, rather than as a set of rules or constraints.
Disambiguatory Signals are Stronger in Word-initial Positions
Psycholinguistic studies of human word processing and lexical access provide ample evidence of the preferred nature of word-initial versus word-final segments, e. g., in terms of attention paid by listeners (greater) or the likelihood of reduction by speakers (lower).
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing.
Morphologically Aware Word-Level Translation
We propose a novel morphologically aware probability model for bilingual lexicon induction, which jointly models lexeme translation and inflectional morphology in a structured way.
SIGTYP 2020 Shared Task: Prediction of Typological Features
Typological knowledge bases (KBs) such as WALS (Dryer and Haspelmath, 2013) contain information about linguistic properties of the world's languages.
Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model
Across languages, multiple consecutive adjectives modifying a noun (e. g. "the big red dog") follow certain unmarked ordering rules.
If beam search is the answer, what was the question?
This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question?
Intrinsic Probing through Dimension Selection
Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks.
Please Mind the Root: Decoding Arborescences for Dependency Parsing
The connection between dependency trees and spanning trees is exploited by the NLP community to train and to decode graph-based dependency parsers.
Pareto Probing: Trading Off Accuracy for Complexity
In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.
Speakers Fill Lexical Semantic Gaps with Context
For a language to be clear and efficiently encoded, we posit that the lexical ambiguity of a word type should correlate with how much information context provides about it, on average.
Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation
Their method takes pre-trained word representations as input and attempts to isolate a linear subspace that captures most of the gender bias in the representations.
Efficient Computation of Expectations under Spanning Tree Distributions
We propose unified algorithms for the important cases of first-order expectations and second-order expectations in edge-factored, non-projective spanning-tree models.
Best-First Beam Search
Decoding for many NLP tasks requires an effective heuristic algorithm for approximating exact search since the problem of searching the full output space is often intractable, or impractical in many settings.
Metaphor Detection using Context and Concreteness
We report the results of our system on the Metaphor Detection Shared Task at the Second Workshop on Figurative Language Processing 2020.
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Systems were developed using data from 45 languages and just 5 language families, fine-tuned with data from an additional 45 languages and 10 language families (13 in total), and evaluated on all 90 languages.
A Corpus for Large-Scale Phonetic Typology
A major hurdle in data-driven research on typology is having sufficient data in many languages to draw meaningful conclusions.
Applying the Transformer to Character-level Transduction
The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks.
Phonotactic Complexity and its Trade-offs
We present methods for calculating a measure of phonotactic complexity---bits per phoneme---that permits a straightforward cross-linguistic comparison.
It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
The performance of neural machine translation systems is commonly evaluated in terms of BLEU.
The Paradigm Discovery Problem
Our benchmark system first makes use of word embeddings and string similarity to cluster forms by cell and by paradigm.
A Tale of a Probe and a Parser
One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations.
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs
We also find that there are statistically significant relationships between the grammatical genders of inanimate nouns and the verbs that take those nouns as direct objects, as indirect objects, and as subjects.
Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing
Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i. e. over-confident) predictions, a common sign of overfitting.
Predicting Declension Class from Form and Meaning
We find for two Indo-European languages (Czech and German) that form and meaning respectively share significant amounts of information with class (and contribute additional information above and beyond gender).
UniMorph 3.0: Universal Morphology
The project comprises two major thrusts: a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Information-Theoretic Probing for Linguistic Structure
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language.
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages
In this work, we propose a Bayesian generative model for the space of neural parameters.
Morphological Segmentation Inside-Out
Morphological segmentation has traditionally been modeled with non-hierarchical models, which yield flat segmentations as output.
Weird Inflects but OK: Making Sense of Morphological Generation Errors
We conduct a manual error analysis of the CoNLL-SIGMORPHON Shared Task on Morphological Reinflection.
Quantifying the Semantic Core of Gender Systems
To that end, we use canonical correlation analysis to correlate the grammatical gender of inanimate nouns with an externally grounded definition of their lexical semantics.
The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection
The SIGMORPHON 2019 shared task on cross-lingual transfer and contextual analysis in morphology examined transfer learning of inflection between 100 language pairs, as well as contextual lemmatization and morphosyntactic description in 66 languages.
Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language.
Examining Gender Bias in Languages with Grammatical Gender
Recent studies have shown that word embeddings exhibit gender bias inherited from the training corpora.
It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution
An alternative approach is Counterfactual Data Augmentation (CDA), in which a corpus is duplicated and augmented to remove bias, e. g. by swapping all inherently-gendered words in the copy.
Rethinking Phonotactic Complexity
In this work, we propose the use of phone-level language models to estimate phonotactic complexity{---}measured in bits per phoneme{---}which makes cross-linguistic comparison straightforward.
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study
Embedding a clause inside another ({``}the girl [who likes cars [that run fast]] has arrived{''}) is a fundamental resource that has been argued to be a key driver of linguistic expressiveness.
Morphological Irregularity Correlates with Frequency
We present a study of morphological irregularity.
Uncovering Probabilistic Implications in Typological Knowledge Bases
The study of linguistic typology is rooted in the implications we find between linguistic features, such as the fact that languages with object-verb word ordering tend to have post-positions.
Meaning to Form: Measuring Systematicity as Information
A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade?
What Kind of Language Is Hard to Language-Model?
Trying to answer the question of what features difficult languages have in common, we try and fail to reproduce our earlier (Cotterell et al., 2018) observation about morphological complexity and instead reveal far simpler statistics of the data that seem to drive complexity in a much larger sample.
Unsupervised Discovery of Gendered Language through Latent-Variable Modeling
Studying the ways in which language is gendered has long been an area of interest in sociolinguistics.
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems.
Exact Hard Monotonic Attention for Character-Level Transduction
Our models achieve state-of-the-art performance on morphological inflection.
Contextualization of Morphological Inflection
Critical to natural language generation is the production of correctly inflected text.
Gender Bias in Contextualized Word Embeddings
In this paper, we quantify, analyze and mitigate gender bias exhibited in ELMo's contextualized word vectors.
Combining Sentiment Lexica with a Multi-View Variational Autoencoder
When assigning quantitative labels to a dataset, different methodologies may rely on different scales.
A Simple Joint Model for Improved Contextual Neural Lemmatization
English verbs have multiple forms.
A Probabilistic Generative Model of Linguistic Typology
In the principles-and-parameters framework, the structural features of languages depend on parameters that may be toggled on or off, with a single parameter often dictating the status of multiple features.
