Search Results for author:
Found 23 papers, 8 papers with code
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
We further find that SAS competes with other beneficial traits and capabilities during training, and that briefly suppressing SAS can improve model quality.
Latent State Models of Training Dynamics
We use the HMM representation to study phase transitions and identify latent "detour" states that slow down convergence.
Dynamic Masking Rate Schedules for MLM Pretraining
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%.
One Venue, Two Conferences: The Separation of Chinese and American Citation Networks
At NeurIPS, American and Chinese institutions cite papers from each other's regions substantially less than they cite endogamously.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Benchmarking Compositionality with Formal Languages
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability.
Linear Connectivity Reveals Generalization Strategies
It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained.
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge -- e. g. syntactic structure -- in contextual representations.
LSTMs Compose---and Learn---Bottom-Up
To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data.
LSTMs Compose (and Learn) Bottom-Up
To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data.
Pareto Probing: Trading Off Accuracy for Complexity
In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.
Word Interdependence Exposes How LSTMs Compose Representations
Recent work in NLP shows that LSTM language models capture compositional structure in language data.
How to Evaluate Word Representations of Informal Domain?
Diverse word representations have surged in most state-of-the-art natural language processing (NLP) applications.
Sparsity Emerges Naturally in Neural Language Models
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks.
Do LSTMs Learn Compositionally?
LSTM-based language models exhibit compositionality in their representations, but how this behavior emerges over the course of training has not been explored.
Language Models Learn POS First
A glut of recent research shows that language models capture linguistic structure.
Understanding Learning Dynamics Of Language Models with SVCCA
Research has shown that neural models implicitly encode linguistic features, but there has been no research showing \emph{how} these encodings arise as the models are trained.
DyNet: The Dynamic Neural Network Toolkit
In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives.
Evaluating Informal-Domain Word Representations With UrbanDictionary
Existing corpora for intrinsic evaluation are not targeted towards tasks in informal domains such as Twitter or news comment forums.
Understanding Objects in Detail with Fine-Grained Attributes
We show that the collected data can be used to study the relation between part detection and attribute prediction by diagnosing the performance of classifiers that pool information from different parts of an object.
A framework for (under)specifying dependency syntax without overloading annotators
We introduce a framework for lightweight dependency syntax annotation.
