Search Results for author:
Found 33 papers, 12 papers with code
Fast Forwarding Low-Rank Training
Parameter efficient finetuning methods like low-rank adaptation (LoRA) aim to reduce the computational costs of finetuning pretrained Language Models (LMs).
Benchmarks as Microscopes: A Call for Model Metrology
Modern language models (LMs) pose a new challenge in capability assessment.
ChatGPT Doesn't Trust Chargers Fans: Guardrail Sensitivity in Context
By generating user biographies that offer ideological and demographic information, we find a number of biases in guardrail sensitivity on GPT-3. 5.
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data.
Transcendence: Generative Models Can Outperform The Experts That Train Them
Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on.
Knowing Your Nonlinearities: Shapley Interactions Reveal the Underlying Structure of Data
Measuring nonlinear feature interaction is an established approach to understanding complex patterns of attribution in many models.
Towards out-of-distribution generalization in large-scale astronomical surveys: robust networks learn similar representations
The generalization of machine learning (ML) models to out-of-distribution (OOD) examples remains a key challenge in extracting information from upcoming astronomical surveys.
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models
Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs).
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
We propose Trust Region Aware Minimization (TRAM), a SAM algorithm fine-tuning for low parameter sharpness and smooth, informative representations preserving pre-trained structure.
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model.
Latent State Models of Training Dynamics
We use the HMM representation to study phase transitions and identify latent "detour" states that slow down convergence.
Dynamic Masking Rate Schedules for MLM Pretraining
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%.
One Venue, Two Conferences: The Separation of Chinese and American Citation Networks
At NeurIPS, American and Chinese institutions cite papers from each other's regions substantially less than they cite endogamously.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Benchmarking Compositionality with Formal Languages
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability.
Linear Connectivity Reveals Generalization Strategies
It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained.
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge -- e. g. syntactic structure -- in contextual representations.
LSTMs Compose---and Learn---Bottom-Up
To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data.
LSTMs Compose (and Learn) Bottom-Up
To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data.
Pareto Probing: Trading Off Accuracy for Complexity
In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.
Word Interdependence Exposes How LSTMs Compose Representations
Recent work in NLP shows that LSTM language models capture compositional structure in language data.
How to Evaluate Word Representations of Informal Domain?
Diverse word representations have surged in most state-of-the-art natural language processing (NLP) applications.
Sparsity Emerges Naturally in Neural Language Models
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks.
Do LSTMs Learn Compositionally?
LSTM-based language models exhibit compositionality in their representations, but how this behavior emerges over the course of training has not been explored.
Understanding Learning Dynamics Of Language Models with SVCCA
Research has shown that neural models implicitly encode linguistic features, but there has been no research showing \emph{how} these encodings arise as the models are trained.
Language Models Learn POS First
A glut of recent research shows that language models capture linguistic structure.
DyNet: The Dynamic Neural Network Toolkit
In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives.
Evaluating Informal-Domain Word Representations With UrbanDictionary
Existing corpora for intrinsic evaluation are not targeted towards tasks in informal domains such as Twitter or news comment forums.
Understanding Objects in Detail with Fine-Grained Attributes
We show that the collected data can be used to study the relation between part detection and attribute prediction by diagnosing the performance of classifiers that pool information from different parts of an object.
A framework for (under)specifying dependency syntax without overloading annotators
We introduce a framework for lightweight dependency syntax annotation.
