Search Results for author:
Found 24 papers, 12 papers with code
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
We introduce methods for discovering and applying sparse feature circuits.
Label Semantic Aware Pre-training for Few-shot Text Classification
Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction.
Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children.
Cross-Linguistic Syntactic Evaluation of Word Prediction Models
On other constructions, agreement accuracy was generally higher in languages with richer morphology.
Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling
Neural topic models can augment or replace bag-of-words inputs with the learned representations of deep pre-trained transformer-based word prediction models.
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts.
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases
Accurate syntactic representations are essential for robust generalization in natural language.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not.
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates.
Decoding Methods for Neural Narrative Generation
Narrative generation is an open-ended NLP task in which a model generates a story given a prompt.
Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models
However, much of this analysis has focused on monolingual models, and analyses of multilingual models have employed correlational methods that are confounded by the choice of probing tasks.
Modeling Color Terminology Across Thousands of Languages
There is an extensive history of scholarship into what constitutes a "basic" color term, as well as a broadly attested acquisition sequence of basic color terms across many languages, as articulated in the seminal work of Berlin and Kay (1969).
Quantity doesn't buy quality syntax with neural language models
We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained.
The Johns Hopkins University Bible Corpus: 1600+ Tongues for Typological Exploration
The corpus consists of over 4000 unique translations of the Christian Bible and counting.
An Analysis of Massively Multilingual Neural Machine Translation for Low-Resource Languages
We find that best practices in this domain are highly language-specific: adding more languages to a training set is often better, but too many harms performance{---}the best number depends on the source language.
Fine-grained Morphosyntactic Analysis and Generation Tools for More Than One Thousand Languages
Exploiting the broad translation of the Bible into the world{'}s languages, we train and distribute morphosyntactic tools for approximately one thousand languages, vastly outstripping previous distributions of tools devoted to the processing of inflectional morphology.
Demographic Representation and Collective Storytelling in the Me Too Twitter Hashtag Activism Movement
The #MeToo movement on Twitter has drawn attention to the pervasive nature of sexual harassment and violence.
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
We present the results of the NLP Community Metasurvey.
Language model acceptability judgements are not always robust to context
In this paper, we investigate the stability of language models' performance on targeted syntactic evaluations as we vary properties of the input context: the length of the context, the types of syntactic phenomena it contains, and whether or not there are violations of grammaticality.
Inverse Scaling: When Bigger Isn't Better
Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e. g., due to flaws in the training objective and data.
Meta-training with Demonstration Retrieval for Efficient Few-shot Learning
Meta-training allows one to leverage smaller models for few-shot generalization in a domain-general and task-agnostic manner; however, these methods alone results in models that may not have sufficient parameterization or knowledge to adapt quickly to a large variety of tasks.
Function Vectors in Large Language Models
Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV).
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
