Search Results for author:
Found 49 papers, 24 papers with code
It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
Recent works have demonstrated that multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages.
On Linear Representations and Pretraining Data Frequency in Language Models
Pretraining data has a direct impact on the behaviors and quality of language models (LMs), but we only understand the most basic principles of this relationship.
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time.
Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases
The increased adoption of Large Language Models (LLMs) and their potential to shape public opinion have sparked interest in assessing these models' political leanings.
GRADE: Quantifying Sample Diversity in Text-to-Image Models
GRADE leverages the world knowledge embedded in large language models and visual question-answering systems to identify relevant concept-specific axes of diversity (e. g., ``shape'' and ``color'' for the concept ``cookie'').
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
We analyze features from the routing model to identify characteristics of instances that can benefit from human feedback, e. g., prompts with a moderate safety concern or moderate intent complexity.
How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold
We seek to determine the point at which a model was trained on enough instances to imitate a concept -- the imitation threshold.
Data Contamination Report from the 2024 CONDA Shared Task
The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models.
Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
Furthermore, while model performance improves across all tasks as LLM size increases, only factual question answering shows an increase in memorization, whereas machine translation and reasoning tasks exhibit greater generalization, producing more novel outputs.
Detection and Measurement of Syntactic Templates in Generated Text
Recent work on evaluating the diversity of text generated by LLMs has focused on word-level features.
Evaluating $n$-Gram Novelty of Language Models Using Rusty-DAWG
In this work, we investigate the extent to which modern LMs generate $n$-grams from their training data, evaluating both (i) the probability LMs assign to complete training $n$-grams and (ii) $n$-novelty, the proportion of $n$-grams generated by an LM that did not appear in the training data (for arbitrarily large $n$).
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation
Most works on gender bias focus on intrinsic bias -- removing traces of information about a protected group from the model's internal representation.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Calibrating Large Language Models with Sample Consistency
Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations.
Paloma: A Benchmark for Evaluating Language Model Fit
Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e. g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance.
What's In My Big Data?
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
The Bias Amplification Paradox in Text-to-Image Generation
Bias amplification is a phenomenon in which models exacerbate biases or stereotypes present in the training data.
Estimating the Causal Effect of Early ArXiving on Paper Acceptance
However, since quality is a challenging construct to estimate, we use the negative outcome control method, using paper citation count as a control variable to debias the quality confounding effect.
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B.
At Your Fingertips: Extracting Piano Fingering Instructions from Videos
Piano fingering -- knowing which finger to use to play each note in a musical piece, is a hard and important skill to master when learning to play the piano.
Lexical Generalization Improves with Larger Models and Longer Training
While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap.
CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm
We first align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions
Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.
Text-based NP Enrichment
Understanding the relations between entities denoted by NPs in a text is a critical part of human-like natural language understanding.
Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes
We explore Few-Shot Learning (FSL) for Relation Classification (RC).
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for evaluation.
Ranked #24 on
Coreference Resolution
on Winograd Schema Challenge
Contrastive Explanations for Model Interpretability
Our method is based on projecting model representation to a latent space that captures only the features that are useful (to the model) to differentiate two potential decisions.
Measuring and Improving Consistency in Pretrained Language Models
In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge?
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning.
It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
Recent works have demonstrated that multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages.
The Extraordinary Failure of Complement Coercion Crowdsourcing
In this work, we follow known methodologies of collecting labeled data for the complement coercion phenomenon.
Do Language Embeddings Capture Scales?
Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense, and factual knowledge.
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations
Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and syntactic tasks.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
In this work, we point out the inability to infer behavioral conclusions from probing results and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded.
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
The ability to control for the kinds of information encoded in neural representation has a variety of use cases, especially in light of the challenge of interpreting these models.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
oLMpics -- On what Language Model Pre-training Captures
A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data.
Adversarial Removal of Demographic Attributes Revisited
Elazar and Goldberg (2018) showed that protected attributes can be extracted from the representations of a debiased neural network for mention detection at above-chance levels, by evaluating a diagnostic classifier on a held-out subsample of the data it was trained on.
At Your Fingertips: Automatic Piano Fingering Detection
Automatic Piano Fingering is a hard task which computers can learn using data.
How Large Are Lions? Inducing Distributions over Quantitative Attributes
Most current NLP systems have little knowledge about quantitative attributes of objects and events.
Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolution
We provide the first computational treatment of fused-heads constructions (FH), focusing on the numeric fused-heads (NFH).
Ranked #1 on
Missing Elements
on Numeric Fused-Head (dev)
Where's My Head? Definition, Data Set, and Models for Numeric Fused-Head Identification and Resolution
We provide the first computational treatment of fused-heads constructions (FHs), focusing on the numeric fused-heads (NFHs).
Adversarial Removal of Demographic Attributes from Text Data
Recent advances in Representation Learning and Adversarial Training seem to succeed in removing unwanted features from the learned representation.
Privacy and Fairness in Recommender Systems via Adversarial Training of User Representations
Latent factor models for recommender systems represent users and items as low dimensional vectors.
