Search Results for author:
Found 104 papers, 71 papers with code
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts.
AmazonQAC: A Large-Scale, Naturalistic Query Autocomplete Dataset
Query Autocomplete (QAC) is a critical feature in modern search engines, facilitating user interaction by predicting search queries based on input prefixes.
Bayesian scaling laws for in-context learning
In all cases, Bayesian scaling laws accurately predict the conditions under which ICL will cause the suppressed behavior to reemerge, which sheds light on the ineffectiveness of post-training at increasing LLM safety.
Retrieval Augmented Spelling Correction for E-Commerce Applications
The rapid introduction of new brand names into everyday language poses a unique challenge for e-commerce spelling correction services, which must distinguish genuine misspellings from novel brand names that use unconventional spelling.
Improving Pretraining Data Using Perplexity Correlations
Quality pretraining data is often seen as the key to high-performance language models.
Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations
In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction.
Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training.
Demystifying Verbatim Memorization in Large Language Models
Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications.
Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
Natural Language Processing (NLP) systems are increasingly taking the form of sophisticated modular pipelines, e. g., Retrieval Augmented Generation (RAG), where each module may involve a distinct Language Model (LM) and an associated prompt template.
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules.
Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels
We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels.
Updating CLIP to Prefer Descriptions Over Captions
This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction.
MoEUT: Mixture-of-Experts Universal Transformers
The resulting UT model, for the first time, slightly outperforms standard Transformers on language modeling tasks such as BLiMP and PIQA, while using significantly less compute and memory.
ReFT: Representation Finetuning for Language Models
We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency.
Mapping the Increasing Use of LLMs in Scientific Papers
To address this gap, we conduct the first systematic, large-scale analysis across 950, 965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time.
pyvene: A Library for Understanding and Improving PyTorch Models via Interventions
Interventions on model-internal states are fundamental operations in many areas of AI, including model editing, steering, robustness, and interpretability.
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations
Individual neurons participate in the representation of multiple high-level concepts.
CommVQA: Situating Visual Question Answering in Communicative Contexts
To evaluate how situating images within naturalistic contexts shapes visual questions, we introduce CommVQA, a VQA dataset consisting of images, image descriptions, real-world communicative scenarios where the image might appear (e. g., a travel website), and follow-up questions and answers conditioned on the scenario and description.
CausalGym: Benchmarking causal interpretability methods on linguistic tasks
Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e. g., surprisal comparisons).
A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments
We respond to the recent paper by Makelov et al. (2023), which reviews subspace interchange intervention methods like distributed alignment search (DAS; Geiger et al. 2023) and claims that these methods potentially cause "interpretability illusions".
In-Context Learning for Extreme Multi-Label Classification
Multi-label classification problems with thousands of classes are hard to solve with in-context learning alone, as language models (LMs) might lack prior knowledge about the precise classes or how to assign them, and it is generally infeasible to demonstrate every class in a prompt.
Mission: Impossible Language Models
Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn.
I am a Strange Dataset: Metalinguistic Tests for Language Models
We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all.
Building Efficient and Effective OpenQA Systems for Low-Resource Languages
Our results show that SQuAD-TR makes OpenQA feasible for Turkish, which we hope encourages researchers to build OpenQA systems in other low-resource languages.
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems.
Multi-teacher Distillation for Multilingual Spelling Correction
Accurate spelling correction is a critical step in modern search interfaces, especially in an era of mobile devices and speech-to-text interfaces.
Flexible Model Interpretability through Natural Language Model Editing
Model interpretability and model editing are crucial goals in the age of large language models.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate.
CAW-coref: Conjunction-Aware Word-level Coreference Resolution
State-of-the-art coreference resolutions systems depend on multiple LLM calls per document and are thus prohibitively expensive for many use cases (e. g., information extraction with large corpora).
Ranked #4 on
Coreference Resolution
on OntoNotes
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks.
ContextRef: Evaluating Referenceless Metrics For Image Description Generation
None of the methods is successful with ContextRef, but we show that careful fine-tuning yields substantial improvements.
Rigorously Assessing Natural Language Explanations of Neurons
Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging.
Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering
We also find that context effects are especially important when participants can't see the image.
GIO: Gradient Information Optimization for Training Dataset Selection
Our contribution is in showing that it can be made highly scalable through a simple relaxation of the objective and a highly efficient implementation.
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
For in-context learning, we test InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning.
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option.
BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
We introduce BioDEX, a large-scale resource for Biomedical adverse Drug Event Extraction, rooted in the historical output of drug safety reporting in the U. S. BioDEX consists of 65k abstracts and 19k full-text biomedical papers with 256k associated document-level safety reports created by medical experts.
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables.
ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation
We argue that this concern is realized for the COGS benchmark.
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations.
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Many information retrieval tasks require large labeled datasets for fine-tuning.
Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability
Causal abstraction provides a theoretical foundation for mechanistic interpretability, the field concerned with providing intelligible algorithms that are faithful simplifications of the known, but opaque low-level details of black box AI models.
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).
Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training
Language tasks involving character-level manipulations (e. g., spelling corrections, arithmetic operations, word games) are challenging for models operating on subword units.
Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature
Our task is to automatically identify contradictory claims about COVID-19 drug efficacy.
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks.
Systematicity in GPT-3's Interpretation of Novel English Noun Compounds
However, this evidence is consistent with GPT3 reasoning only about specific lexical items rather than the more abstract conceptual categories of Levin et al.'s theory.
Causal Proxy Models for Concept-Based Model Explanations
The core of our proposal is the Causal Proxy Model (CPM).
Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models
Probabilistic models of language understanding are valuable tools for investigating human language use.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP).
Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics
Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users.
PLAID: An Efficient Engine for Late Interaction Retrieval
PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7$\times$ on a GPU and 45$\times$ on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality.
Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning
It seems likely that these patterns are shaped by the environment a speaker is exposed to in complex ways.
Causal Distillation for Language Models
Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks.
Ranked #6 on
Zero-shot Text Search
on BEIR
Inducing Causal Structure for Interpretable Neural Networks
In IIT, we (1) align variables in a causal model (e. g., a deterministic program or Bayesian network) with representations in a neural model and (2) train the neural model to match the counterfactual behavior of the causal model on a base input when aligned representations in both models are set to be the value they would be for a source input.
Hindsight: Posterior-guided training of retrievers for improved open-ended generation
Many text generation systems benefit from using a retriever to retrieve passages from a textual knowledge corpus (e. g., Wikipedia) which are then provided as additional context to the generator.
ReaSCAN: Compositional Reasoning in Language Grounding
The ability to compositionally map language to referents, relations, and actions is an essential component of language understanding.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Causal Abstractions of Neural Networks
Structural analysis methods (e. g., probing and feature attribution) are increasingly important tools for neural network analysis.
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.
Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data.
Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP
Cryptic crosswords, the dominant crossword variety in the UK, are a promising target for advancing NLP systems that seek to process semantically complex, highly compositional language.
Concadia: Towards Image-Based Text Generation with a Purpose
Current deep learning models often achieve excellent results on benchmark image-to-text datasets but fail to generate texts that are useful in practice.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Multi-hop reasoning (i. e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge.
DynaSent: A Dynamic Benchmark for Sentiment Analysis
We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis.
Relevance-guided Supervision for OpenQA with ColBERT
In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.
Modeling Subjective Assessments of Guilt in Newspaper Crime Narratives
Crime reporting is a prevalent form of journalism with the power to shape public perceptions and social policies.
Relational reasoning and generalization using non-symbolic neural networks
In the two latter cases, our models perform tasks proposed in previous work to demarcate human-unique symbolic abilities.
Data and Representation for Turkish Natural Language Inference
In this paper, we offer a positive response for natural language inference (NLI) in Turkish.
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
We address whether neural models for Natural Language Inference (NLI) can learn the compositional interactions between lexical entailment and negation, using four methods: the behavioral evaluation methods of (1) challenge test sets and (2) systematic generalization tasks, and the structural evaluation methods of (3) probes and (4) interventions.
Pragmatic Issue-Sensitive Image Captioning
Image captioning systems have recently improved dramatically, but they still tend to produce captions that are insensitive to the communicative goals that captions should meet.
TalkDown: A Corpus for Condescension Detection in Context
Condescending language use is caustic; it can bring dialogues to an end and bifurcate communities.
Communication-based Evaluation for Natural Language Generation
Natural language generation (NLG) systems are commonly evaluated using n-gram overlap measures (e. g. BLEU, ROUGE).
Recursive Routing Networks: Learning to Compose Modules for Language Understanding
The model jointly optimizes the parameters of the functions and the meta-learner{'}s policy for routing inputs through those functions.
Modeling Drug-Disease Relations with Linguistic and Knowledge Graph Constraints
FDA drug labels are rich sources of information about drugs and drug-disease relations, but their complexity makes them challenging texts to analyze in isolation.
Effective Feature Representation for Clinical Text Concept Extraction
Crucial information about the practice of healthcare is recorded only in free-form text, which creates an enormous opportunity for high-impact NLP.
Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity.
An Incremental Iterated Response Model of Pragmatics
Recent Iterated Response (IR) models of pragmatics conceptualize language use as a recursive process in which agents reason about each other to increase communicative efficiency.
A case for deep learning in semantics
Pater's target article builds a persuasive case for establishing stronger ties between theoretical linguistics and connectionism (deep learning).
Representing Social Media Users for Sarcasm Detection
We explore two methods for representing authors in the context of textual sarcasm detection: a Bayesian approach that directly represents authors' propensities to be sarcastic, and a dense embedding approach that can learn interactions between the author and the text.
Pragmatically Informative Image Captioning with Character-Level Inference
We combine a neural image captioner with a Rational Speech Acts (RSA) model to make a system that is pragmatically informative: its objective is to produce captions that are not merely true but also distinguish their inputs from similar images.
Mittens: An Extension of GloVe for Learning Domain-Specialized Representations
We present a simple extension of the GloVe representation learning model that begins with general-purpose representations and updates them based on data from a specialized domain.
Generating Bilingual Pragmatic Color References
Contextual influences on language often exhibit substantial cross-lingual regularities; for example, we are more verbose in situations that require finer distinctions.
On the Effective Use of Pretraining for Natural Language Inference
Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters.
Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations
Knowledge graphs are a versatile framework to encode richly structured data relationships, but it can be challenging to combine these graphs with unstructured data.
Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework.
Learning to Generate Compositional Color Descriptions
The production of color language is essential for grounded language generation.
A Fast Unified Model for Parsing and Sentence Understanding
Tree-structured neural networks exploit valuable syntactic parse information as they interpret the meanings of sentences.
Ranked #86 on
Natural Language Inference
on SNLI
Learning in the Rational Speech Acts Model
The Rational Speech Acts (RSA) model treats language use as a recursive process in which probabilistic speaker and listener agents reason about each other's intentions to enrich the literal semantics of their language along broadly Gricean lines.
A large annotated corpus for learning natural language inference
Understanding entailment and contradiction is fundamental to understanding natural language, and inference about entailment and contradiction is a valuable testing ground for the development of semantic representations.
Ranked #91 on
Natural Language Inference
on SNLI
Tree-structured composition in neural networks without tree-structured architectures
We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data.
Text to 3D Scene Generation with Rich Lexical Grounding
The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics.
Learning Distributed Word Representations for Natural Logic Reasoning
Natural logic offers a powerful relational conception of meaning that is a natural counterpart to distributed semantic representations, which have proven valuable in a wide range of sophisticated language tasks.
Exploiting Social Network Structure for Person-to-Person Sentiment Analysis
Person-to-person evaluations are prevalent in all kinds of discourse and important for establishing reputations, building social bonds, and shaping public opinion.
Recursive Neural Networks Can Learn Logical Semantics
Tree-structured recursive neural networks (TreeRNNs) for sentence meaning have been successful for many applications, but it remains an open question whether the fixed-length representations that they learn can support tasks as demanding as logical deduction.
A Computational Approach to Politeness with Application to Social Factors
We propose a computational framework for identifying linguistic aspects of politeness.
