Search Results for author:
Found 90 papers, 61 papers with code
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks.
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Relevance-guided Supervision for OpenQA with ColBERT
In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Multi-hop reasoning (i. e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge.
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks.
Ranked #6 on
Zero-shot Text Search
on BEIR
PLAID: An Efficient Engine for Late Interaction Retrieval
PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7$\times$ on a GPU and 45$\times$ on a CPU against vanilla ColBERTv2, while continuing to deliver state-of-the-art retrieval quality.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Many information retrieval tasks require large labeled datasets for fine-tuning.
pyvene: A Library for Understanding and Improving PyTorch Models via Interventions
Interventions on model-internal states are fundamental operations in many areas of AI, including model editing, steering, robustness, and interpretability.
ReFT: Representation Finetuning for Language Models
LoReFT is a drop-in replacement for existing PEFTs and learns interventions that are 10x-50x more parameter-efficient than prior state-of-the-art PEFTs.
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables.
In-Context Learning for Extreme Multi-Label Classification
Multi-label classification problems with thousands of classes are hard to solve with in-context learning alone, as language models (LMs) might lack prior knowledge about the precise classes or how to assign them, and it is generally infeasible to demonstrate every class in a prompt.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate.
Mittens: An Extension of GloVe for Learning Domain-Specialized Representations
We present a simple extension of the GloVe representation learning model that begins with general-purpose representations and updates them based on data from a specialized domain.
A Fast Unified Model for Parsing and Sentence Understanding
Tree-structured neural networks exploit valuable syntactic parse information as they interpret the meanings of sentences.
Ranked #86 on
Natural Language Inference
on SNLI
Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations
Knowledge graphs are a versatile framework to encode richly structured data relationships, but it can be challenging to combine these graphs with unstructured data.
DynaSent: A Dynamic Benchmark for Sentiment Analysis
We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis.
Tree-structured composition in neural networks without tree-structured architectures
We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data.
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option.
A large annotated corpus for learning natural language inference
Understanding entailment and contradiction is fundamental to understanding natural language, and inference about entailment and contradiction is a valuable testing ground for the development of semantic representations.
Ranked #91 on
Natural Language Inference
on SNLI
Data and Representation for Turkish Natural Language Inference
In this paper, we offer a positive response for natural language inference (NLI) in Turkish.
BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
We introduce BioDEX, a large-scale resource for Biomedical adverse Drug Event Extraction, rooted in the historical output of drug safety reporting in the U. S. BioDEX consists of 65k abstracts and 19k full-text biomedical papers with 256k associated document-level safety reports created by medical experts.
Learning to Generate Compositional Color Descriptions
The production of color language is essential for grounded language generation.
I am a Strange Dataset: Metalinguistic Tests for Language Models
We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all.
ReaSCAN: Compositional Reasoning in Language Grounding
The ability to compositionally map language to referents, relations, and actions is an essential component of language understanding.
Inducing Causal Structure for Interpretable Neural Networks
In IIT, we (1) align variables in a causal model (e. g., a deterministic program or Bayesian network) with representations in a neural model and (2) train the neural model to match the counterfactual behavior of the causal model on a base input when aligned representations in both models are set to be the value they would be for a source input.
Causal Distillation for Language Models
Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
CausalGym: Benchmarking causal interpretability methods on linguistic tasks
Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e. g., surprisal comparisons).
Concadia: Towards Image-Based Text Generation with a Purpose
Current deep learning models often achieve excellent results on benchmark image-to-text datasets but fail to generate texts that are useful in practice.
Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models
Probabilistic models of language understanding are valuable tools for investigating human language use.
Generating Bilingual Pragmatic Color References
Contextual influences on language often exhibit substantial cross-lingual regularities; for example, we are more verbose in situations that require finer distinctions.
Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework.
CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP).
Representing Social Media Users for Sarcasm Detection
We explore two methods for representing authors in the context of textual sarcasm detection: a Bayesian approach that directly represents authors' propensities to be sarcastic, and a dense embedding approach that can learn interactions between the author and the text.
Mission: Impossible Language Models
Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn.
TalkDown: A Corpus for Condescension Detection in Context
Condescending language use is caustic; it can bring dialogues to an end and bifurcate communities.
Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP
Cryptic crosswords, the dominant crossword variety in the UK, are a promising target for advancing NLP systems that seek to process semantically complex, highly compositional language.
Causal Proxy Models for Concept-Based Model Explanations
The core of our proposal is the Causal Proxy Model (CPM).
ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation
We argue that this concern is realized for the COGS benchmark.
CAW-coref: Conjunction-Aware Word-level Coreference Resolution
State-of-the-art coreference resolutions systems depend on multiple LLM calls per document and are thus prohibitively expensive for many use cases (e. g., information extraction with large corpora).
Ranked #3 on
Coreference Resolution
on OntoNotes
Causal Abstractions of Neural Networks
Structural analysis methods (e. g., probing and feature attribution) are increasingly important tools for neural network analysis.
Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics
Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users.
GIO: Gradient Information Optimization for Training Dataset Selection
It is often advantageous to train models on a subset of the available train examples, because the examples are of variable quality or because one would like to train with fewer examples, without sacrificing performance.
Building Efficient and Effective OpenQA Systems for Low-Resource Languages
We present SQuAD-TR, a machine translation of SQuAD2. 0, and we build our OpenQA system by adapting ColBERT-QA for Turkish.
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations
Individual neurons participate in the representation of multiple high-level concepts.
Communication-based Evaluation for Natural Language Generation
Natural language generation (NLG) systems are commonly evaluated using n-gram overlap measures (e. g. BLEU, ROUGE).
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
For in-context learning, we test InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning.
A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments
We respond to the recent paper by Makelov et al. (2023), which reviews subspace interchange intervention methods like distributed alignment search (DAS; Geiger et al. 2023) and claims that these methods potentially cause "interpretability illusions".
Relational reasoning and generalization using non-symbolic neural networks
In the two latter cases, our models perform tasks proposed in previous work to demarcate human-unique symbolic abilities.
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
We address whether neural models for Natural Language Inference (NLI) can learn the compositional interactions between lexical entailment and negation, using four methods: the behavioral evaluation methods of (1) challenge test sets and (2) systematic generalization tasks, and the structural evaluation methods of (3) probes and (4) interventions.
Pragmatic Issue-Sensitive Image Captioning
Image captioning systems have recently improved dramatically, but they still tend to produce captions that are insensitive to the communicative goals that captions should meet.
Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning
It seems likely that these patterns are shaped by the environment a speaker is exposed to in complex ways.
ContextRef: Evaluating Referenceless Metrics For Image Description Generation
None of the methods is successful with ContextRef, but we show that careful fine-tuning yields substantial improvements.
Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data.
Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering
We also find that context effects are especially important when participants can't see the image.
CommVQA: Situating Visual Question Answering in Communicative Contexts
Current visual question answering (VQA) models tend to be trained and evaluated on image-question pairs in isolation.
Pragmatically Informative Image Captioning with Character-Level Inference
We combine a neural image captioner with a Rational Speech Acts (RSA) model to make a system that is pragmatically informative: its objective is to produce captions that are not merely true but also distinguish their inputs from similar images.
On the Effective Use of Pretraining for Natural Language Inference
Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters.
Learning in the Rational Speech Acts Model
The Rational Speech Acts (RSA) model treats language use as a recursive process in which probabilistic speaker and listener agents reason about each other's intentions to enrich the literal semantics of their language along broadly Gricean lines.
Text to 3D Scene Generation with Rich Lexical Grounding
The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics.
Recursive Neural Networks Can Learn Logical Semantics
Tree-structured recursive neural networks (TreeRNNs) for sentence meaning have been successful for many applications, but it remains an open question whether the fixed-length representations that they learn can support tasks as demanding as logical deduction.
Learning Distributed Word Representations for Natural Logic Reasoning
Natural logic offers a powerful relational conception of meaning that is a natural counterpart to distributed semantic representations, which have proven valuable in a wide range of sophisticated language tasks.
Exploiting Social Network Structure for Person-to-Person Sentiment Analysis
Person-to-person evaluations are prevalent in all kinds of discourse and important for establishing reputations, building social bonds, and shaping public opinion.
A Computational Approach to Politeness with Application to Social Factors
We propose a computational framework for identifying linguistic aspects of politeness.
A case for deep learning in semantics
Pater's target article builds a persuasive case for establishing stronger ties between theoretical linguistics and connectionism (deep learning).
An Incremental Iterated Response Model of Pragmatics
Recent Iterated Response (IR) models of pragmatics conceptualize language use as a recursive process in which agents reason about each other to increase communicative efficiency.
Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity.
Effective Feature Representation for Clinical Text Concept Extraction
Crucial information about the practice of healthcare is recorded only in free-form text, which creates an enormous opportunity for high-impact NLP.
Modeling Drug-Disease Relations with Linguistic and Knowledge Graph Constraints
FDA drug labels are rich sources of information about drugs and drug-disease relations, but their complexity makes them challenging texts to analyze in isolation.
Recursive Routing Networks: Learning to Compose Modules for Language Understanding
The model jointly optimizes the parameters of the functions and the meta-learner{'}s policy for routing inputs through those functions.
Modeling Subjective Assessments of Guilt in Newspaper Crime Narratives
Crime reporting is a prevalent form of journalism with the power to shape public perceptions and social policies.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.
Hindsight: Posterior-guided training of retrievers for improved open-ended generation
Many text generation systems benefit from using a retriever to retrieve passages from a textual knowledge corpus (e. g., Wikipedia) which are then provided as additional context to the generator.
Systematicity in GPT-3's Interpretation of Novel English Noun Compounds
However, this evidence is consistent with GPT3 reasoning only about specific lexical items rather than the more abstract conceptual categories of Levin et al.'s theory.
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking
Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks.
Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training
Language tasks involving character-level manipulations (e. g., spelling corrections, arithmetic operations, word games) are challenging for models operating on subword units.
Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature
Our task is to automatically identify contradictory claims about COVID-19 drug efficacy.
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations.
Rigorously Assessing Natural Language Explanations of Neurons
Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging.
Flexible Model Interpretability through Natural Language Model Editing
Model interpretability and model editing are crucial goals in the age of large language models.
Multi-teacher Distillation for Multilingual Spelling Correction
Accurate spelling correction is a critical step in modern search interfaces, especially in an era of mobile devices and speech-to-text interfaces.
Mapping the Increasing Use of LLMs in Scientific Papers
To address this gap, we conduct the first systematic, large-scale analysis across 950, 965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time.
