Search Results for author:
Found 84 papers, 69 papers with code
The Deployment of End-to-End Audio Language Models Should Take into Account the Principle of Least Privilege
We are at a turning point for language models that accept audio input.
Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving
On the miniF2F benchmark, it achieves a 57. 6% success rate (Pass@32), exceeding the previous best open-source model by 7. 6%.
LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
LongProc consists of six diverse procedural generation tasks, such as extracting structured information from HTML pages into a TSV format and executing complex search procedures to create travel plans.
Metadata Conditioning Accelerates Language Model Pre-training
The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these heterogeneous data sources is challenging.
Continual Memorization of Factoids in Large Language Models
REMIX prevents forgetting by mixing generic data sampled from pretraining corpora or even randomly generated word sequences during each stage, despite being unrelated to the memorized factoids in the first stage.
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences.
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
There have been many benchmarks for evaluating long-context language models (LCLMs), but developers often rely on synthetic tasks like needle-in-a-haystack (NIAH) or arbitrary subsets of tasks.
How to Train Long-Context Language Models (Effectively)
We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information.
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.
Representing Rule-based Chatbots with Transformers
Next, we train Transformers on a dataset of synthetically generated ELIZA conversations and investigate the mechanisms the models learn.
LitSearch: A Retrieval Benchmark for Scientific Literature Search
LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions manually written by authors about their recently published papers.
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
All models lag far behind human performance of 80. 5%, underscoring weaknesses in the chart understanding capabilities of existing MLLMs.
Finding Transformer Circuits with Edge Pruning
Our method finds circuits in GPT-2 that use less than half the number of edges compared to circuits found by previous methods while being equally faithful to the full model predictions on standard circuit-finding tasks.
SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors
First, existing methods often use coarse-grained taxonomies of unsafe topics, and are over-representing some fine-grained topics.
Fantastic Copyrighted Beasts and How (Not) to Generate Them
Our evaluation systematically shows that both image and video generation models can still generate characters even if characters' names are not explicitly mentioned in the prompt, sometimes with only two generic keywords (e. g., prompting with "videogame, plumber" consistently generates Nintendo's Mario character).
AI Risk Management Should Incorporate Both Safety and Security
The exposure of security vulnerabilities in safety-aligned language models, e. g., susceptibility to adversarial attacks, has shed light on the intricate interplay between AI safety and AI security.
SimPO: Simple Preference Optimization with a Reference-Free Reward
Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability.
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
Mixture-of-experts (MoE) models facilitate efficient scaling; however, training the router network introduces the challenge of optimizing a non-differentiable, discrete objective.
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
Instead of finding competing subnetworks, we find that all subnetworks -- whether they generalize or not -- share a set of attention heads, which we refer to as the heuristic core.
Reliable, Adaptable, and Attributable Language Models with Retrieval
Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability.
Long-Context Language Modeling with Parallel Context Encoding
However, the substantial computational cost of transformers and limited generalization of positional encoding restrict the size of their context window.
Improving Language Understanding from Screenshots
An emerging family of language models (LMs), capable of processing both text and images within a single visual view, has the promise to unlock complex tasks such as chart understanding and UI navigation.
QuRating: Selecting High-Quality Data for Training Language Models
We train a QuRater model to learn scalar ratings from pairwise judgments, and use it to annotate a 260B training corpus with quality ratings for each of the four criteria.
LESS: Selecting Influential Data for Targeted Instruction Tuning
Instruction tuning has unlocked powerful capabilities in large language models (LLMs), effectively using combined datasets to develop generalpurpose chatbots.
Interpretability Illusions in the Generalization of Simplified Models
We find consistent generalization gaps: cases in which the simplified proxies are more faithful to the original model on the in-distribution evaluations and less faithful on various tests of systematic generalization.
Poisoning Retrieval Corpora by Injecting Adversarial Passages
Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but to what extent can they be safely deployed in real-world applications?
Detecting Pretraining Data from Large Language Models
Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data.
Evaluating Large Language Models at Evaluating Instruction Following
As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models.
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack.
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models.
Ranked #45 on
Question Answering
on PIQA
Learning Transformer Programs
Recent research in mechanistic interpretability has attempted to reverse-engineer Transformer models by carefully inspecting network weights and activations.
Fine-Tuning Language Models with Just Forward Passes
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory.
C-STS: Conditional Semantic Textual Similarity
Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information retrieval and natural language understanding.
Privacy Implications of Retrieval-Based Language Models
Crucially, we find that $k$NN-LMs are more susceptible to leaking private information from their private datastore than parametric models.
Enabling Large Language Models to Generate Text with Citations
We propose ALCE, the first benchmark for Automatic LLMs' Citation Evaluation.
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option.
Adapting Language Models to Compress Contexts
Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents.
Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations
We investigate the inductive biases of ICL from the perspective of feature bias: which feature ICL is more likely to use given a set of underspecified demonstrations in which two features are equally predictive of the labels.
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
Large language models (LLMs) exploit in-context learning (ICL) to solve tasks with only a few demonstrations, but its mechanisms are not yet well-understood.
Controllable Text Generation with Language Constraints
We consider the task of text generation in language models with constraints specified in natural language.
Training Trajectories of Language Models Across Scales
Why do larger language models demonstrate more desirable behaviors?
MABEL: Attenuating Gender Bias using Textual Entailment Data
To this end, we propose MABEL (a Method for Attenuating Gender Bias using Entailment Labels), an intermediate pre-training approach for mitigating gender bias in contextualized representations.
Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models
Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.
Finding Dataset Shortcuts with Grammar Induction
Many NLP datasets have been found to contain shortcuts: simple decision rules that achieve surprisingly high accuracy.
A Kernel-Based View of Language Model Fine-Tuning
It has become standard to solve NLP tasks by fine-tuning pre-trained language models (LMs), especially in low-data settings.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models
In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks.
Optimizing Test-Time Query Representations for Dense Retrieval
In this paper, we introduce TOUR (Test-Time Optimization of Query Representations), which further optimizes instance-level query representations guided by signals from test-time retrieval results.
Training Language Models with Memory Augmentation
Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component.
Generating Natural Language Proofs with Verifier-Guided Search
In this paper, we present a novel stepwise method, NLProofS (Natural Language Proof Search), which learns to generate relevant steps conditioning on the hypothesis.
Recovering Private Text in Federated Learning of Language Models
For the first time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences.
Can Rationalization Improve Robustness?
Our experiments reveal that the rationale models show the promise to improve robustness, while they struggle in certain scenarios--when the rationalizer is sensitive to positional bias or lexical choices of attack text.
Structured Pruning Learns Compact and Accurate Models
The growing size of neural language models has led to increased attention in model compression.
Should You Mask 15% in Masked Language Modeling?
In this work, we revisit this important choice of MLM pre-training.
Ditch the Gold Standard: Re-evaluating Conversational Question Answering
In this work, we conduct the first large-scale human evaluation of state-of-the-art conversational QA systems, where human evaluators converse with models and judge the correctness of their answers.
Single-dataset Experts for Multi-dataset Question Answering
Many datasets have been created for training reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) generalize and transfer better to new datasets.
Simple Entity-Centric Questions Challenge Dense Retrievers
Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples.
Ranked #3 on
Passage Retrieval
on EntityQuestions
Phrase Retrieval Learns Passage Retrieval, Too
Dense retrieval methods have shown great promise over sparse retrieval methods in a range of NLP problems.
Non-Parametric Few-Shot Learning for Word Sense Disambiguation
One significant challenge in supervised all-words WSD is to classify among senses for a majority of words that lie in the long-tail distribution.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings.
Ranked #3 on
Linear-Probe Classification
on SentEval
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Making Pre-trained Language Models Better Few-shot Learners
We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples.
Ranked #6 on
Zero-Shot Text Classification
on AG News
Learning Dense Representations of Phrases at Scale
Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019).
Ranked #1 on
Question Answering
on Natural Questions (long)
A Frustratingly Easy Approach for Entity and Relation Extraction
Our approach essentially builds on two independent encoders and merely uses the entity model to construct the input for the relation model.
Ranked #1 on
Named Entity Recognition (NER)
on ACE 2005
TextHide: Tackling Data Privacy in Language Understanding Tasks
In addition, TextHide fits well with the popular framework of fine-tuning pre-trained language models (e. g., BERT) for any sentence or sentence-pair task.
Open-Domain Question Answering
This tutorial provides a comprehensive and coherent overview of cutting-edge research in open-domain question answering (QA), the task of answering questions using a large collection of documents of diversified topics.
Dense Passage Retrieval for Open-Domain Question Answering
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method.
Ranked #1 on
Question Answering
on NaturalQA
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
We introduce an approach for open-domain question answering (QA) that retrieves and reads a passage graph, where vertices are passages of text and edges represent relationships that are derived from an external knowledge base or co-occurrence in the same article.
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems.
A Discrete Hard EM Approach for Weakly Supervised Question Answering
Many question answering (QA) tasks only provide weak supervision for how the answer should be computed.
Ranked #2 on
Question Answering
on NarrativeQA
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
Ranked #1 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(Wasserstein Distance (WD) metric, using extra
training data)
SpanBERT: Improving Pre-training by Representing and Predicting Spans
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text.
Ranked #1 on
Question Answering
on TriviaQA
(F1 metric)
CoQA: A Conversational Question Answering Challenge
Humans gather information by engaging in conversations involving a series of interconnected questions and answers.
Ranked #3 on
Generative Question Answering
on CoQA
Position-aware Attention and Supervised Data Improve Slot Filling
The combination of better supervised data and a more appropriate high-capacity model enables much better relation extraction performance.
Ranked #7 on
Relation Extraction
on Re-TACRED
Reading Wikipedia to Answer Open-Domain Questions
This paper proposes to tackle open- domain question answering using Wikipedia as the unique knowledge source: the answer to any factoid question is a text span in a Wikipedia article.
Ranked #1 on
Open-Domain Question Answering
on SQuAD1.1
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP.
Ranked #3 on
Question Answering
on CNN / Daily Mail
Reasoning With Neural Tensor Networks for Knowledge Base Completion
We assess the model by considering the problem of predicting additional true relations between entities given a partial knowledge base.
