Search Results for author:
Found 61 papers, 35 papers with code
Findings of the 2021 Conference on Machine Translation (WMT21)
This paper presents the results of the newstranslation task, the multilingual low-resourcetranslation for Indo-European languages, thetriangular translation task, and the automaticpost-editing task organised as part of the Con-ference on Machine Translation (WMT) 2021. In the news task, participants were asked tobuild machine translation systems for any of10 language pairs, to be evaluated on test setsconsisting mainly of news stories.
Reframing Instructional Prompts to GPTk’s Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Do pretrained Transformers Really Learn In-context by Gradient Descent?
For example, the theoretical hand-constructed weights used in these studies have properties that don't match those of real LLMs.
SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation
Existing watermarking algorithms are vulnerable to paraphrase attacks because of their token-level design.
Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models
Text generation models are notoriously vulnerable to errors in the training data.
The Trickle-down Impact of Reward (In-)consistency on RLHF
We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM.
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution
GEAR achieves better efficiency by delegating tool grounding and execution to small language models (SLM) and LLM, respectively; while leveraging semantic and pattern-based evaluation at both question and answer levels for generalizable tool grounding.
"According to ..." Prompting Language Models Improves Quoting from Pre-Training Data
Large Language Models (LLMs) may hallucinate and generate fake information, despite pre-training on factual data.
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency
We provide theoretical foundations for this metric and its relationship with other prompt selection metrics, providing a comprehensive understanding of existing methods.
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a wealth of world knowledge.
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations.
Generating Sequences by Learning to Self-Correct
Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content.
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks
How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model?
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
ProsocialDialog: A Prosocial Backbone for Conversational Agents
With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
Ranked #1 on
Dialogue Safety Prediction
on ProsocialDialog
Representation Projection Invariance Mitigates Representation Collapse
In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Many applications of text generation require incorporating different constraints to control the semantics or style of generated text.
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training
We present UnifiedQA-v2, a QA model built with the same process as UnifiedQA, except that it utilizes more supervision -- roughly 3x the number of datasets used for UnifiedQA.
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance.
Hey AI, Can You Solve Complex Tasks by Talking to Agents?
To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language.
Reframing Instructional Prompts to GPTk's Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes.
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones.
GooAQ: Open Question Answering with Diverse Answer Types
GooAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results.
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly.
ParsiNLU: A Suite of Language Understanding Challenges for Persian
Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English.
UnQovering Stereotyping Biases via Underspecified Questions
Our broad study reveals that (1) all these models, with and without fine-tuning, have notable stereotyping biases in these classes; (2) larger models often have higher bias; and (3) the effect of fine-tuning on bias varies strongly with the dataset and the model size.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.
Temporal Common Sense Acquisition with Minimal Supervision
Temporal common sense (e. g., duration and frequency of events) is crucial for understanding natural language.
UnifiedQA: Crossing Format Boundaries With a Single QA System
As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats.
Ranked #6 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
TransOMCS: From Linguistic Graphs to Commonsense Knowledge
Commonsense knowledge acquisition is a key problem for artificial intelligence.
More Bang for Your Buck: Natural Perturbation for Robust Question Answering
While recent models have achieved human-level scores on many NLP datasets, we observe that they are considerably sensitive to small changes in input.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues.
``Going on a vacation'' takes longer than ``Going for a walk'': A Study of Temporal Commonsense Understanding
Understanding time is crucial for understanding events expressed in natural language.
"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
Understanding time is crucial for understanding events expressed in natural language.
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.
Reasoning-Driven Question-Answering for Natural Language Understanding
In the second part, we propose two new challenge datasets.
Solving Hard Coreference Problems
Coreference resolution is a key problem in natural language understanding that still escapes reliable solutions.
Zero-Shot Open Entity Typing as Type-Compatible Grounding
We evaluate our system on a broad range of datasets, including standard fine-grained and coarse-grained entity typing datasets, and also a dataset in the biological domain.
PerspectroScope: A Window to the World of Diverse Perspectives
This work presents PerspectroScope, a web-based system which lets users query a discussion-worthy natural language claim, and extract and visualize various perspectives in support or against the claim, along with evidence supporting each perspective.
Natural Language Inference
Natural Language Understanding
+1
Question Answering as Global Reasoning over Semantic Abstractions
We propose a novel method for exploiting the semantic structure of text to answer multiple-choice questions.
Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
Inherently, this is a natural language understanding task, and we propose to address it as such.
Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims
Inherently, this is a natural language understanding task, and we propose to address it as such.
On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections
The idea is to consider two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic space that captures a noisy grounding of the meaning space in the words of a language---the level at which all systems, whether neural or symbolic, operate.
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences.
Learning What is Essential in Questions
Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains.
Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks
In particular, we provide an initial prototype for a relational and graph traversal query language where queries are directly used as relational features for structured machine learning models.
Better call Saul: Flexible Programming for Learning and Inference in NLP
We present a novel way for designing complex joint inference and learning models using Saul (Kordjamshidi et al., 2015), a recently-introduced declarative learning-based programming language (DeLBP).
Adversarial Delays in Online Strongly-Convex Optimization
We consider the problem of strongly-convex online optimization in presence of adversarial delays; in a T-iteration online game, the feedback of the player's query at time t is arbitrarily delayed by an adversary for d_t rounds and delivered before the game ends, at iteration t+d_t-1.
EDISON: Feature Extraction for NLP, Simplified
We present EDISON, a Java library of feature generation functions used in a suite of state-of-the-art NLP tools, based on a set of generic NLP data structures.
Question Answering via Integer Programming over Semi-Structured Knowledge
We propose a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.
Online Learning with Adversarial Delays
We study the performance of standard online learning algorithms when the feedback is delayed by an adversary.
Clustering With Side Information: From a Probabilistic Model to a Deterministic Algorithm
Empirical studies have been carried out to compare our work with many constrained clustering algorithms from the literature on both a variety of data sets and under a variety of conditions such as using noisy side information and erroneous k values.
