Search Results for author:
Found 69 papers, 35 papers with code
Attentiveness to Answer Choices Doesn't Always Entail High QA Accuracy
For example, encouraging models to generate a valid answer choice can, in fact, be detrimental to task performance for some LMs, and prior probability normalization methods are less effective (sometimes even detrimental) to instruction-tuned LMs.
Improving Language Models via Plug-and-Play Retrieval Feedback
ReFeed first generates initial outputs, then utilizes a retrieval model to acquire relevant information from large document collections, and finally incorporates the retrieved information into the in-context demonstration for output refinement, thereby addressing the limitations of LLMs in a more efficient and cost-effective manner.
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability.
Language Models with Rationality
Our goal is to uncover such dependencies and reduce inconsistencies among them, so that answers are supported by faithful, system-believed chains of reasoning drawn from a consistent network of beliefs.
Specializing Smaller Language Models towards Multi-Step Reasoning
by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability.
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
This is insufficient, however, when the necessary knowledge is not available or up-to-date within a model's parameters.
DISCO: Distilling Counterfactuals with Large Language Models
However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale.
Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs
Can we teach natural language understanding models to track their beliefs through intermediate points in text?
Lila: A Unified Benchmark for Mathematical Reasoning
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling.
Ranked #1 on
Mathematical Reasoning
on Lila (OOD)
A Logic for Expressing Log-Precision Transformers
One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text.
Decomposed Prompting: A Modular Approach for Solving Complex Tasks
On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.
Complexity-Based Prompting for Multi-Step Reasoning
In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
The Parallelism Tradeoff: Limitations of Log-Precision Transformers
Despite their omnipresence in modern NLP, characterizing the computational power of transformer neural nets remains an interesting open question.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts
We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion.
Better Retrieval May Not Lead to Better Question Answering
Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC).
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment
We propose Hard RegSet as a challenging instruction learning task, and a controlled environment for studying instruction learning.
Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability
Our results, however, reveal important limitations too: a careful sampling of training data is crucial for building models that generalize to larger problems, and transformer models' limited scale-invariance suggests they are far from learning robust deductive reasoning algorithms.
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI
FPs are commonly used in quizzes and interviews to bring out and evaluate the creative reasoning abilities of humans.
Hey AI, Can You Solve Complex Tasks by Talking to Agents?
To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language.
MuSiQue: Multihop Questions via Single-hop Question Composition
Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts.
Saturated Transformers are Constant-Depth Threshold Circuits
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages.
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes.
GooAQ: Open Question Answering with Diverse Answer Types
GooAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results.
Multi-Modal Answer Validation for Knowledge-Based VQA
Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source.
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.
ReadOnce Transformers: Reusable Representations of Text for Transformers
We present ReadOnce Transformers, an approach to convert a transformer-based model into one that can build an information-capturing, task-independent, and compressed representation of text.
Temporal Reasoning on Implicit Events from Distant Supervision
We propose TRACIE, a novel temporal reasoning dataset that evaluates the degree to which systems understand implicit events -- events that are not mentioned explicitly in natural language text but can be inferred from it.
UnQovering Stereotyping Biases via Underspecified Questions
Our broad study reveals that (1) all these models, with and without fine-tuning, have notable stereotyping biases in these classes; (2) larger models often have higher bias; and (3) the effect of fine-tuning on bias varies strongly with the dataset and the model size.
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.
Belief Propagation Neural Networks
Learned neural solvers have successfully been used to solve combinatorial optimization and decision problems.
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
UnifiedQA: Crossing Format Boundaries With a Single QA System
As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats.
Ranked #1 on
Question Answering
on CommonsenseQA
A Simple Yet Strong Pipeline for HotpotQA
State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition.
Ranked #40 on
Question Answering
on HotpotQA
More Bang for Your Buck: Natural Perturbation for Robust Question Answering
While recent models have achieved human-level scores on many NLP datasets, we observe that they are considerably sensitive to small changes in input.
Adversarial Filters of Dataset Biases
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.
What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge
Open-domain question answering (QA) is known to involve several underlying knowledge and reasoning challenges, but are models actually learning such knowledge when trained on benchmark tasks?
Approximating the Permanent by Sampling from Adaptive Partitions
Computing the permanent of a non-negative matrix is a core problem with practical applications ranging from target tracking to statistical thermodynamics.
Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues.
QASC: A Dataset for Question Answering via Sentence Composition
Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges.
Towards Efficient Discrete Integration via Adaptive Quantile Queries
On a suite of UAI inference challenge benchmarks, it saves 81. 5% of WISH queries while retaining the quality of results.
What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
We propose jointly training a model to simultaneously fill this knowledge gap and compose it with the provided partial knowledge.
Probing Natural Language Inference Models through Semantic Fragments
Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new fragments, even though the phenomena probed here are central to the NLI task.
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.
Question Answering as Global Reasoning over Semantic Abstractions
We propose a novel method for exploiting the semantic structure of text to answer multiple-choice questions.
Repurposing Entailment for Multi-Hop Question Answering Tasks
We introduce Multee, a general architecture that can effectively use entailment models for multi-hop QA tasks.
On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections
The idea is to consider two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic space that captures a noisy grounding of the meaning space in the words of a language---the level at which all systems, whether neural or symbolic, operate.
Expanding Holographic Embeddings for Knowledge Completion
Neural models operating over structured spaces such as knowledge graphs require a continuous embedding of the discrete elements of this space (such as entities) as well as the relationships between them.
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships
Many natural language questions require recognizing and reasoning with qualitative relationships (e. g., in science, economics, and medicine), but are challenging to answer with corpus-based methods.
Exploiting Explicit Paths for Multi-hop Reading Comprehension
To capture additional context, PathNet also composes the passage representations along each path to compute a passage-based representation.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.
Bridging Knowledge Gaps in Neural Entailment via Symbolic Models
We focus on filling these knowledge gaps in the Science Entailment task, by leveraging an external structured knowledge base (KB) of science facts.
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples
We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering.
Approximate Inference via Weighted Rademacher Complexity
Rademacher complexity is often used to characterize the learnability of a hypothesis class and is known to be related to the class size.
Learning What is Essential in Questions
Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains.
Answering Complex Questions Using Open Information Extraction
While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques.
Knowledge Completion for Generics using Guided Tensor Factorization
Given a knowledge base or KB containing (noisy) facts about common nouns or generics, such as "all trees produce oxygen" or "some animals live in forests", we consider the problem of inferring additional such facts at a precision similar to that of the starting KB.
Adaptive Concentration Inequalities for Sequential Decision Problems
A key challenge in sequential decision problems is to determine how many samples are needed for an agent to make reliable decisions with good probabilistic guarantees.
Question Answering via Integer Programming over Semi-Structured Knowledge
We propose a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.
Selecting Near-Optimal Learners via Incremental Data Allocation
We study a novel machine learning (ML) problem setting of sequentially allocating small subsets of training data amongst a large set of classifiers.
Markov Logic Networks for Natural Language Question Answering
In the first, we simply use the extracted science rules directly as MLN clauses.
Parsing Algebraic Word Problems into Equations
This paper formalizes the problem of solving multi-sentence algebraic word problems as that of generating and scoring equation trees.
Embed and Project: Discrete Sampling with Universal Hashing
We consider the problem of sampling from a probability distribution defined over a high-dimensional discrete set, specified for instance by a graphical model.
Optimization With Parity Constraints: From Binary Codes to Discrete Integration
Many probabilistic inference tasks involve summations over exponentially large sets.
Density Propagation and Improved Bounds on the Partition Function
Given a probabilistic graphical model, its density of states is a function that, for any likelihood value, gives the number of configurations with that probability.
Accelerated Adaptive Markov Chain for Partition Function Computation
We propose a novel Adaptive Markov Chain Monte Carlo algorithm to compute the partition function.
