Search Results for author:
Found 112 papers, 47 papers with code
proScript: Partially Ordered Scripts Generation
Scripts – prototypical event sequences describing everyday activities – have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information.
Think about it! Improving defeasible reasoning by first modeling the question scenario.
Defeasible reasoning is the mode of reasoning where conclusions can be overturned by taking into account new evidence.
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories.
DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets?
DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents
However, developing and evaluating an AI agent's capacity for end-to-end scientific reasoning is challenging as running real-world experiments is often prohibitively expensive or infeasible.
Can Language Models Serve as Text-Based World Simulators?
Can current language models themselves serve as world simulators, correctly predicting how actions change different world states, thus bypassing the need for extensive manual coding?
PDDLEGO: Iterative Planning in Textual Environments
A recent, promising line of work uses LLMs to generate a formal representation of the environment that can be solved by a symbolic planner.
Learning to Reason via Program Generation, Emulation, and Search
CoGEX works by (1) training LMs to generate their own pseudo-programs, (2) teaching them to emulate their generated program's execution, including those leaf functions, allowing the LM's knowledge to fill in the execution gaps; and (3) using them to search over many programs to find an optimal one.
PROC2PDDL: Open-Domain Planning Representations from Texts
Planning in a text-based environment continues to be a major challenge for AI systems.
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic
Recent language models enable new opportunities for structured reasoning with text, such as the construction of intuitive, proof-like textual entailment trees without relying on brittle formal logic.
Data-driven Discovery with Large Generative Models
With the accumulation of data at an unprecedented rate, its potential to fuel scientific discovery is growing exponentially.
Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills
We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement.
The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
In this paper, we present the surprising conclusion that current pretrained language models often generalize relatively well from easy to hard data, even performing as well as oracle models finetuned on hard data.
BaRDa: A Belief and Reasoning Dataset that Separates Factual Accuracy and Reasoning Ability
This shows the clear progression of models towards improved factual accuracy and entailment reasoning, and the dataset provides a new benchmark that more cleanly separates and quantifies these two notions.
Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization
Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need.
Leveraging Code to Improve In-context Learning for Semantic Parsing
In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization.
Digital Socrates: Evaluating LLMs through Explanation Critiques
While LLMs can provide reasoned explanations along with their answers, the nature and quality of those explanations are still poorly understood.
ADaPT: As-Needed Decomposition and Planning with Language Models
Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.
Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
Our experiments with ChatGPT-3. 5 show that this bias is ubiquitous - 80% of our personas demonstrate bias; it is significant - some datasets show performance drops of 70%+; and can be especially harmful for certain groups - some personas suffer statistically significant drops on 80%+ of the datasets.
QualEval: Qualitative Evaluation for Model Improvement
In this work, we address the shortcomings of quantitative metrics by proposing QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement.
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Language agents have shown some ability to interact with an external environment, e. g., a virtual world such as ScienceWorld, to perform complex tasks, e. g., growing a plant, without the startup costs of reinforcement learning.
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy
For example, both normalization and prompting methods for reducing SFC can be ineffective or even detrimental to task performance for some LMs.
Language Models with Rationality
To address this, our goals are to make model beliefs and their inferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reasoning drawn from a consistent network of beliefs.
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability.
Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation
In this paper, we present a novel approach for distilling math word problem solving capabilities from large language models (LLMs) into smaller, more efficient student models.
RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
Despite their unprecedented success, even the largest language models make mistakes.
Self-Refine: Iterative Refinement with Self-Feedback
Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement.
Do language models have coherent mental models of everyday things?
Using these questions as probes, we observe that state-of-the-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent "parts mental models" (54-59% accurate, 19-43% conditional constraint violation).
Lila: A Unified Benchmark for Mathematical Reasoning
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling.
Ranked #1 on
Mathematical Reasoning
on Lila (OOD)
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE
We hypothesize that to perform this task well, the reader needs to mentally elaborate the scene being described to identify a sensible meaning of the language.
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning.
Decomposed Prompting: A Modular Approach for Solving Complex Tasks
On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.
Complexity-Based Prompting for Multi-Step Reasoning
In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.
Ranked #6 on
Science Question Answering
on ScienceQA
NELLIE: A Neuro-Symbolic Inference Engine for Grounded, Compositional, and Explainable Reasoning
Our goal is a modern approach to answering questions via systematic reasoning where answers are supported by human interpretable proof trees grounded in an NL corpus of authoritative facts.
Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement
Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time.
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment
We propose Hard RegSet as a challenging instruction learning task, and a controlled environment for studying instruction learning.
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems.
Memory-assisted prompt editing to improve GPT-3 after deployment
Large LMs such as GPT-3 are powerful, but can commit mistakes that are obvious to humans.
Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback
Our goal is for an LM to continue to improve after deployment, without retraining, using feedback from the user.
DREAM: Improving Situational QA by First Elaborating the Situation
To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model.
Interscript: A dataset for interactive learning of scripts through error feedback
We present a new dataset, Interscript, containing user feedback on a deployed model that generates complex everyday tasks.
How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI
FPs are commonly used in quizzes and interviews to bring out and evaluate the creative reasoning abilities of humans.
Think about it! Improving defeasible reasoning by first modeling the question scenario
Defeasible reasoning is the mode of reasoning where conclusions can be overturned by taking into account new evidence.
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
We show that, in a controlled experimental setting, these two mechanisms result in more consistent beliefs in the overall system, improving both the accuracy and consistency of its answers over time.
General-Purpose Question-Answering with Macaw
Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available.
Improving Neural Model Performance through Natural Language Feedback on Their Explanations
A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures contain errors?
Explaining Answers with Entailment Trees
Our approach is to generate explanations in the form of entailment trees, namely a tree of multipremise entailment steps from facts that are known, through intermediate conclusions, to the hypothesis of interest (namely the question + answer).
Enriching a Model's Notion of Belief using a Persistent Memory
(This is an old and now obsolete draft.
proScript: Partially Ordered Scripts Generation via Pre-trained Language Models
Scripts - standardized event sequences describing typical everyday activities - have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information.
CURIE: An Iterative Querying Approach for Reasoning About Situations
Recently, models have been shown to predict the effects of unexpected situations, e. g., would cloudy skies help or hinder plant growth?
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
In this work we show that a generative model, called ProofWriter, can reliably generate both implications of a theory and the natural language proof(s) that support them.
A Dataset for Tracking Entities in Open Domain Procedural Text
Our solution is a new task formulation where given just a procedural text as input, the task is to generate a set of state change tuples(entity, at-tribute, before-state, after-state)for each step, where the entity, attribute, and state values must be predicted from an open vocabulary.
Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering
The third dataset eOBQA is constructed by adding explanation annotations to the OBQA dataset to test generalization of models trained on eQASC.
Ranked #1 on
Reasoning Chain Explanations
on eQASC
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.
Do Dogs have Whiskers? A New Knowledge Base of hasPart Relations
We present a new knowledge-base of hasPart relationships, extracted from a large corpus of generic statements.
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
In this work, we provide a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements.
Knowledge Patterns
From a modeling perspective, knowledge patterns provide an important insight into the structure of a formal ontology: rather than viewing a formal ontology simply as a list of terms and axioms, knowledge patterns views it as a collection of abstract, modular theories (the "knowledge patterns") plus a collection of modeling decisions stating how different aspects of the world can be modeled using those theories.
What-if I ask you to explain: Explaining the effects of perturbations in procedural text
We address the task of explaining the effects of perturbations in procedural text, an important test of process comprehension.
UnifiedQA: Crossing Format Boundaries With a Single QA System
As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats.
Ranked #5 on
Common Sense Reasoning
on WinoGrande
GenericsKB: A Knowledge Base of Generic Statements
We present a new resource for the NLP community, namely a large (3. 5M+ sentence) knowledge base of *generic statements*, e. g., "Trees remove carbon dioxide from the atmosphere", collected from multiple corpora.
Transformers as Soft Reasoners over Language
However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research.
Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report
This paper reports on the results of the shared tasks of the COIN workshop at EMNLP-IJCNLP 2019.
WIQA: A dataset for ``What if...'' reasoning over procedural text
We introduce WIQA, the first large-scale dataset of {``}What if...{''} questions over procedural text.
QASC: A Dataset for Question Answering via Sentence Composition
Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges.
What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
We propose jointly training a model to simultaneously fill this knowledge gap and compose it with the provided partial knowledge.
Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text
Our goal is to better comprehend procedural text, e. g., a paragraph about photosynthesis, by not only predicting what happens, but why some actions need to happen before others.
WIQA: A dataset for "What if..." reasoning over procedural text
We introduce WIQA, the first large-scale dataset of "What if..." questions over procedural text.
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
QuaRTz contains general qualitative statements, e. g., "A sunscreen with a higher SPF protects the skin longer.
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.
Reasoning Over Paragraph Effects in Situations
A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation.
Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
Prior work has demonstrated that question classification (QC), recognizing the problem domain of a question, can help answer it more accurately.
Be Consistent! Improving Procedural Text Comprehension using Label Consistency
Our goal is procedural text comprehension, namely tracking how the properties of entities (e. g., their location) change with time given a procedural text (e. g., a paragraph about photosynthesis, a recipe).
Declarative Question Answering over Knowledge Bases containing Natural Language Text with Answer Set Programming
While in recent years machine learning (ML) based approaches have been the popular approach in developing end-to-end question answering systems, such systems often struggle when additional knowledge is needed to correctly answer the questions.
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships
Many natural language questions require recognizing and reasoning with qualitative relationships (e. g., in science, economics, and medicine), but are challenging to answer with corpus-based methods.
Exploiting Explicit Paths for Multi-hop Reading Comprehension
To capture additional context, PathNet also composes the passage representations along each path to compute a passage-based representation.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.
Ranked #26 on
Question Answering
on OpenBookQA
Reasoning about Actions and State Changes by Injecting Commonsense Knowledge
Comprehending procedural text, e. g., a paragraph describing photosynthesis, requires modeling actions and the state changes they produce, so that questions about entities at different timepoints can be answered.
Bridging Knowledge Gaps in Neural Entailment via Symbolic Models
We focus on filling these knowledge gaps in the Science Entailment task, by leveraging an external structured knowledge base (KB) of science facts.
What Knowledge is Needed to Solve the RTE5 Textual Entailment Challenge?
The analysis ignores shallow statistical matching techniques between T and H, and rather asks: What would it take to reasonably infer that T implies H?
Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints).
Ranked #4 on
Procedural Text Understanding
on ProPara
What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text
To supply this knowledge, we leverage VerbNet to build a rulebase (called the Semantic Lexicon) of the preconditions and effects of actions, and use it along with commonsense knowledge of persistence to answer questions about change.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering.
Story Generation and Aviation Incident Representation
This working note discusses the topic of story generation, with a view to identifying the knowledge required to understand aviation incident narratives (which have structural similarities to stories), following the premise that to understand aviation incidents, one should at least be able to generate examples of them.
Tell Me Why: Using Question Answering as Distant Supervision for Answer Justification
We propose a neural network architecture for QA that reranks answer justifications as an intermediate (and human-interpretable) step in answer selection.
Ranked #1 on
Question Answering
on AI2 Kaggle Dataset
Framing QA as Building and Ranking Intersentence Answer Justifications
Our best configuration answers 44{\%} of the questions correctly, where the top justifications for 57{\%} of these correct answers contain a compelling human-readable justification that explains the inference required to arrive at the correct answer.
Answering Complex Questions Using Open Information Extraction
While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques.
Domain-Targeted, High Precision Knowledge Extraction
Our goal is to construct a domain-targeted, high precision knowledge base (KB), containing general (subject, predicate, object) statements about the world, in support of a downstream question-answering (QA) application.
What's in an Explanation? Characterizing Knowledge and Inference Requirements for Elementary Science Exams
These explanations are used to create a fine-grained categorization of the requirements.
Creating Causal Embeddings for Question Answering with Minimal Supervision
We argue that a better approach is to look for answers that are related to the question in a relevant way, according to the information need of the question, which may be determined through task-specific embeddings.
Question Answering via Integer Programming over Semi-Structured Knowledge
We propose a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.
Moving Beyond the Turing Test with the Allen AI Science Challenge
Given recent successes in AI (e. g., AlphaGo's victory against Lee Sedol in the game of GO), it's become increasingly important to assess: how close are AI systems to human-level intelligence?
Markov Logic Networks for Natural Language Question Answering
In the first, we simply use the extracted science rules directly as MLN clauses.
Higher-order Lexical Semantic Models for Non-factoid Answer Reranking
We introduce a higher-order formalism that allows all these lexical semantic models to chain direct evidence to construct indirect associations between question and answer texts, by casting the task as the traversal of graphs that encode direct term associations.
