Search Results for author:
Found 54 papers, 37 papers with code
Retrieval Data Augmentation Informed by Downstream Question Answering Performance
Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages in those corpora.
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories.
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users.
DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets?
DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents
However, developing and evaluating an AI agent's capacity for end-to-end scientific reasoning is challenging as running real-world experiments is often prohibitively expensive or infeasible.
Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
Despite using 7B models, Husky matches or even exceeds frontier LMs such as GPT-4 on these tasks, showcasing the efficacy of our holistic approach in addressing complex reasoning problems.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
ADaPT: As-Needed Decomposition and Planning with Language Models
Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.
Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
Our experiments with ChatGPT-3. 5 show that this bias is ubiquitous - 80% of our personas demonstrate bias; it is significant - some datasets show performance drops of 70%+; and can be especially harmful for certain groups - some personas suffer statistically significant drops on 80%+ of the datasets.
SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design
We are given a small "personalized dataset" of 10-20 drug synergy relationships in the context of specific cancer cell targets.
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap.
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging.
Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback
We study whether multiple large language models (LLMs) can autonomously improve each other in a negotiation game by playing, reflecting, and criticizing.
Specializing Smaller Language Models towards Multi-Step Reasoning
by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability.
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA.
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks
How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model?
Decomposed Prompting: A Modular Approach for Solving Complex Tasks
On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.
Complexity-Based Prompting for Multi-Step Reasoning
In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts
We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion.
Better Retrieval May Not Lead to Better Question Answering
Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC).
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
Hey AI, Can You Solve Complex Tasks by Talking to Agents?
To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language.
MuSiQue: Multihop Questions via Single-hop Question Composition
Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts.
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
We investigate the effectiveness of natural language interventions for reading-comprehension systems, studying this in the context of social stereotypes.
GooAQ: Open Question Answering with Diverse Answer Types
GooAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results.
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly.
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
However, most existing reading comprehension (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system's performance at identifying a potential lack of sufficient information and locating sources for that information.
ReadOnce Transformers: Reusable Representations of Text for Transformers
We present ReadOnce Transformers, an approach to convert a transformer-based model into one that can build an information-capturing, task-independent, and compressed representation of text.
Temporal Reasoning on Implicit Events from Distant Supervision
We propose TRACIE, a novel temporal reasoning dataset that evaluates the degree to which systems understand implicit events -- events that are not mentioned explicitly in natural language text but can be inferred from it.
UnQovering Stereotyping Biases via Underspecified Questions
Our broad study reveals that (1) all these models, with and without fine-tuning, have notable stereotyping biases in these classes; (2) larger models often have higher bias; and (3) the effect of fine-tuning on bias varies strongly with the dataset and the model size.
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models.
UnifiedQA: Crossing Format Boundaries With a Single QA System
As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats.
Ranked #5 on
Common Sense Reasoning
on WinoGrande
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline.
A Simple Yet Strong Pipeline for HotpotQA
State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition.
Ranked #37 on
Question Answering
on HotpotQA
More Bang for Your Buck: Natural Perturbation for Robust Question Answering
While recent models have achieved human-level scores on many NLP datasets, we observe that they are considerably sensitive to small changes in input.
QASC: A Dataset for Question Answering via Sentence Composition
Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges.
What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
We propose jointly training a model to simultaneously fill this knowledge gap and compose it with the provided partial knowledge.
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.
Question Answering as Global Reasoning over Semantic Abstractions
We propose a novel method for exploiting the semantic structure of text to answer multiple-choice questions.
Repurposing Entailment for Multi-Hop Question Answering Tasks
We introduce Multee, a general architecture that can effectively use entailment models for multi-hop QA tasks.
On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections
The idea is to consider two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic space that captures a noisy grounding of the meaning space in the words of a language---the level at which all systems, whether neural or symbolic, operate.
Exploiting Explicit Paths for Multi-hop Reading Comprehension
To capture additional context, PathNet also composes the passage representations along each path to compute a passage-based representation.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts.
Ranked #26 on
Question Answering
on OpenBookQA
Bridging Knowledge Gaps in Neural Entailment via Symbolic Models
We focus on filling these knowledge gaps in the Science Entailment task, by leveraging an external structured knowledge base (KB) of science facts.
Structure Learning for Relational Logistic Regression: An Ensemble Approach
We consider the problem of learning Relational Logistic Regression (RLR).
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples
We consider the problem of learning textual entailment models with limited supervision (5K-10K training examples), and present two complementary approaches for it.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering.
Learning What is Essential in Questions
Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains.
Answering Complex Questions Using Open Information Extraction
While there has been substantial progress in factoid question-answering (QA), answering complex questions remains challenging, typically requiring both a large body of knowledge and inference techniques.
Question Answering via Integer Programming over Semi-Structured Knowledge
We propose a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.
Markov Logic Networks for Natural Language Question Answering
In the first, we simply use the extracted science rules directly as MLN clauses.
