Search Results for author:
Found 49 papers, 23 papers with code
“Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding
When reading a literary piece, readers often make inferences about various characters’ roles, personalities, relationships, intents, actions, etc.
``You are grounded!'': Latent Name Artifacts in Pre-trained Language Models
Pre-trained language models (LMs) may perpetuate biases originating in their training corpus to downstream models.
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation
Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e. g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code.
2 OLMo 2 Furious
Our modified model architecture and training recipe achieve both better training stability and improved per-token efficiency.
From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering
We show that, when added to a general corpus (e. g., Wikipedia), microtheories can supply critical, topical information not necessarily present in the corpus, improving both a model's ability to ground its answers to verifiable knowledge (i. e., show how answers are systematically entailed by documents in the corpus, fully grounding up to +8% more answers), and the accuracy of those grounded answers (up to +8% absolute).
Establishing Task Scaling Laws via Compute-Efficient Model Ladders
We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting.
Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones.
SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs
"), and (c) judgment ("Mary paid for the chips.
OLMoE: Open Mixture-of-Experts Language Models
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE).
Answer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions
Multiple-choice question answering (MCQA) is a key competence of performant transformer language models that is tested by mainstream benchmarks.
OLMES: A Standard for Language Model Evaluations
Evaluating language models in particular is challenging, as small changes to how a model is evaluated on a task can lead to large changes in measured performance.
DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents
However, developing and evaluating an AI agent's capacity for end-to-end scientific reasoning is challenging as running real-world experiments is often prohibitively expensive or infeasible.
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic
Recent language models enable new opportunities for structured reasoning with text, such as the construction of intuitive, proof-like textual entailment trees without relying on brittle formal logic.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations.
Paloma: A Benchmark for Evaluating Language Model Fit
Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.
Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
The success of large language models has shifted the evaluation paradigms in natural language processing (NLP).
BaRDa: A Belief and Reasoning Dataset that Separates Factual Accuracy and Reasoning Ability
This shows the clear progression of models towards improved factual accuracy and entailment reasoning, and the dataset provides a new benchmark that more cleanly separates and quantifies these two notions.
Digital Socrates: Evaluating LLMs through Explanation Critiques
While LLMs can provide reasoned explanations along with their answers, the nature and quality of those explanations are still poorly understood.
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Language agents have shown some ability to interact with an external environment, e. g., a virtual world such as ScienceWorld, to perform complex tasks, e. g., growing a plant, without the startup costs of reinforcement learning.
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy
For example, both normalization and prompting methods for reducing SFC can be ineffective or even detrimental to task performance for some LMs.
Language Models with Rationality
To address this, our goals are to make model beliefs and their inferential relationships explicit, and to resolve inconsistencies that may exist, so that answers are supported by interpretable chains of reasoning drawn from a consistent network of beliefs.
Lila: A Unified Benchmark for Mathematical Reasoning
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling.
Ranked #1 on
Mathematical Reasoning
on Lila (OOD)
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.
Ranked #6 on
Science Question Answering
on ScienceQA
Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement
Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time.
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
We show that, in a controlled experimental setting, these two mechanisms result in more consistent beliefs in the overall system, improving both the accuracy and consistency of its answers over time.
"Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding
When reading a literary piece, readers often make inferences about various characters' roles, personalities, relationships, intents, actions, etc.
General-Purpose Question-Answering with Macaw
Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available.
Explaining Answers with Entailment Trees
Our approach is to generate explanations in the form of entailment trees, namely a tree of multipremise entailment steps from facts that are known, through intermediate conclusions, to the hypothesis of interest (namely the question + answer).
Enriching a Model's Notion of Belief using a Persistent Memory
(This is an old and now obsolete draft.
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
In this work we show that a generative model, called ProofWriter, can reliably generate both implications of a theory and the natural language proof(s) that support them.
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
In this work, we provide a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements.
UnifiedQA: Crossing Format Boundaries With a Single QA System
As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats.
Ranked #5 on
Common Sense Reasoning
on WinoGrande
"You are grounded!": Latent Name Artifacts in Pre-trained Language Models
Pre-trained language models (LMs) may perpetuate biases originating in their training corpus to downstream models.
Transformers as Soft Reasoners over Language
However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research.
SUPP.AI: Finding Evidence for Supplement-Drug Interactions
We fine-tune the contextualized word representations of the RoBERTa language model using labeled DDI data, and apply the fine-tuned model to identify supplement interactions.
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
QuaRTz contains general qualitative statements, e. g., "A sunscreen with a higher SPF protects the skin longer.
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions.
Reasoning Over Paragraph Effects in Situations
A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation.
Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
Prior work has demonstrated that question classification (QC), recognizing the problem domain of a question, can help answer it more accurately.
Declarative Question Answering over Knowledge Bases containing Natural Language Text with Answer Set Programming
While in recent years machine learning (ML) based approaches have been the popular approach in developing end-to-end question answering systems, such systems often struggle when additional knowledge is needed to correctly answer the questions.
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships
Many natural language questions require recognizing and reasoning with qualitative relationships (e. g., in science, economics, and medicine), but are challenging to answer with corpus-based methods.
AllenNLP: A Deep Semantic Natural Language Processing Platform
This paper describes AllenNLP, a platform for research on deep learning methods in natural language understanding.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering.
Semantic Parsing to Probabilistic Programs for Situated Question Answering
Situated question answering is the problem of answering questions about an environment such as an image or diagram.
Moving Beyond the Turing Test with the Allen AI Science Challenge
Given recent successes in AI (e. g., AlphaGo's victory against Lee Sedol in the game of GO), it's become increasingly important to assess: how close are AI systems to human-level intelligence?
