Search Results for author:
Found 76 papers, 38 papers with code
Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
We investigate the potential scale of this vulnerability by measuring the degree to which AI assistants can complete assessment questions in standard university-level STEM courses.
A Logical Fallacy-Informed Framework for Argument Generation
Despite the remarkable performance of Large Language Models (LLMs), they still struggle with generating logically sound arguments, resulting in potential risks such as spreading misinformation.
Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network
Large Language Models (LLMs) have been shown to be effective models of the human language system, with some models predicting most explainable variance of brain activity in current datasets.
ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark
We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems.
Improving Autoformalization using Type Checking
Using GPT-4o as a base model, and combining our method with self-consistency, we obtain a +18. 3% absolute increase in formalization accuracy, and achieve a new state-of-the-art of 53. 2% on ProofNet with Lean 4.
Course Recommender Systems Need to Consider the Job Market
In light of the job market's rapid changes and the current state of research in course recommender systems, we outline essential properties for course recommender systems to address these demands effectively, including explainable, sequential, unsupervised, and aligned with the job market and user's goals.
A Design Space for Intelligent and Interactive Writing Assistants
In our era of rapid technological advancement, the research landscape for writing assistants has become increasingly fragmented across various research communities.
ConGeo: Robust Cross-view Geo-localization across Ground View Variations
Cross-view geo-localization aims at localizing a ground-level query image by matching it to its corresponding geo-referenced aerial view.
Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs
Event commonsense reasoning requires the ability to reason about the relationship between events, as well as infer implicit context underlying that relationship.
"Flex Tape Can't Fix That": Bias and Misinformation in Edited Language Models
We introduce a novel benchmark dataset, Seesaw-CF, for measuring bias-related harms of model editing and conduct the first in-depth investigation of how different weight-editing methods impact model bias.
DiffuCOMET: Contextual Commonsense Knowledge Diffusion
Inferring contextually-relevant and diverse commonsense to understand narratives remains challenging for knowledge models.
Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
In this paper, we perform a causal mediation analysis on twelve LLMs to examine how intermediate reasoning steps generated by the LLM influence the final outcome and find that LLMs do not reliably use their intermediate reasoning steps when generating an answer.
ConVQG: Contrastive Visual Question Generation with Multimodal Guidance
However, generating focused questions using textual constraints while enforcing a high relevance to the image content remains a challenge, as VQG systems often ignore one or both forms of grounding.
Rethinking Skill Extraction in the Job Market Domain using Large Language Models
Skill Extraction involves identifying skills and qualifications mentioned in documents such as job postings and resumes.
JOBSKAPE: A Framework for Generating Synthetic Job Postings to Enhance Skill Matching
Recent approaches in skill matching, employing synthetic training data for classification or similarity model training, have shown promising results, reducing the need for time-consuming and expensive annotations.
Efficient Tool Use with Chain-of-Abstraction Reasoning
LLM agents trained with our method also show more efficient tool use, with inference speed being on average ~1. 4x faster than baseline tool-augmented LLMs.
Evaluating Language Model Agency through Negotiations
We introduce an approach to evaluate language model (LM) agency using negotiation games.
Exploring Defeasibility in Causal Reasoning
CESAR achieves a significant 69. 7% relative improvement over existing metrics, increasing from 47. 2% to 80. 1% in capturing the causal strength change brought by supporters and defeaters.
Instruction-tuning Aligns LLMs to the Human Brain
Instruction-tuning is a widely adopted finetuning method that enables large language models (LLMs) to generate output that more closely resembles human responses.
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Large language models (LLMs) can potentially democratize access to medical knowledge.
Ranked #1 on
Multiple Choice Question Answering (MCQA)
on MedMCQA
(Dev Set (Acc-%) metric)
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives.
Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention
We evaluate the cross-lingual reasoning abilities of MultiLMs in two schemes: (1) where the language of the context and the question remain the same in the new languages that are tested (i. e., the reasoning is still monolingual, but the model must transfer the learned reasoning ability across languages), and (2) where the language of the context and the question is different (which we term code-switched reasoning).
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases.
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Recent efforts in natural language processing (NLP) commonsense reasoning research have yielded a considerable number of new datasets and benchmarks.
Discovering Knowledge-Critical Subnetworks in Pretrained Language Models
In this work, we investigate whether pretrained language models contain various knowledge-critical subnetworks: particular sparse computational subgraphs responsible for encoding specific knowledge the model has memorized.
Let Me Teach You: Pedagogical Foundations of Feedback for Language Models
Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning Large Language Models (LLMs) to human preferences.
Mitigating Label Biases for In-context Learning
In this work, we define a typology for three types of label biases in ICL for text classification: vanilla-label bias, context-label bias, and domain-label bias (which we conceptualize and detect for the first time).
CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering
The task of zero-shot commonsense question answering evaluates models on their capacity to reason about general scenarios beyond those presented in specific datasets.
RECKONING: Reasoning through Dynamic Knowledge Encoding
In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge.
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives
They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story.
REFINER: Reasoning Feedback on Intermediate Representations
Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e. g., chain-of-thought prompting.
DISCO: Distilling Counterfactuals with Large Language Models
However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale.
kogito: A Commonsense Knowledge Inference Toolkit
We also include helper functions for converting natural language texts into a format ingestible by knowledge models - intermediate pipeline stages such as knowledge head extraction from text, heuristic and model-based knowledge head-relation matching, and an ability to define and use custom knowledge relations.
ComFact: A Benchmark for Linking Contextual Commonsense Knowledge
Understanding rich narratives, such as dialogues and stories, often requires natural language processing systems to access relevant knowledge from commonsense knowledge graphs.
Deep Bidirectional Language-Knowledge Graph Pretraining
Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks.
Ranked #1 on
Riddle Sense
on RiddleSense
Memory-Based Model Editing at Scale
We find that only SERAC achieves high performance on all three problems, consistently outperforming existing approaches to model editing by a significant margin.
Conditional set generation using Seq2seq models
Conditional set generation learns a mapping from an input sequence of tokens to a set.
Discovering Language-neutral Sub-networks in Multilingual Language Models
Multilingual pre-trained language models transfer remarkably well on cross-lingual downstream tasks.
Synthetic Disinformation Attacks on Automated Fact Verification Systems
Automated fact-checking is a needed technology to curtail the spread of online misinformation.
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it.
Fast Model Editing at Scale
To enable easy post-hoc editing at scale, we propose Model Editor Networks using Gradient Decomposition (MEND), a collection of small auxiliary editing networks that use a single desired input-output pair to make fast, local edits to a pre-trained model's behavior.
GreaseLM: Graph REASoning Enhanced Language Models
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it.
Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules
Our reasoner uses a state-of-the-art transformer-based generative commonsense knowledge base (KB) as its source of background knowledge for reasoning.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Understanding manipulated media, from automatically generated {`}deepfakes{'} to manually edited ones, raises novel research challenges.
End-to-End Task-Oriented Dialog Modeling with Semi-Structured Knowledge Management
To address this task, we propose a TOD system with semi-structured knowledge management, SeKnow, which extends the belief state to manage knowledge with both structured and unstructured contents.
``I'm Not Mad'': Commonsense Implications of Negation and Contradiction
In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions.
"I'm Not Mad": Commonsense Implications of Negation and Contradiction
In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions.
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG.
Ranked #2 on
Riddle Sense
on RiddleSense
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
On-the-Fly Attention Modulation for Neural Generation
Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense.
Analyzing Commonsense Emergence in Few-shot Knowledge Models
Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining.
Edited Media Understanding: Reasoning About Implications of Manipulated Images
The difference between this example, and harmful edits that spread disinformation, is one of intent.
The Amazing World of Neural Language Generation
Neural Language Generation (NLG) {--} using neural network models to generate coherent text {--} is among the most promising methods for automated text creation.
COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Next, we show that ATOMIC 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Abductive and counterfactual reasoning, core abilities of everyday human cognition, require reasoning about what might have happened at time t, while conditioning on multiple contexts from the relative past and future.
Commonsense Reasoning for Natural Language Processing
We organize this tutorial to provide researchers with the critical foundations and recent advances in commonsense representation and reasoning, in the hopes of casting a brighter light on this promising area of future research.
Procedural Reading Comprehension with Attribute-Aware Context Flow
Procedural texts often describe processes (e. g., photosynthesis and cooking) that happen over entities (e. g., light, food).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering
Understanding narratives requires reasoning about implicit world knowledge related to the causes, effects, and states of situations described in text.
WIQA: A dataset for ``What if...'' reasoning over procedural text
We introduce WIQA, the first large-scale dataset of {``}What if...{''} questions over procedural text.
Commonsense Knowledge Base Completion with Structural and Semantic Context
Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1. 5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency.
WIQA: A dataset for "What if..." reasoning over procedural text
We introduce WIQA, the first large-scale dataset of "What if..." questions over procedural text.
Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text
Our goal is to better comprehend procedural text, e. g., a paragraph about photosynthesis, by not only predicting what happens, but why some actions need to happen before others.
Counterfactual Story Reasoning and Generation
Counterfactual reasoning requires predicting how alternative events, contrary to what actually happened, might have resulted in different outcomes.
Discourse Understanding and Factual Consistency in Abstractive Summarization
We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary.
Be Consistent! Improving Procedural Text Comprehension using Label Consistency
Our goal is procedural text comprehension, namely tracking how the properties of entities (e. g., their location) change with time given a procedural text (e. g., a paragraph about photosynthesis, a recipe).
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017).
Efficient Adaptation of Pretrained Transformers for Abstractive Summarization
Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks.
Abstractive Text Summarization
Natural Language Understanding
Reasoning about Actions and State Changes by Injecting Commonsense Knowledge
Comprehending procedural text, e. g., a paragraph describing photosynthesis, requires modeling actions and the state changes they produce, so that questions about entities at different timepoints can be answered.
Modeling Naive Psychology of Characters in Simple Commonsense Stories
Understanding a narrative requires reading between the lines and reasoning about the unspoken but obvious implications about events and people's mental states - a capability that is trivial for humans but remarkably hard for machines.
Ranked #2 on
Emotion Classification
on ROCStories
Learning to Write with Cooperative Discriminators
Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models, but when used to generate natural language their output tends to be overly generic, repetitive, and self-contradictory.
Discourse-Aware Neural Rewards for Coherent Text Generation
In this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text.
Deep Communicating Agents for Abstractive Summarization
We present deep communicating agents in an encoder-decoder architecture to address the challenges of representing a long document for abstractive summarization.
Ranked #31 on
Abstractive Text Summarization
on CNN / Daily Mail
(using extra training data)
Learning to Write by Learning the Objective
Human evaluation demonstrates that text generated by the resulting generator is preferred over that of baselines by a large margin and significantly enhances the overall coherence, style, and information content of the generated text.
Simulating Action Dynamics with Neural Process Networks
Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated.
