Search Results for author:
Found 248 papers, 131 papers with code
The Curious Case of Neural Text Degeneration
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL.
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Image Retrieval
on MS COCO
(Recall@10 metric)
VinVL: Revisiting Visual Representations in Vision-Language Models
In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks.
Ranked #2 on
Image-text matching
on CommercialAdsDataset
Defending Against Neural Fake News
We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data.
Ranked #2 on
Fake News Detection
on Grover-Mega
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved.
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017).
Neural Motifs: Scene Graph Parsing with Global Context
We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7. 1% relative gain.
Ranked #8 on
Panoptic Scene Graph Generation
on PSG Dataset
From Recognition to Cognition: Visual Commonsense Reasoning
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+1
Agent Lumos: Unified and Modular Training for Open-Source Language Agents
To foster generalizable agent learning, we collect large-scale, unified, and high-quality training annotations derived from diverse ground-truth reasoning rationales across various complex interactive tasks.
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem.
MAUVE Scores for Generative Models: Theory and Practice
We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images.
MERLOT: Multimodal Neural Script Knowledge Models
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future.
SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration
The potential social harms that large language models pose, such as generating offensive content and reinforcing biases, are steeply rising.
COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Next, we show that ATOMIC 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events.
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Data scarcity has been a long standing issue in the field of open-domain social dialogue.
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics.
RewardBench: Evaluating Reward Models for Language Modeling
In this paper, we present RewardBench, a benchmark dataset and code-base for evaluation, to enhance scientific understanding of reward models.
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans.
Ranked #1 on
Hallucination Pair-wise Detection (4-ref)
on FOIL
Hallucination Pair-wise Detection (1-ref)
Hallucination Pair-wise Detection (4-ref)
+3
Neural AMR: Sequence-to-Sequence Models for Parsing and Generation
Sequence-to-sequence models have shown strong performance across a broad range of applications.
Ranked #6 on
AMR Parsing
on LDC2015E86
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning.
Ranked #1 on
Text Generation
on CommonGen
Dynamic Entity Representations in Neural Language Models
Understanding a long document requires tracking how entities are introduced and evolve over time.
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
We apply this to the ATOMIC resource, and share our new symbolic knowledge graph and commonsense models.
Commonsense Knowledge Base Completion with Structural and Semantic Context
Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1. 5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency.
NaturalProofs: Mathematical Theorem Proving in Natural Language
Understanding and creating mathematics using natural mathematical language - the mixture of symbolic and natural language used by humans - is a challenging and important problem for driving progress in machine learning.
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text.
Counterfactual Story Reasoning and Generation
Counterfactual reasoning requires predicting how alternative events, contrary to what actually happened, might have resulted in different outcomes.
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
First, we propose a new multitask benchmark, RAINBOW, to promote research on commonsense models that generalize well over multiple tasks and datasets.
Ranked #1 on
Question Answering
on SIQA
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Many applications of text generation require incorporating different constraints to control the semantics or style of generated text.
Generated Knowledge Prompting for Commonsense Reasoning
It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations.
Ranked #9 on
Coreference Resolution
on Winograd Schema Challenge
PIQA: Reasoning about Physical Commonsense in Natural Language
Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems.
Ranked #36 on
Question Answering
on PIQA
Natural Language Understanding
Physical Commonsense Reasoning
+1
Unsupervised Commonsense Question Answering with Self-Talk
Natural language understanding involves reading between the lines with implicit background knowledge.
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Although neural conversation models are effective in learning how to produce fluent responses, their primary challenge lies in knowing what to say to make the conversation contentful and non-vacuous.
DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension
DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge.
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem.
Efficient Adaptation of Pretrained Transformers for Abstractive Summarization
Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks.
Abstractive Text Summarization
Natural Language Understanding
Ultra-Fine Entity Typing
We introduce a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e. g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity.
Ranked #4 on
Entity Typing
on Ontonotes v5 (English)
Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information
However, this comparison provides little understanding of how difficult each instance in a given distribution is, or what attributes make the dataset difficult for a given model.
TIMEDIAL: Temporal Commonsense Reasoning in Dialog
In this paper, we present the first study to investigate pre-trained LMs for their temporal reasoning capabilities in dialogs by introducing a new task and a crowd-sourced English challenge set, TIMEDIAL.
Quark: Controllable Text Generation with Reinforced Unlearning
Large-scale language models often learn behaviors that are misaligned with user expectations.
PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
We propose the task of outline-conditioned story generation: given an outline as a set of phrases that describe key characters and events to appear in a story, the task is to generate a coherent narrative that is consistent with the provided outline.
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et.
Ranked #3 on
Vision-Language Navigation
on Room2Room
Do Membership Inference Attacks Work on Large Language Models?
Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data.
Learning to Write with Cooperative Discriminators
Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models, but when used to generate natural language their output tends to be overly generic, repetitive, and self-contradictory.
Neural Metaphor Detection in Context
We present end-to-end neural models for detecting metaphorical word use in context.
ProsocialDialog: A Prosocial Backbone for Conversational Agents
With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
Ranked #1 on
Dialogue Safety Prediction
on ProsocialDialog
Surface Form Competition: Why the Highest Probability Answer Isn't Always Right
Large language models have shown promising results in zero-shot settings (Brown et al., 2020; Radford et al., 2019).
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Large language models have shown promising results in zero-shot settings.
CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
To train CHAMPAGNE, we collect and release YTD-18M, a large-scale corpus of 18M video-based dialogues.
RealTime QA: What's the Answer Right Now?
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version).
Agent AI: Surveying the Horizons of Multimodal Interaction
To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions.
Tuning Language Models by Proxy
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Abductive and counterfactual reasoning, core abilities of everyday human cognition, require reasoning about what might have happened at time t, while conditioning on multiple contexts from the relative past and future.
Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences
In social settings, much of human behavior is governed by unspoken rules of conduct.
We're Afraid Language Models Aren't Modeling Ambiguity
We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset.
A Call for Clarity in Beam Search: How It Works and When It Stops
Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search.
Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes
As AI systems become an increasing part of people's everyday lives, it becomes ever more important that they understand people's ethical norms.
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting.
HellaSwag: Can a Machine Really Finish Your Sentence?
In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.
Ranked #67 on
Sentence Completion
on HellaSwag
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
TuringAdvice: A Generative and Dynamic Evaluation of Language Use
We propose TuringAdvice, a new challenge task and dataset for language understanding models.
Reframing Human-AI Collaboration for Generating Free-Text Explanations
We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop.
NaturalProver: Grounded Mathematical Proof Generation with Language Models
Theorem proving in natural mathematical language - the mixture of symbolic and natural language used by humans - plays a central role in mathematical advances and education, and tests aspects of reasoning that are core to intelligence.
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge
We introduce NeuroComparatives, a novel framework for comparative knowledge distillation overgenerated from language models such as GPT-variants and LLaMA, followed by stringent filtering of the generated knowledge.
Paragraph-level Commonsense Transformers with Recurrent Memory
Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text.
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns.
Zero-Shot Activity Recognition with Verb Attribute Induction
In this paper, we investigate large-scale zero-shot activity recognition by modeling the visual and linguistic attributes of action verbs.
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation.
Twist Decoding: Diverse Generators Guide Each Other
Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available.
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights.
Social Chemistry 101: Learning to Reason about Social and Moral Norms
We present Social Chemistry, a new conceptual formalism to study people's everyday social norms and moral judgments over a rich spectrum of real life situations described in natural language.
Contrastive Explanations for Model Interpretability
Our method is based on projecting model representation to a latent space that captures only the features that are useful (to the model) to differentiate two potential decisions.
Do Neural Language Representations Learn Physical Commonsense?
Humans understand language based on the rich background knowledge about how the physical world works, which in turn allows us to reason about the physical world through language.
Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation
Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation.
Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data.
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
Faith and Fate: Limits of Transformers on Compositionality
We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures.
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer
In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2. 2\% R@1.
Adversarial Filters of Dataset Biases
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
Structured Chemistry Reasoning with Large Language Models
Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in the field of chemistry.
Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
The ability to derive underlying principles from a handful of observations and then generalize to novel situations -- known as inductive reasoning -- is central to human intelligence.
Generative Data Augmentation for Commonsense Reasoning
Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance.
Ranked #1 on
Question Answering
on CODAH
Transparent Human Evaluation for Image Captioning
We establish THumB, a rubric-based human evaluation protocol for image captioning models.
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
To improve AI systems to better reflect value pluralism, the first-order challenge is to explore the extent to which AI systems can model pluralistic human values, rights, and duties as well as their interaction.
MacGyver: Are Large Language Models Creative Problem Solvers?
We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting.
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited.
It's not Rocket Science : Interpreting Figurative Language in Narratives
Figurative language is ubiquitous in English.
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements
Despite the much discussed capabilities of today's language models, they are still prone to silly and unexpected commonsense failures.
Thinking Like a Skeptic: Defeasible Inference in Natural Language
Defeasible inference is a mode of reasoning in which an inference (X is a bird, therefore X flies) may be weakened or overturned in light of new evidence (X is a penguin).
Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics
Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance.
Out-of-Distribution Generalization
Systematic Generalization
Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines
We propose Misinfo Reaction Frames (MRF), a pragmatic formalism for modeling how readers might react to a news headline.
Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines
Even to a simple and short news headline, readers react in a multitude of ways: cognitively (e. g. inferring the writer’s intent), emotionally (e. g. feeling distrust), and behaviorally (e. g. sharing the news with their friends).
REV: Information-Theoretic Evaluation of Free-Text Rationales
More concretely, we propose a metric called REV (Rationale Evaluation with conditional V-information), to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label.
Analyzing Commonsense Emergence in Few-shot Knowledge Models
Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining.
PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning
In addition, we introduce a novel task, Counterfactual Planning, that requires a revision of a plan to cope with a counterfactual situation.
FiLM: Fill-in Language Models for Any-Order Generation
Language models have become the backbone of today's AI systems.
Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs' logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns.
Can Machines Learn Morality? The Delphi Experiment
As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Context is everything, even in commonsense moral reasoning.
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms
NORMLENS consists of 10K human judgments accompanied by free-form explanations covering 2K multimodal situations, and serves as a probe to address two questions: (1) to what extent can models align with average human judgment?
Robust Navigation with Language Pretraining and Stochastic Sampling
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments.
Do Neural Language Models Overcome Reporting Bias?
Mining commonsense knowledge from corpora suffers from reporting bias, over-representing the rare at the expense of the trivial (Gordon and Van Durme, 2013).
ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
We present ATOMIC, an atlas of everyday commonsense reasoning, organized through 877k textual descriptions of inferential knowledge.
NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation
While counterfactual data augmentation offers a promising step towards robust generalization in natural language processing, producing a set of counterfactuals that offer valuable inductive bias for models remains a challenge.
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle.
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales
Existing metrics like task performance of the LM generating the rationales, or similarity between generated and gold rationales are not good indicators of their human utility.
Crystal: Introspective Reasoners Reinforced with Self-Feedback
Extensive work has shown that the performance and interpretability of commonsense reasoning can be improved via knowledge-augmented reasoning methods, where the knowledge that underpins the reasoning process is explicitly verbalized and utilized.
In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search
We further use the data generated by LINK to construct a dataset Logic-Induced-Long-Tail (LINT) that can be used to evaluate downstream models on the long-tail distribution; LINT contains 108K knowledge statements spanning four domains.
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
A writer's style depends not just on personal traits but also on her intent and mental state.
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.
Tailoring Self-Rationalizers with Multi-Reward Distillation
Results on five difficult question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC show that not only does MaRio improve task accuracy, but it also improves the self-rationalization quality of small LMs across the aforementioned axes better than a supervised fine-tuning (SFT) baseline.
STEER: Unified Style Transfer with Expert Reinforcement
We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified frame-work developed to overcome the challenge of limited parallel data for style transfer.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine").
Ranked #4 on
Common Sense Reasoning
on SWAG
Benchmarking Hierarchical Script Knowledge
Understanding procedural language requires reasoning about both hierarchical and temporal relations between events.
Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs
In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs.
Statistical and Computational Guarantees for Influence Diagnostics
Influence diagnostics such as influence functions and approximate maximum influence perturbations are popular in machine learning and in AI domain applications.
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
We release our corpus of chain-of-thought samples and code.
Balancing Shared Autonomy with Human-Robot Communication
Robotic agents that share autonomy with a human should leverage human domain knowledge and account for their preferences when completing a task.
Event2Mind: Commonsense Inference on Events, Intents, and Reactions
We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
Modeling Naive Psychology of Characters in Simple Commonsense Stories
Understanding a narrative requires reading between the lines and reasoning about the unspoken but obvious implications about events and people's mental states - a capability that is trivial for humans but remarkably hard for machines.
Ranked #2 on
Emotion Classification
on ROCStories
Simulating Action Dynamics with Neural Process Networks
Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated.
Discourse-Aware Neural Rewards for Coherent Text Generation
In this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text.
Deep Communicating Agents for Abstractive Summarization
We present deep communicating agents in an encoder-decoder architecture to address the challenges of representing a long document for abstractive summarization.
Ranked #31 on
Abstractive Text Summarization
on CNN / Daily Mail
(using extra training data)
Sounding Board: A User-Centric and Content-Driven Social Chatbot
We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize.
Learning Interpretable Spatial Operations in a Rich 3D Blocks World
In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world.
Verb Physics: Relative Physical Knowledge of Actions and Objects
Learning commonsense knowledge from natural language text is nontrivial due to reporting bias: people rarely state the obvious, e. g., "My house is bigger than me."
Detecting English Writing Styles For Non Native Speakers
This paper presents the first attempt, up to our knowledge, to classify English writing styles on this scale with the challenge of classifying day to day language written by writers with different backgrounds covering various areas of topics. The paper proposes simple machine learning algorithms and simple to generate features to solve hard problems.
Connotation Frames: A Data-Driven Investigation
Through a particular choice of a predicate (e. g., "x violated y"), a writer can subtly connote a range of implied sentiments and presupposed facts about the entities x and y: (1) writer's perspective: projecting x as an "antagonist"and y as a "victim", (2) entities' perspective: y probably dislikes x, (3) effect: something bad happened to y, (4) value: y is something valuable, and (5) mental state: y is distressed by the event.
Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objects
In this paper, we introduce a method to automatically infer object sizes, leveraging visual and textual information from web.
Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing
Next, we show that the association of high-quality segmentations to textual phrases aids in richer semantic understanding and reasoning of these textual phrases.
QuAC : Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Early Fusion for Goal Directed Robotic Vision
Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline.
QuAC: Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Multilingual Connotation Frames: A Case Study on Social Media for Targeted Sentiment Analysis and Forecast
People around the globe respond to major real world events through social media.
Connotation Frames of Power and Agency in Modern Films
The framing of an action influences how we perceive its actor.
Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking
We present an analytic study on the language of news media in the context of political fact-checking and fake news detection.
Story Cloze Task: UW NLP System
This paper describes University of Washington NLP{'}s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task{---}the Story Cloze Task.
Learning to Write by Learning the Objective
Human evaluation demonstrates that text generated by the resulting generator is preferred over that of baselines by a large margin and significantly enhances the overall coherence, style, and information content of the generated text.
TreeTalk: Composition and Compression of Trees for Image Descriptions
We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions.
SocialIQA: Commonsense Reasoning about Social Interactions
We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations.
Ranked #9 on
Question Answering
on SIQA
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
We introduce a new representation language to model precise operation programs corresponding to each math problem that aim to improve both the performance and the interpretability of the learned models.
DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
We present DREAM, the first dialogue-based multiple-choice reading comprehension data set.
Discourse Understanding and Factual Consistency in Abstractive Summarization
We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary.
The Risk of Racial Bias in Hate Speech Detection
We investigate how annotators{'} insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
In this paper, we introduce Cosmos QA, a large-scale dataset of 35, 600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions.
BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle
In this paper, we propose a novel approach to unsupervised sentence summarization by mapping the Information Bottleneck principle to a conditional language modelling objective: given a sentence, our approach seeks a compressed sentence that can best predict the next sentence.
Social IQa: Commonsense Reasoning about Social Interactions
We introduce Social IQa, the first large-scale benchmark for commonsense reasoning about social situations.
Social Bias Frames: Reasoning about Social and Power Implications of Language
We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering
Understanding narratives requires reasoning about implicit world knowledge related to the causes, effects, and states of situations described in text.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
Procedural Reading Comprehension with Attribute-Aware Context Flow
Procedural texts often describe processes (e. g., photosynthesis and cooking) that happen over entities (e. g., light, food).
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
In addition, we provide person-grounding (i. e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text.
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
We introduce a measure of narrative flow and use this to examine the narratives for imagined and recalled events.
Commonsense Reasoning for Natural Language Processing
We organize this tutorial to provide researchers with the critical foundations and recent advances in commonsense representation and reasoning, in the hopes of casting a brighter light on this promising area of future research.
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models
In this paper, we present Reflective Decoding, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks.
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction
Unconscious biases continue to be prevalent in modern text and media, calling for algorithms that can assist writers with bias correction.
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples.
GO FIGURE: A Meta Evaluation of Factuality in Summarization
While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question.
Edited Media Understanding: Reasoning About Implications of Manipulated Images
The difference between this example, and harmful edits that spread disinformation, is one of intent.
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision
In this paper, we investigate the extent to which neural models can reason about natural language rationales that explain model predictions, relying only on distant supervision with no additional annotation cost for human-written rationales.
On-the-Fly Attention Modulation for Neural Generation
Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense.
MultiTalk: A Highly-Branching Dialog Testbed for Diverse Conversations
We study conversational dialog in which there are many possible responses to a given history.
"I'm Not Mad": Commonsense Implications of Negation and Contradiction
In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions.
proScript: Partially Ordered Scripts Generation via Pre-trained Language Models
Scripts - standardized event sequences describing typical everyday activities - have been shown to help understand narratives by providing expectations, resolving ambiguity, and filling in unstated information.
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
``I'm Not Mad'': Commonsense Implications of Negation and Contradiction
In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions.
Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals
The spectacular success of deep generative models calls for quantitative tools to measure their statistical performance.
Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text
To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of Scarecrow -- such as redundancy, commonsense errors, and incoherence -- are identified through several rounds of crowd annotation experiments without a predefined ontology.
Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Understanding manipulated media, from automatically generated {`}deepfakes{'} to manually edited ones, raises novel research challenges.
Reframing Instructional Prompts to GPTk's Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules
Our reasoner uses a state-of-the-art transformer-based generative commonsense knowledge base (KB) as its source of background knowledge for reasoning.
