Search Results for author:
Found 113 papers, 73 papers with code
EchoGen: Generating Conclusions from Echocardiogram Notes
Generating a summary from findings has been recently explored (Zhang et al., 2018, 2020) in note types such as radiology reports that typically have short length.
Contemporary NLP Modeling in Six Comprehensive Programming Assignments
We present a series of programming assignments, adaptable to a range of experience levels from advanced undergraduate to PhD, to teach students design and implementation of modern NLP systems.
Can NLI Models Verify QA Systems’ Predictions?
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just “good enough” in the context of imperfect QA datasets.
RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models
Although large language models (LLMs) have become generally more capable and accurate across many tasks, some fundamental sources of unreliability remain in their behavior.
Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding
This task is generally treated as a binary judgment of whether the claim is supported or unsupported in relation to the source.
ProofWala: Multilingual Proof Data Synthesis and Theorem-Proving
We address this weakness with a multilingual proof framework, ${\rm P{\small ROOF}W{\small ALA}}$, that allows a standardized form of interaction between neural theorem-provers and two established ITPs (Coq and Lean).
LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
LongProc consists of six diverse procedural generation tasks, such as extracting structured information from HTML pages into a TSV format and executing complex search procedures to create travel plans.
Understanding Synthetic Context Extension via Retrieval Heads
To defray the cost of pretraining LLMs over long contexts, recent work takes an approach of synthetic context extension: fine-tuning LLMs with synthetically generated long-context data in a post-training stage.
Contrastive Learning to Improve Retrieval for Real-world Fact Checking
By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset.
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs).
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
An instance in our benchmark consists of a synthetic API function update paired with a program synthesis example that uses the updated functionality; our goal is to update an LLM to be able to solve this program synthesis example without providing documentation of the update at inference time.
Learning to Refine with Fine-Grained Natural Language Feedback
Recent work has explored the capability of large language models (LLMs) to identify and correct errors in LLM-generated responses.
Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification
Automatic factuality verification of large language model (LLM) generations is becoming more and more widely used to combat hallucinations.
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
Alignment suppresses irrelevant and unhelpful content while shifting the output distribution toward longer responses that cover information spanning several responses from the base LLM, essentially presenting diverse information in a single response.
LoFiT: Localized Fine-tuning on LLM Representations
For truthfulness and reasoning tasks, we find that LoFiT's intervention vectors are more effective for LLM adaptation than vectors from representation intervention methods such as Inference-time Intervention.
SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation
It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints.
D2PO: Discriminator-Guided DPO with Response Evaluation Models
Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO.
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
We release LLM-AggreFact, code for data synthesis, and models.
Which questions should I answer? Salience Prediction of Inquisitive Questions
QSALIENCE is instruction-tuned over our dataset of linguist-annotated salience scores of 1, 766 (context, question) pairs.
A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation
Recent work has shown that energy-based language modeling is an effective framework for controllable text generation because it enables flexible integration of arbitrary discriminators.
MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.
QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsing
Questions Under Discussion (QUD) is a versatile linguistic framework in which discourse progresses as continuously asking questions and answering them.
A Long Way to Go: Investigating Length Correlations in RLHF
Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering.
X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs
Aligned paragraphs are sourced from Wikipedia pages in different languages, reflecting real information divergences observed in the wild.
Deductive Additivity for Planning of Natural Language Proofs
Specifically, we evaluate whether embedding spaces exhibit a property we call deductive additivity: the sum of premise statement embeddings should be close to embeddings of conclusions based on those premises.
Propagating Knowledge Updates to LMs Through Distillation
Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set.
EEL: Efficiently Encoding Lattices for Reranking
Standard decoding approaches for conditional text generation tasks typically search for an output hypothesis with high model probability, but this may not yield the best hypothesis according to human judgments of quality.
Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses
To tackle this problem, we propose a controlled text generation method that uses a novel contrastive learning strategy to encourage models to differentiate between generating likely and less likely outputs according to humans.
Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing
Developers often dedicate significant time to maintaining and refactoring existing code.
Drafting Event Schemas using Language Models
Past work has studied event prediction and event language modeling, sometimes mediated through structured representations of knowledge in the form of event schemas.
Using Natural Language Explanations to Rescale Human Judgments
These scores should reflect the annotators' underlying assessments of the example.
Complex Claim Verification with Evidence Retrieved in the Wild
Evidence retrieval is a core part of automatic fact-checking.
SatLM: Satisfiability-Aided Language Models Using Declarative Prompting
In this paper, we propose a new satisfiability-aided language modeling (SatLM) approach for improving the reasoning capabilities of LLMs.
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge
Pre-trained language models (LMs) are used for knowledge intensive tasks like question answering, but their knowledge gets continuously outdated as the world changes.
TypeT5: Seq2seq Type Inference using Static Analysis
There has been growing interest in automatically predicting missing type annotations in programs written in Python and JavaScript.
WiCE: Real-World Entailment for Claims in Wikipedia
Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation.
Modeling Complex Event Scenarios via Simple Entity-focused Questions
Event scenarios are often complex and involve multiple event sequences connected through different entity participants.
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
We first generate sets of candidate explanations for each example in the prompt using a leave-one-out scheme, then find an effective combination of these explanations with a two-stage framework.
Prompted Opinion Summarization with GPT-3.5
Large language models have shown impressive performance across a wide variety of tasks, including text summarization.
Complementary Explanations for Effective In-Context Learning
Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective.
Natural Language Deduction with Incomplete Information
A growing body of work studies how to answer a question or verify a claim by generating a natural language "proof": a chain of deductive inferences yielding the answer based on a set of premises.
Assessing Out-of-Domain Language Model Performance from Few Examples
We address the task of predicting out-of-domain (OOD) performance in a few-shot fashion: given a few target-domain examples and a set of models with similar training performance, can we understand how these models will perform on OOD test data?
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Despite recent progress in abstractive summarization, models often generate summaries with factual errors.
Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion
Human evaluation results show that QUD dependency parsing is possible for language models trained with this crowdsourced, generalizable annotation scheme.
News Summarization and Evaluation in the Era of GPT-3
Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting.
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
SNaC: Coherence Error Detection for Narrative Summarization
In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
Generating Literal and Implied Subquestions to Fact-check Complex Claims
Verifying complex political claims is a challenging task, especially when politicians use various tactics to subtly misrepresent the facts.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning?
Entity Cloze By Date: What LMs Know About Unseen Entities
Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge.
Natural Language Deduction through Search over Statement Compositions
In settings from fact-checking to question answering, we frequently want to know whether a collection of evidence (premises) entails a hypothesis.
Massive-scale Decoding for Text Generation using Lattices
Conditional neural text generation models generate high-quality outputs, but often concentrate around a mode when what we really want is a diverse set of options.
Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections
While there has been substantial progress in text comprehension through simple factoid question answering, more holistic comprehension of a discourse still presents a major challenge (Dunietz et al., 2020).
Training Dynamics for Text Summarization Models
Across different datasets (CNN/DM, XSum, MediaSum) and summary properties, such as abstractiveness and hallucination, we study what the model learns at different stages of its fine-tuning process.
ASPECTNEWS: Aspect-Oriented Summarization of News Documents
Generic summaries try to cover an entire document and query-based summaries try to answer document-specific questions.
Cross-Lingual Fine-Grained Entity Typing
In this paper, we present a unified cross-lingual fine-grained entity typing model capable of handling over 100 languages and analyze this model's ability to generalize to languages and entities unseen during training.
Making Document-Level Information Extraction Right for the Right Reasons
We show that regularization with small amounts of evidence supervision during training can substantially improve the quality of extracted evidence.
Can Explanations Be Useful for Calibrating Black Box Models?
Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not.
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
We introduce CREAK, a testbed for commonsense reasoning about entity knowledge, bridging fact-checking about entities (Harry Potter is a wizard and is skilled at riding a broomstick) with commonsense inferences (if you're good at a skill you can teach others how to do it).
Don't Let Discourse Confine Your Model: Sequence Perturbations for Improved Event Language Models
Event language models represent plausible sequences of events.
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
Despite the prominence of neural abstractive summarization models, we know little about how they actually form summaries and how to understand where their decisions come from.
Can NLI Models Verify QA Systems' Predictions?
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets.
Flexible Generation of Natural Language Deductions
Natural language is an attractive representation for this purpose -- it is both highly expressive and easy for humans to understand.
Annotating and Modeling Fine-grained Factuality in Summarization
Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors.
Did they answer? Subjective acts and intents in conversational discourse
Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences.
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model's prediction might change as well.
Model Agnostic Answer Reranking System for Adversarial Question Answering
While numerous methods have been proposed as defenses against adversarial examples in question answering (QA), these techniques are often model specific, require retraining of the model, and give only marginal improvements in performance over vanilla models.
Modeling Fine-Grained Entity Types with Box Embeddings
Neural entity typing models typically represent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types' complex interdependencies.
Ranked #9 on
Entity Typing
on Open Entity
Conditional Generation of Temporally-ordered Event Sequences
We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence.
Effective Distant Supervision for Temporal Relation Extraction
A principal barrier to training temporal relation extraction models in new domains is the lack of varied, high quality examples and the challenge of collecting more.
Understanding Neural Abstractive Summarization Models via Uncertainty
An advantage of seq2seq abstractive summarization models is that they generate text in a free-form manner, but this flexibility makes it difficult to interpret model behavior.
Compressive Summarization with Plausibility and Salience Modeling
Compressive summarization systems typically rely on a crafted set of syntactic rules to determine what spans of possible summary sentences can be deleted, then learn a model of what to actually delete by optimizing for content selection (ROUGE).
Evaluating Factuality in Generation with Dependency-level Entailment
Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.
Optimal Neural Program Synthesis from Multimodal Specifications
Multimodal program synthesis, which leverages different types of user input to synthesize a desired program, is an attractive way to scale program synthesis to challenging settings; however, it requires integrating noisy signals from the user, like natural language, with hard constraints on the program's behavior.
Inquisitive Question Generation for High Level Text Comprehension
Inquisitive probing questions come naturally to humans in a variety of settings, but is a challenging task for automatic systems.
Tradeoffs in Sentence Selection Techniques for Open-Domain Question Answering
Current methods in open-domain question answering (QA) usually employ a pipeline of first retrieving relevant documents, then applying strong reading comprehension (RC) models to that retrieved text.
Narrative Interpolation for Generating and Understanding Stories
We propose a method for controlled narrative/story generation where we are able to guide the model to produce coherent narratives with user-specified target endings by interpolation: for example, we are told that Jim went hiking and at the end Jim needed to be rescued, and we want the model to incrementally generate steps along the way.
Neural Syntactic Preordering for Controlled Paraphrase Generation
Paraphrasing natural language sentences is a multifaceted process: it might involve replacing individual words or short phrases, local rearrangement of content, or high-level restructuring like topicalization or passivization.
Benchmarking Multimodal Regex Synthesis with Complex Structures
Existing datasets for regular expression (regex) generation from natural language are limited in complexity; compared to regex tasks that users post on StackOverflow, the regexes in these datasets are simple, and the language used to describe them is not diverse.
Interpretable Entity Representations through Large-Scale Typing
On entity probing tasks involving recognizing entity identity, our embeddings used in parameter-free downstream models achieve competitive performance with ELMo- and BERT-based embeddings in trained models.
Robust Question Answering Through Sub-part Alignment
Current textual question answering models achieve strong performance on in-domain test sets, but often do so by fitting surface-level patterns in the data, so they fail to generalize to out-of-distribution settings.
LambdaNet: Probabilistic Type Inference using Graph Neural Networks
Given this program abstraction, we then use a graph neural network to propagate information between related type variables and eventually make type predictions.
Byte Pair Encoding is Suboptimal for Language Model Pretraining
We analyze differences between BPE and unigram LM tokenization, finding that the latter method recovers subword units that align more closely with morphology and avoids problems stemming from BPE's greedy construction procedure.
Calibration of Pre-trained Transformers
Pre-trained Transformers are now ubiquitous in natural language processing, but despite their high end-task performance, little is known empirically about whether they are calibrated.
Multi-hop Question Answering via Reasoning Chains
Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way.
Ranked #4 on
Question Answering
on WikiHop
Query-Focused Scenario Construction
The news coverage of events often contains not one but multiple incompatible accounts of what happened.
Fine-Grained Entity Typing for Domain Independent Entity Linking
For this problem, a domain is characterized not just by genre of text but even by factors as specific as the particular distribution of entities, as neural models tend to overfit by memorizing properties of frequent entities in a dataset.
Effective Use of Transformer Networks for Entity Tracking
Tracking entities in procedural language requires understanding the transformations arising from actions on entities as well as those entities' interactions.
Sketch-Driven Regular Expression Generation from Natural Language and Examples
Our system achieves state-of-the-art performance on the prior datasets and solves 57% of the real-world dataset, which existing neural systems completely fail on.
Embedding time expressions for deep temporal ordering models
Data-driven models have demonstrated state-of-the-art performance in inferring the temporal ordering of events in text.
Evaluating Discourse in Structured Text Representations
Discourse structure is integral to understanding a text and is helpful in many NLP tasks.
Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation
Sequence-to-sequence models for open-domain dialogue generation tend to favor generic, uninformative responses.
Learning to Denoise Distantly-Labeled Data for Entity Typing
In this work, we propose a two-stage procedure for handling this type of data: denoise it with a learned model, then train our final model on clean and denoised distant data with standard supervised training.
Ranked #2 on
Entity Typing
on Ontonotes v5 (English)
Understanding Dataset Design Choices for Multi-hop Reasoning
First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets.
Tracking Discrete and Continuous Entity State for Process Understanding
The global discrete state structure is explicitly modeled with a neural CRF over the changing hidden representation of the entity.
Ranked #2 on
Procedural Text Understanding
on ProPara
Neural Extractive Text Summarization with Syntactic Compression
In this work, we present a neural model for single-document summarization based on joint extraction and syntactic compression.
Domain Agnostic Real-Valued Specificity Prediction
Sentence specificity quantifies the level of detail in a sentence, characterizing the organization of information in discourse.
Picking Apart Story Salads
During natural disasters and conflicts, information about what happened is often confusing, messy, and distributed across many sources.
Effective Use of Context in Noisy Entity Linking
To disambiguate between closely related concepts, entity linking systems need to effectively distill cues from their context, which may be quite noisy.
Spherical Latent Spaces for Stable Variational Autoencoders
A hallmark of variational autoencoders (VAEs) for text processing is their combination of powerful encoder-decoder models, such as LSTMs, with simple latent distributions, typically multivariate Gaussians.
Ranked #3 on
Topic Models
on AG News
Modeling Semantic Plausibility by Injecting World Knowledge
Distributional data tells us that a man can swallow candy, but not that a man can swallow a paintball, since this is never attested.
Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation
One weakness of machine-learned NLP models is that they typically perform poorly on out-of-domain data.
Capturing Semantic Similarity for Entity Linking with Convolutional Neural Networks
A key challenge in entity linking is making effective use of contextual information to disambiguate mentions that might refer to different entities in different contexts.
Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints
We present a discriminative model for single-document summarization that integrally combines compression and anaphoricity constraints.
Neural CRF Parsing
This paper describes a parsing model that combines the exact dynamic programming of CRF parsing with the rich nonlinear featurization of neural net approaches.
A Joint Model for Entity Analysis: Coreference, Typing, and Linking
We present a joint model of three core tasks in the entity analysis stack: coreference resolution (within-document clustering), named entity recognition (coarse semantic typing), and entity linking (matching to Wikipedia entities).
Ranked #28 on
Named Entity Recognition (NER)
on Ontonotes v5 (English)
