Search Results for author:
Found 20 papers, 15 papers with code
Contemporary NLP Modeling in Six Comprehensive Programming Assignments
We present a series of programming assignments, adaptable to a range of experience levels from advanced undergraduate to PhD, to teach students design and implementation of modern NLP systems.
FABLES: Evaluating faithfulness and content selection in book-length summarization
While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims.
Evaluating Large Language Models at Evaluating Instruction Following
As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models.
A Long Way to Go: Investigating Length Correlations in RLHF
Furthermore, we find that even running RLHF with a reward based solely on length can reproduce most of the downstream improvements over the initial policy model, showing that reward models in these settings have a long way to go.
BooookScore: A systematic exploration of book-length summarization in the era of LLMs
We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models.
WiCE: Real-World Entailment for Claims in Wikipedia
Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation.
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Despite recent progress in abstractive summarization, models often generate summaries with factual errors.
News Summarization and Evaluation in the Era of GPT-3
Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting.
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
SNaC: Coherence Error Detection for Narrative Summarization
In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on.
Training Dynamics for Text Summarization Models
Across different datasets (CNN/DM, XSum, MediaSum) and summary properties, such as abstractiveness and hallucination, we study what the model learns at different stages of its fine-tuning process.
HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Summarization systems make numerous "decisions" about summary properties during inference, e. g. degree of copying, specificity and length of outputs, etc.
HydraSum - Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs.
Annotating and Modeling Fine-grained Factuality in Summarization
Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors.
Evaluating Factuality in Generation with Dependency-level Entailment
Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.
Neural Syntactic Preordering for Controlled Paraphrase Generation
Paraphrasing natural language sentences is a multifaceted process: it might involve replacing individual words or short phrases, local rearrangement of content, or high-level restructuring like topicalization or passivization.
Embedding time expressions for deep temporal ordering models
Data-driven models have demonstrated state-of-the-art performance in inferring the temporal ordering of events in text.
Frustrated, Polite, or Formal: Quantifying Feelings and Tone in Email
We present a novel approach to model human frustration in text.
An Empirical Analysis of Edit Importance between Document Versions
In this paper, we present a novel approach to infer significance of various textual edits to documents.
