Search Results for author:
Found 24 papers, 10 papers with code
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries
Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference.
On the Role of Summary Content Units in Text Summarization Evaluation
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs).
Finding Replicable Human Evaluations via Stable Ranking Probability
Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult.
LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback
Recent large language models (LLM) are leveraging human feedback to improve their generation quality.
There's no Data Like Better Data: Using QE Metrics for MT Data Filtering
Quality Estimation (QE), the evaluation of machine translation output without the need of explicit references, has seen big improvements in the last years with the use of neural metrics.
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Specifically, we propose a novel competition setting in which we select a list of allowed LLMs and disallow fine-tuning to ensure a focus on prompting.
Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level
As research on machine translation moves to translating text beyond the sentence level, it remains unclear how effective automatic evaluation metrics are at scoring longer translations.
The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems.
Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration
Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations.
Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization.
On the Limitations of Reference-Free Evaluations of Generated Text
There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely unavailable in online applications.
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code
We introduce Repro, an open-source library which aims at improving the reproducibility and usability of research code.
Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics
Question answering-based summarization evaluation metrics must automatically determine whether the QA model's prediction is correct or not, a task known as answer verification.
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations.
Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selection
In this work, we propose a method for incorporating question-answering (QA) signals into a summarization model.
A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods
After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations.
Is Killed More Significant than Fled? A Contextual Model for Salient Event Detection
Identifying the key events in a document is critical to holistically understanding its important information.
Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries
Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference.
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference.
SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics
We present SacreROUGE, an open-source library for using and developing summarization evaluation metrics.
A General-Purpose Algorithm for Constrained Sequential Inference
We experimentally show the benefits of our algorithm on constituency parsing and semantic role labeling.
Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization
A key challenge in topic-focused summarization is determining what information should be included in the summary, a problem known as content selection.
A Distributional and Orthographic Aggregation Model for English Derivational Morphology
Modeling derivational morphology to generate words with particular semantics is useful in many text generation tasks, such as machine translation or abstractive question answering.
