Search Results for author:
Found 12 papers, 3 papers with code
SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation
Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task.
Reward Gaming in Conditional Text Generation
To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations.
Dialect-robust Evaluation of Generated Text
Evaluation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects.
SQuId: Measuring Speech Naturalness in Many Languages
The main insight is that training one model on many locales consistently outperforms mono-locale baselines.
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations.
Learning Compact Metrics for MT
Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT.
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem.
BLEURT: Learning Robust Metrics for Text Generation
We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples.
A Multilingual View of Unsupervised Machine Translation
We present a probabilistic framework for multilingual neural machine translation that encompasses supervised and unsupervised setups, focusing on unsupervised translation.
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
We address the issue of hallucination in data-to-text generation, i. e., reducing the generation of text that is unsupported by the source.
