Search Results for author:
Found 44 papers, 9 papers with code
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain
Contrary to previous years’ editions, this year we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM).
A Natural Diet: Towards Improving Naturalness of Machine Translation Output
In this work we propose a method for training MT systems to achieve a more natural style, i. e. mirroring the style of text originally written in the target language.
Findings of the WMT 2020 Shared Task on Automatic Post-Editing
Due to i) the different source/domain of data compared to the past (Wikipedia vs Information Technology), ii) the different quality of the initial translations to be corrected and iii) the introduction of a new language pair (English-Chinese), this year’s results are not directly comparable with last year’s round.
Results of the WMT20 Metrics Shared Task
Participants were asked to score the outputs of the translation systems competing in the WMT20 News Translation Task with automatic metrics.
Findings of the 2021 Conference on Machine Translation (WMT21)
This paper presents the results of the newstranslation task, the multilingual low-resourcetranslation for Indo-European languages, thetriangular translation task, and the automaticpost-editing task organised as part of the Con-ference on Machine Translation (WMT) 2021. In the news task, participants were asked tobuild machine translation systems for any of10 language pairs, to be evaluated on test setsconsisting mainly of news stories.
Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation
Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations.
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback
In particular, since the advent of neural metrics, like COMET, BLEURT, and SEScore2, the newest generation of metrics show a high correlation with human judgment.
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Scaling Laws for Multilingual Neural Machine Translation
Through a novel joint scaling law formulation, we compute the effective number of parameters allocated to each language pair and examine the role of language similarity in the scaling behavior of our models.
Prompting PaLM for Translation: Assessing Strategies and Performance
Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages.
Language Models are Multilingual Chain-of-Thought Reasoners
Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment.
Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance
We show that these two factors have a large causal effect on the MT performance, in addition to the test-model direction mismatch highlighted by existing work on the impact of translationese.
Toward More Effective Human Evaluation for Machine Translation
Improvements in text generation technologies such as machine translation have necessitated more costly and time-consuming human evaluation procedures to ensure an accurate signal.
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans.
Scaling Laws for Neural Machine Translation
We show that cross-entropy loss as a function of model size follows a certain scaling law.
Using Machine Translation to Localize Task Oriented NLG Output
One of the challenges in a task oriented natural language application like the Google Assistant, Siri, or Alexa is to localize the output to many languages.
On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions.
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions.
Assessing Reference-Free Peer Evaluation for Machine Translation
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains.
Complete Multilingual Neural Machine Translation
We reintroduce this direct parallel data from multi-way aligned corpora between all source and target languages.
Human-Paraphrased References Improve Neural Machine Translation
When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment.
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem.
KoBE: Knowledge-Based Machine Translation Evaluation
Our approach achieves the highest correlation with human judgements on 9 out of the 18 language pairs from the WMT19 benchmark for evaluation without references, which is the largest number of wins for a single evaluation method on this task.
Translationese as a Language in ``Multilingual'' NMT
Machine translation has an undesirable propensity to produce {``}translationese{''} artifacts, which can lead to higher BLEU scores while being liked less by human raters.
BLEU might be Guilty but References are not Innocent
The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems.
Translationese as a Language in "Multilingual" NMT
Machine translation has an undesirable propensity to produce "translationese" artifacts, which can lead to higher BLEU scores while being liked less by human raters.
APE at Scale and its Implications on MT Evaluation Biases
In this work, we train an Automatic Post-Editing (APE) model and use it to reveal biases in standard Machine Translation (MT) evaluation procedures.
Unsupervised Natural Language Generation with Denoising Autoencoders
Generating text from structured data is important for various tasks such as question answering and dialog systems.
Attention-based Vocabulary Selection for NMT Decoding
Usually, the candidate lists are a combination of external word-to-word aligner, phrase table entries or most frequent words.
Local System Voting Feature for Machine Translation System Combination
In this paper, we enhance the traditional confusion network system combination approach with an additional model trained by a neural network.
Ensemble Distillation for Neural Machine Translation
Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network.
Beam Search Strategies for Neural Machine Translation
In this paper, we concentrate on speeding up the decoder by applying a more flexible beam search strategy whose candidate size may vary at each time step depending on the candidate scores.
Fast Domain Adaptation for Neural Machine Translation
The basic concept in NMT is to train a large Neural Network that maximizes the translation performance on a given parallel corpus.
