Search Results for author:
Found 22 papers, 14 papers with code
Findings of the WMT 2021 Shared Task on Quality Estimation
We report the results of the WMT 2021 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels.
IST-Unbabel 2021 Submission for the Quality Estimation Shared Task
We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation.
Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task
In this paper, we present the joint contribution of Unbabel and IST to the WMT 2021 Metrics Shared Task.
Rejected Dialects: Biases Against African American Language in Reward Models
Preference alignment via reward models helps build safe, helpful, and reliable large language models (LLMs).
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?
The evaluation of image captions, looking at both linguistic fluency and semantic correspondence to visual contents, has witnessed a significant effort.
"I Never Said That": A dataset, taxonomy and baselines on response clarity classification
Equivocation and ambiguity in public speech are well-studied discourse phenomena, especially in political science and analysis of political interviews.
Conformal Prediction for Natural Language Processing: A Survey
The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications.
Non-Exchangeable Conformal Language Generation with Nearest Neighbors
Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i. i. d.
Context-aware Neural Machine Translation for English-Japanese Business Scene Dialogues
We make use of Conditional Cross-Mutual Information (CXMI) to explore how much of the context the model uses and generalise CXMI to study the impact of the extra-sentential context.
Non-Exchangeable Conformal Risk Control
Split conformal prediction has recently sparked great interest due to its ability to provide formally guaranteed uncertainty sets or intervals for predictions made by black-box neural models, ensuring a predefined probability of containing the actual ground truth.
Uncertainty in Natural Language Generation: From Theory to Applications
Recent advances of powerful Language Models have allowed Natural Language Generation (NLG) to emerge as an important technology that can not only perform traditional tasks like summarisation or translation, but also serve as a natural language interface to a variety of applications.
Conformalizing Machine Translation Evaluation
Several uncertainty estimation methods have been recently proposed for machine translation evaluation.
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Although neural-based machine translation evaluation metrics, such as COMET or BLEURT, have achieved strong correlations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered as critical errors, such as deviations in entities and numbers.
Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors
We propose a new back translation-inspired evaluation methodology that utilises earlier outputs of the explainer as ground truth proxies to investigate the consistency of explainers.
CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE).
Disentangling Uncertainty in Machine Translation Evaluation
Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data.
Learning Disentangled Representations of Negation and Uncertainty
Negation and uncertainty modeling are long-standing tasks in natural language processing.
Uncertainty-Aware Machine Translation Evaluation
Several neural-based metrics have been recently proposed to evaluate machine translation quality.
Semantic Enrichment of Pretrained Embedding Output for Unsupervised IR
The rapid growth of scientific literature in the biomedical and clinical domain has significantly com- plicated the identification of information of interest by researchers as well as other practitioners.
Ranked #1 on
Information Retrieval
on Ohsumed
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset
We present MLQE-PE, a new dataset for Machine Translation (MT) Quality Estimation (QE) and Automatic Post-Editing (APE).
Paths for uncertainty: Exploring the intricacies of uncertainty identification for news
We compare the differences in the definition and expression of uncertainty between a scientific domain, i. e., biomedicine, and newswire.
