Search Results for author:
Found 97 papers, 65 papers with code
IST-Unbabel Participation in the WMT20 Quality Estimation Shared Task
We present the joint contribution of IST and Unbabel to the WMT 2020 Shared Task on Quality Estimation.
Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task
In this paper, we present the joint contribution of Unbabel and IST to the WMT 2021 Metrics Shared Task.
IST-Unbabel 2021 Submission for the Quality Estimation Shared Task
We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation.
Findings of the WMT 2021 Shared Task on Quality Estimation
We report the results of the WMT 2021 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels.
Findings of the WMT 2020 Shared Task on Quality Estimation
We report the results of the WMT20 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word, sentence and document levels.
Findings of the WMT 2020 Shared Task on Chat Translation
We report the results of the first edition of the WMT shared task on chat translation.
Document-level Neural MT: A Systematic Comparison
In this paper we provide a systematic comparison of existing and new document-level neural machine translation solutions.
QUARTZ: Quality-Aware Machine Translation
This paper presents QUARTZ, QUality-AwaRe machine Translation, a project led by Unbabel which aims at developing machine translation systems that are more robust and produce fewer critical errors.
Project MAIA: Multilingual AI Agent Assistant
This paper presents the Multilingual Artificial Intelligence Agent Assistant (MAIA), a project led by Unbabel with the collaboration of CMU, INESC-ID and IT Lisbon.
DeepSPIN: Deep Structured Prediction for Natural Language Processing
DeepSPIN is a research project funded by the European Research Council (ERC) whose goal is to develop new neural structured prediction methods, models, and algorithms for improving the quality, interpretability, and data-efficiency of natural language processing (NLP) systems, with special emphasis on machine translation and quality estimation applications.
EuroLLM: Multilingual Language Models for Europe
The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English.
Reranking Laws for Language Generation: A Communication-Theoretic Perspective
In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels.
DOCE: Finding the Sweet Spot for Execution-Based Code Generation
Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation.
How Effective are State Space Models for Machine Translation?
Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts.
A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e. g., machine translation, and general-purpose tasks, e. g., text classification.
xTower: A Multilingual LLM for Explaining and Correcting Translation Errors
While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies.
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
There is an increasing trend towards evaluating NLP models with LLM-generated judgments instead of human judgments.
QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
An important challenge in machine translation (MT) is to generate high-quality and diverse translations.
Can Automatic Metrics Assess High-Quality Translations?
Automatic metrics for evaluating translation quality are typically validated by measuring how well they correlate with human assessments.
XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples
Thus, we introduce XAMPLER: Cross-Lingual Example Retrieval, a method tailored to tackle the challenge of cross-lingual in-context learning using only annotated English data.
Conformal Prediction for Natural Language Processing: A Survey
The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications.
Is Context Helpful for Chat Translation Evaluation?
Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings.
Did Translation Models Get More Robust Without Anyone Even Noticing?
Neural machine translation (MT) models achieve strong results across a variety of settings, but it is widely believed that they are highly sensitive to "noisy" inputs, such as spelling errors, abbreviations, and other formatting issues.
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task.
Non-Exchangeable Conformal Language Generation with Nearest Neighbors
Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i. i. d.
CroissantLLM: A Truly Bilingual French-English Language Model
We introduce CroissantLLM, a 1. 3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware.
MaLA-500: Massive Language Adaptation of Large Language Models
Large language models (LLMs) have advanced the state of the art in natural language processing.
Aligning Neural Machine Translation Models: Human Feedback in Training and Inference
A core ingredient in RLHF's success in aligning and improving large language models (LLMs) is its reward model, trained using human feedback on model outputs.
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Experiments on 10 language pairs show that our proposed approach recovers the original few-shot capabilities while keeping the added benefits of finetuning.
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output.
xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection
Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level score.
Non-Exchangeable Conformal Risk Control
Split conformal prediction has recently sparked great interest due to its ability to provide formally guaranteed uncertainty sets or intervals for predictions made by black-box neural models, ensuring a predefined probability of containing the actual ground truth.
Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task
Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2).
The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems.
Conformalizing Machine Translation Evaluation
Several uncertainty estimation methods have been recently proposed for machine translation evaluation.
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Although neural-based machine translation evaluation metrics, such as COMET or BLEURT, have achieved strong correlations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered as critical errors, such as deviations in entities and numbers.
CREST: A Joint Framework for Rationalization and Counterfactual Text Generation
Selective rationales and counterfactual examples have emerged as two effective, complementary classes of interpretability methods for analyzing and training NLP models.
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining.
Open-Ended Question Answering
Zero-Shot Cross-Lingual Transfer
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i. e., making them better for about 100 languages.
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics
Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU.
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Many recent advances in natural language generation have been fueled by training large language models on internet-scale data.
Hallucinations in Large Multilingual Translation Models
Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages, making them increasingly appealing for real-world applications.
Discrete Latent Structure in Neural Networks
Many types of data from fields including natural language processing, computer vision, and bioinformatics, are well represented by discrete, compositional structures such as trees, sequences, or matchings.
Python Code Generation by Asking Clarification Questions
We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions.
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
Neural machine translation (NMT) has become the de-facto standard in real-world machine translation applications.
Improving abstractive summarization with energy-based re-ranking
Current abstractive summarization systems present important weaknesses which prevent their deployment in real-world applications, such as the omission of relevant information and the generation of factual inconsistencies (also known as hallucinations).
CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE).
Efficient Methods for Natural Language Processing: A Survey
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows.
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground.
Chunk-based Nearest Neighbor Machine Translation
Semi-parametric models, which augment generation with retrieval, have led to impressive results in language modeling and machine translation, due to their ability to retrieve fine-grained information from a datastore of examples.
Quality-Aware Decoding for Neural Machine Translation
Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the most probable translation according to the model (MAP decoding), approximated with beam search.
Efficient Machine Translation Domain Adaptation
On the other hand, semi-parametric models have been shown to successfully perform domain adaptation by retrieving examples from an in-domain datastore (Khandelwal et al., 2021).
Learning to Scaffold: Optimizing Model Explanations for Teaching
In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model.
Disentangling Uncertainty in Machine Translation Evaluation
Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data.
Differentiable Causal Discovery Under Latent Interventions
Recent work has shown promising results in causal discovery by leveraging interventional data with gradient-based methods, even when the intervened variables are unknown.
Modeling Structure with Undirected Neural Networks
In this paper, we combine the representational strengths of factor graphs and of neural networks, proposing undirected neural networks (UNNs): a flexible framework for specifying computations that can be performed in any order.
Predicting Attention Sparsity in Transformers
Transformers' quadratic complexity with respect to the input sequence length has motivated a body of work on efficient sparse approximations to softmax.
When Does Translation Require Context? A Data-driven, Multilingual Exploration
Although proper handling of discourse significantly contributes to the quality of machine translation (MT), these improvements are not adequately measured in common translation quality metrics.
Uncertainty-Aware Machine Translation Evaluation
Several neural-based metrics have been recently proposed to evaluate machine translation quality.
SPECTRA: Sparse Structured Text Rationalization
Selective rationalization aims to produce decisions along with rationales (e. g., text highlights or word alignments between two sentences).
$\infty$-former: Infinite Memory Transformer
Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length.
Ranked #1 on
Dialogue Generation
on CMU-DoG
Sparse Communication via Mixed Distributions
Neural networks and other machine learning models compute continuous representations, while humans communicate mostly through discrete symbols.
Sparse Continuous Distributions and Fenchel-Young Losses
In contrast, for finite domains, recent work on sparse alternatives to softmax (e. g., sparsemax, $\alpha$-entmax, and fusedmax), has led to distributions with varying support.
Do Context-Aware Translation Models Pay the Right Attention?
Are models paying large amounts of attention to the same context?
Measuring and Increasing Context Usage in Context-Aware Machine Translation
Recent work in neural machine translation has demonstrated both the necessity and feasibility of using inter-sentential context -- context from sentences other than those currently being translated.
Multimodal Continuous Visual Attention Mechanisms
Visual attention mechanisms are a key component of neural network models for computer vision.
Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication
Neural networks and other machine learning models compute continuous representations, while humans communicate with discrete symbols.
Smoothing and Shrinking the Sparse Seq2Seq Search Space
Current sequence-to-sequence models are trained to minimize cross-entropy and use softmax to compute the locally normalized probabilities over target sequences.
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset
We present MLQE-PE, a new dataset for Machine Translation (MT) Quality Estimation (QE) and Automatic Post-Editing (APE).
Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning
Latent structure models are a powerful tool for modeling language data: they can mitigate the error propagation and annotation bottleneck in pipeline systems, while simultaneously uncovering linguistic insights about the data.
Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
In this paper, we propose a new training strategy which replaces these estimators by an exact yet efficient marginalization.
Sparse and Continuous Attention Mechanisms
Exponential families are widely used in machine learning; they include many distributions in continuous and discrete domains (e. g., Gaussian, Dirichlet, Poisson, and categorical distributions via the softmax transformation).
Ranked #36 on
Visual Question Answering (VQA)
on VQA v2 test-std
The Explanation Game: Towards Prediction Explainability through Sparse Communication
Explainability is a topic of growing importance in NLP.
Sparse Text Generation
Current state-of-the-art text generators build on powerful language models such as GPT-2, achieving impressive performance.
LP-SparseMAP: Differentiable Relaxed Optimization for Sparse Structured Prediction
Structured prediction requires manipulating a large number of combinatorial structures, e. g., dependency trees or alignments, either as latent or output variables.
Adaptively Sparse Transformers
Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers.
Ranked #1 on
Machine Translation
on IWSLT2017 German-English
Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task
We present the contribution of the Unbabel team to the WMT 2019 Shared Task on Quality Estimation.
Notes on Latent Structure Models and SPIGOT
These notes aim to shed light on the recently proposed structured projected intermediate gradient optimization technique (SPIGOT, Peng et al., 2018).
Translator2Vec: Understanding and Representing Human Post-Editors
The combination of machines and humans for translation is effective, with many studies showing productivity gains when humans post-edit machine-translated output instead of translating from scratch.
Joint Learning of Named Entity Recognition and Entity Linking
Named entity recognition (NER) and entity linking (EL) are two fundamentally related tasks, since in order to perform EL, first the mentions to entities have to be detected.
Ranked #12 on
Entity Linking
on AIDA-CoNLL
Scheduled Sampling for Transformers
In the Transformer model, unlike the RNN, the generation of a new word attends to the full sentence generated so far, not only to the last word, and it is not straightforward to apply the scheduled sampling technique.
A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning
Automatic post-editing (APE) seeks to automatically refine the output of a black-box machine translation (MT) system through human post-edits.
Unbabel's Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing
Analogously to dual-encoder architectures we develop a BERT-based encoder-decoder (BED) model in which a single pretrained BERT encoder receives both the source src and machine translation tgt strings.
Jointly Extracting and Compressing Documents with Summary State Representations
We present a new neural model for text summarization that first extracts sentences from a document and then compresses them.
Selective Attention for Context-aware Neural Machine Translation
Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document.
Learning with Fenchel-Young Losses
Over the past decades, numerous loss functions have been been proposed for a variety of supervised learning tasks, including regression, classification, ranking, and more generally structured prediction.
Towards Dynamic Computation Graphs via Sparse Latent Structure
Deep NLP models benefit from underlying structures in the data---e. g., parse trees---typically extracted using off-the-shelf parsers.
Contextual Neural Model for Translating Bilingual Multi-Speaker Conversations
In this work, we propose the task of translating Bilingual Multi-Speaker Conversations, and explore neural architectures which exploit both source and target-side conversation histories for this task.
Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms
This paper studies Fenchel-Young losses, a generic way to construct convex loss functions from a regularization function.
Sparse and Constrained Attention for Neural Machine Translation
In NMT, words are sometimes dropped from the source or generated repeatedly in the translation.
Marian: Fast Neural Machine Translation in C++
We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs.
SparseMAP: Differentiable Sparse Structured Inference
Structured prediction requires searching over a combinatorial number of structures.
From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities.
