Search Results for author:
Found 200 papers, 41 papers with code
deepQuest-py: Large and Distilled Models for Quality Estimation
We introduce deepQuest-py, a framework for training and evaluation of large and light-weight models for Quality Estimation (QE).
Validating Quality Estimation in a Computer-Aided Translation Workflow: Speed, Cost and Quality Trade-off
In this scenario, a Quality Estimation (QE) tool can be used to score MT outputs, and a threshold on the QE scores can be applied to decide whether an MT output can be used as-is or requires human post-edition.
The IWSLT 2019 Evaluation Campaign
The IWSLT 2019 evaluation campaign featured three tasks: speech translation of (i) TED talks and (ii) How2 instructional videos from English into German and Portuguese, and (iii) text translation of TED talks from English into Czech.
The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification
Second, we conduct the first meta-evaluation of automatic metrics in Text Simplification, using our new data set (and other existing data) to analyze the variation of the correlation between metrics’ scores and human judgments across three dimensions: the perceived simplicity level, the system type, and the set of references used for computation.
Predicting Moments of Mood Changes Overtime from Imbalanced Social Media Data
The second is predicting the degree of suicide risk as a user-level classification task.
ICL’s Submission to the WMT21 Critical Error Detection Shared Task
This paper presents Imperial College London’s submissions to the WMT21 Quality Estimation (QE) Shared Task 3: Critical Error Detection.
Findings of the WMT 2021 Shared Task on Quality Estimation
We report the results of the WMT 2021 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels.
Quality In, Quality Out: Learning from Actual Mistakes
However, QE models are often trained on noisy approximations of quality annotations derived from the proportion of post-edited words in translated sentences instead of direct human annotations of translation errors.
Bayesian Model-Agnostic Meta-Learning with Matrix-Valued Kernels for Quality Estimation
To address these challenges, we propose a Bayesian meta-learning approach for adapting QE models to the needs and preferences of each user with limited supervision.
Leveraging Pre-trained Language Models for Gender Debiasing
This work explores a light-weight method to generate gender variants for a given text using pre-trained language models as the resource, without any task-specific labelled data.
Multilingual and Multimodal Learning for Brazilian Portuguese
The good BLEU and METEOR values obtained for this new language pair, regarding the original English-German VTLM, establish the suitability of the model to other languages.
A Generative Framework for Simultaneous Machine Translation
We propose a generative framework for simultaneous machine translation.
A Taxonomy and Study of Critical Errors in Machine Translation
We also study the impact of the source text on generating critical errors in the translation and, based on this, propose a set of recommendations on aspects of the MT that need further scrutiny, especially for user-generated content, to avoid generating such errors, and hence improve online communication.
Bias Mitigation in Machine Translation Quality Estimation
Machine Translation Quality Estimation (QE) aims to build predictive models to assess the quality of machine-generated translations in the absence of reference translations.
Towards a Better Understanding of Noise in Natural Language Processing
In this paper, we propose a definition and taxonomy of various types of non-standard textual content – generally referred to as “noise” – in Natural Language Processing (NLP).
Multimodal Simultaneous Machine Translation
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible.
BERGAMOT-LATTE Submissions for the WMT20 Quality Estimation Shared Task
We explore (a) a black-box approach to QE based on pre-trained representations; and (b) glass-box approaches that leverage various indicators that can be extracted from the neural MT systems.
Findings of the WMT 2020 Shared Task on Quality Estimation
We report the results of the WMT20 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word, sentence and document levels.
Findings of the WMT 2020 Shared Task on Machine Translation Robustness
We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT).
DiffuseDef: Improved Robustness to Adversarial Attacks
Pretrained language models have significantly advanced performance across various natural language processing tasks.
From Understanding to Utilization: A Survey on Explainability for Large Language Models
This survey underscores the imperative for increased explainability in LLMs, delving into both the research on explainability and the various methodologies and tasks that utilize an understanding of these models.
Reducing Hallucinations in Neural Machine Translation with Feature Attribution
We present a case study focusing on model understanding and regularisation to reduce hallucinations in NMT.
Scene Text Recognition with Semantics
Scene Text Recognition (STR) models have achieved high performance in recent years on benchmark datasets where text images are presented with minimal noise.
Contrastive Video-Language Learning with Fine-grained Frame Sampling
However, even in paired video-text segments, only a subset of the frames are semantically relevant to the corresponding text, with the remainder representing noise; where the ratio of noisy frames is higher for longer videos.
Burst2Vec: An Adversarial Multi-Task Approach for Predicting Emotion, Age, and Origin from Vocal Bursts
We present Burst2Vec, our multi-task learning approach to predict emotion, age, and origin (i. e., native country/language) from vocal bursts.
Logically Consistent Adversarial Attacks for Soft Theorem Provers
We propose a novel, generative adversarial framework for probing and improving these models' reasoning capabilities.
Supervised Visual Attention for Simultaneous Multimodal Machine Translation
A particular use for such multimodal systems is the task of simultaneous machine translation, where visual context has been shown to complement the partial information provided by the source sentence, especially in the early phases of translation.
Revisiting Contextual Toxicity Detection in Conversations
We then propose to bring these findings into computational detection models by introducing and evaluating (a) neural architectures for contextual toxicity detection that are aware of the conversational structure, and (b) data augmentation strategies that can help model contextual toxicity detection.
Ranked #1 on
Toxic Comment Classification
on CAD
Guiding Visual Question Generation
In traditional Visual Question Generation (VQG), most images have multiple concepts (e. g. objects and categories) for which a question could be generated, but models are trained to mimic an arbitrary choice of concept as given in their training data.
Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation
However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements.
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels.
A Survey of Online Hate Speech through the Causal Lens
The societal issue of digital hostility has previously attracted a lot of attention.
Translation Error Detection as Rationale Extraction
Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences.
Continual Quality Estimation with Online Bayesian Meta-Learning
Most current quality estimation (QE) models for machine translation are trained and evaluated in a static setting where training and test data are assumed to be from a fixed distribution.
Knowledge Distillation for Quality Estimation
Quality Estimation (QE) is the task of automatically predicting Machine Translation quality in the absence of reference translations, making it applicable in real-time settings, such as translating online social media conversations.
BERTGEN: Multi-task Generation through BERT
We present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pretrained models VL-BERT and M-BERT, respectively.
SentSim: Crosslingual Semantic Evaluation of Machine Translation
Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data.
Cross-Modal Generative Augmentation for Visual Question Answering
Experiments on Visual Question Answering as downstream task demonstrate the effectiveness of the proposed generative model, which is able to improve strong UpDn-based models to achieve state-of-the-art performance.
What Makes a Scientific Paper be Accepted for Publication?
Despite peer-reviewing being an essential component of academia since the 1600s, it has repeatedly received criticisms for lack of transparency and consistency.
Backtranslation Feedback Improves User Confidence in MT, Not Quality
Translating text into a language unknown to the text's author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility.
Visual Cues and Error Correction for Translation Robustness
Neural Machine Translation models are sensitive to noise in the input texts, such as misspelled words and ungrammatical constructions.
MultiSubs: A Large-scale Multimodal and Multilingual Dataset
The dataset will benefit research on visual grounding of words especially in the context of free-form sentences, and can be obtained from https://doi. org/10. 5281/zenodo. 5034604 under a Creative Commons licence.
Ranked #1 on
Multimodal Text Prediction
on MultiSubs
Multimodal Lexical Translation
Multimodal Text Prediction
+2
Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation
This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced.
Exploring Supervised and Unsupervised Rewards in Machine Translation
Reinforcement Learning (RL) is a powerful framework to address the discrepancy between loss functions used during training and the final evaluation metrics to be used at test time.
Quality Estimation without Human-labeled Data
Quality estimation aims to measure the quality of translated content without access to a reference translation.
Cross-lingual Visual Pre-training for Multimodal Machine Translation
Pre-trained language models have been shown to improve performance in many natural language tasks substantially.
Latent Variable Models for Visual Question Answering
Current work on Visual Question Answering (VQA) explore deterministic approaches conditioned on various types of image and question features.
MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
We hope that the MSVD-Turkish dataset and the results reported in this work will lead to better video captioning and multimodal machine translation models for Turkish and other morphology rich and agglutinative languages.
An Exploratory Study on Multilingual Quality Estimation
Predicting the quality of machine translation has traditionally been addressed with language-specific models, under the assumption that the quality label distribution or linguistic features exhibit traits that are not shared across languages.
Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision
In this paper, we teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Automatic evaluation of language generation systems is a well-studied problem in Natural Language Processing.
FIND: Human-in-the-Loop Debugging Deep Text Classifiers
Since obtaining a perfect training dataset (i. e., a dataset which is considerably large, unbiased, and well-representative of unseen cases) is hardly possible, many real-world text classifiers are trained on the available, yet imperfect, datasets.
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset
We present MLQE-PE, a new dataset for Machine Translation (MT) Quality Estimation (QE) and Automatic Post-Editing (APE).
Simultaneous Machine Translation with Visual Context
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible.
Exploring Model Consensus to Generate Translation Paraphrases
This paper describes our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE).
Are we Estimating or Guesstimating Translation Quality?
Recent advances in pre-trained multilingual language models lead to state-of-the-art results on the task of quality estimation (QE) for machine translation.
Multimodal Quality Estimation for Machine Translation
We propose approaches to Quality Estimation (QE) for Machine Translation that explore both text and visual modalities for Multimodal QE.
Multi-Hypothesis Machine Translation Evaluation
Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem.
Unsupervised Quality Estimation for Neural Machine Translation
Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time.
ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations
Furthermore, we motivate the need for developing better methods for automatic evaluation using ASSET, since we show that current popular metrics may not be suitable when multiple simplification transformations are performed.
A Post-Editing Dataset in the Legal Domain: Do we Underestimate Neural Machine Translation Quality?
We introduce a machine translation dataset for three pairs of languages in the legal domain with post-edited high-quality neural machine translation and independent human references.
Data-Driven Sentence Simplification: Survey and Benchmark
Sentence Simplification (SS) aims to modify a sentence in order to make it easier to read and understand.
Multimodal Machine Translation through Visuals and Speech
Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data.
Ranked #4 on
Multimodal Machine Translation
on Multi30K
Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back-Translation
Neural Machine Translation (NMT) models have been proved strong when translating clean texts, but they are very sensitive to noise in the input.
Deep Copycat Networks for Text-to-Text Generation
Most text-to-text generation tasks, for example text summarisation and text simplification, require copying words from the input to the output.
Transformer-based Cascaded Multimodal Speech Translation
Upon conducting extensive experiments, we found that (i) the explored visual integration schemes often harm the translation performance for the transformer and additive deliberation, but considerably improve the cascade deliberation; (ii) the transformer and cascade deliberation integrate the visual modality better than the additive deliberation, as shown by the incongruence analysis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Imperial College London Submission to VATEX Video Captioning Task
This paper describes the Imperial College London team's submission to the 2019' VATEX video captioning challenge, where we first explore two sequence-to-sequence models, namely a recurrent (GRU) model and a transformer model, which generate captions from the I3D action features.
Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality
To that end, we report experiments on a dataset with newly-collected post-editing indicators and show their usefulness when estimating post-editing effort.
Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back Translation
Neural Machine Translation (NMT) models have been proved strong when translating clean texts, but they are very sensitive to noise in the input.
Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments
Much work has been dedicated to the improvement of evaluation metrics to achieve a higher correlation with human judgments.
Phrase Localization Without Paired Training Examples
Localizing phrases in images is an important part of image understanding and can be useful in many applications that require mappings between textual and visual information.
EASSE: Easier Automatic Sentence Simplification Evaluation
We introduce EASSE, a Python package aiming to facilitate and standardise automatic evaluation and comparison of Sentence Simplification (SS) systems.
Predicting Actions to Help Predict Translations
The question we ask ourselves is whether visual features can support the translation process, in particular, given that this is a dataset extracted from videos, we focus on the translation of actions, which we believe are poorly captured in current static image-text datasets currently used for multimodal translation.
A Comparison on Fine-grained Pre-trained Embeddings for the WMT19Chinese-English News Translation Task
This paper describes our submission to the WMT 2019 Chinese-English (zh-en) news translation shared task.
WMDO: Fluency-based Word Mover's Distance for Machine Translation Evaluation
We propose WMDO, a metric based on distance between distributions in the semantic vector space.
Cross-Sentence Transformations in Text Simplification
Current approaches to Text Simplification focus on simplifying sentences individually.
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions
It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description.
Is artificial data useful for biomedical Natural Language Processing algorithms?
A major obstacle to the development of Natural Language Processing (NLP) methods in the biomedical domain is data accessibility.
Distilling Translations with Visual Awareness
Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.
Ranked #3 on
Multimodal Machine Translation
on Multi30K
(Meteor (EN-FR) metric)
Grounded Word Sense Translation
Recent work on visually grounded language learning has focused on broader applications of grounded representations, such as visual question answering and multimodal machine translation.
Grounded language learning
Multimodal Machine Translation
+3
Probing the Need for Visual Context in Multimodal Machine Translation
Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial.
End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn {`}distributional similarity{'} in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
How2: A Large-scale Dataset for Multimodal Language Understanding
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Assessing Crosslingual Discourse Relations in Machine Translation
In an attempt to improve overall translation quality, there has been an increasing focus on integrating more linguistic elements into Machine Translation (MT).
Findings of the Third Shared Task on Multimodal Machine Translation
In this task a source sentence in English is supplemented by an image and participating systems are required to generate a translation for such a sentence into German, French or Czech.
Sheffield Submissions for the WMT18 Quality Estimation Shared Task
In this paper we present the University of Sheffield submissions for the WMT18 Quality Estimation shared task.
Sheffield Submissions for WMT18 Multimodal Translation Shared Task
For task 1b, we explore three approaches: (i) re-ranking based on cross-lingual word sense disambiguation (as for task 1), (ii) re-ranking based on consensus of NMT n-best lists from German-Czech, French-Czech and English-Czech systems, and (iii) data augmentation by generating English source data through machine translation from French to English and from German to English followed by hypothesis selection using a multimodal-reranker.
Findings of the WMT 2018 Shared Task on Quality Estimation
We report the results of the WMT18 shared task on Quality Estimation, i. e. the task of predicting the quality of the output of machine translation systems at various granularity levels: word, phrase, sentence and document.
End-to-end Image Captioning Exploits Multimodal Distributional Similarity
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn `distributional similarity' in a multimodal feature space by mapping a test image to similar training images in this space and generating a caption from the same space.
Exploring Gap Filling as a Cheaper Alternative to Reading Comprehension Questionnaires when Evaluating Machine Translation for Gisting
A popular application of machine translation (MT) is gisting: MT is consumed as is to make sense of text in a foreign language.
deepQuest: A Framework for Neural-based Quality Estimation
Our approach is significantly faster and yields performance improvements for a range of document-level quality estimation tasks.
Learning Simplifications for Specific Target Audiences
Text simplification (TS) is a monolingual text-to-text transformation task where an original (complex) text is transformed into a target (simpler) text.
Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output
Machine Translation systems are usually evaluated and compared using automated evaluation metrics such as BLEU and METEOR to score the generated translations against human translations.
Defoiling Foiled Image Captions
We address the task of detecting foiled image captions, i. e. identifying whether a caption contains a word that has been deliberately replaced by a semantically similar word, thus rendering it inaccurate with respect to the image being described.
A Report on the Complex Word Identification Shared Task 2018
We report the findings of the second Complex Word Identification (CWI) shared task organized as part of the BEA workshop co-located with NAACL-HLT'2018.
Object Counts! Bringing Explicit Detections Back into Image Captioning
The use of explicit object detectors as an intermediate step to image captioning - which used to constitute an essential stage in early work - is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding.
What is image captioning made of?
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
MUSST: A Multilingual Syntactic Simplification Tool
Our implementation includes a set of general-purpose simplification rules, as well as a sentence selection module (to select sentences to be simplified) and a confidence model (to select only promising simplifications).
MASSAlign: Alignment and Annotation of Comparable Documents
We introduce MASSAlign: a Python library for the alignment and annotation of monolingual comparable documents.
Learning How to Simplify From Explicit Labeling of Complex-Simplified Text Pairs
Current research in text simplification has been hampered by two central problems: (i) the small amount of high-quality parallel simplification data available, and (ii) the lack of explicit annotations of simplification operations, such as deletions or substitutions, on existing data.
Ranked #8 on
Text Simplification
on PWKP / WikiSmall
(SARI metric)
The Ultimate Presentation Makeup Tutorial: How to Polish your Posters, Slides and Presentations Skills
There is no question that our research community have, and still has been producing an insurmountable amount of interesting strategies, models and tools to a wide array of problems and challenges in diverse areas of knowledge.
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description
The multilingual image description task was changed such that at test time, only the image is given.
Complex Word Identification: Challenges in Data Annotation and System Performance
This paper revisits the problem of complex word identification (CWI) following up the SemEval CWI shared task.
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Semantic Textual Similarity (STS) measures the meaning similarity of sentences.
SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
Semantic Textual Similarity (STS) measures the meaning similarity of sentences.
Lexical Simplification with Neural Ranking
We present a new Lexical Simplification approach that exploits Neural Networks to learn substitutions from the Newsela corpus - a large set of professionally produced simplifications.
Vicinity-Driven Paragraph and Sentence Alignment for Comparable Corpora
Parallel corpora have driven great progress in the field of Text Simplification.
Anita: An Intelligent Text Adaptation Tool
We introduce Anita: a flexible and intelligent Text Adaptation tool for web content that provides Text Simplification and Text Enhancement modules.
Quality Estimation for Language Output Applications
The goal of QE is to estimate the quality of language output applications without the need of human references.
Understanding the Lexical Simplification Needs of Non-Native Speakers of English
We report three user studies in which the Lexical Simplification needs of non-native English speakers are investigated.
Collecting and Exploring Everyday Language for Predicting Psycholinguistic Properties of Words
Exploring language usage through frequency analysis in large corpora is a defining feature in most recent work in corpus and computational linguistics.
Personalized Machine Translation: Preserving Original Author Traits
The language that we produce reflects our personality, and various personal and demographic characteristics can be detected in natural language texts.
Exploring Prediction Uncertainty in Machine Translation Quality Estimation
Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation environments.
Multi30K: Multilingual English-German Image Descriptions
We introduce the Multi30K dataset to stimulate multilingual multimodal research.
Cohere: A Toolkit for Local Coherence
We describe COHERE, our coherence toolkit which incorporates various complementary models for capturing and measuring different aspects of text coherence.
Benchmarking Lexical Simplification Systems
Lexical Simplification is the task of replacing complex words in a text with simpler alternatives.
MARMOT: A Toolkit for Translation Quality Estimation at the Word Level
The tool has a set of state-of-the-art features for QE, and new features can easily be added.
A Reading Comprehension Corpus for Machine Translation Evaluation
Effectively assessing Natural Language Processing output tasks is a challenge for research in the area.
Phrase Level Segmentation and Labelling of Machine Translation Errors
This paper presents our work towards a novel approach for Quality Estimation (QE) of machine translation based on sequences of adjacent words, the so-called phrases.