Search Results for author:
Found 24 papers, 7 papers with code
Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets
We benchmark metric performance, assess their incremental performance over successive campaigns, and measure their sensitivity to a range of linguistic phenomena.
ACES: Translation Accuracy Challenge Sets at WMT 2023
We benchmark the performance of segmentlevel metrics submitted to WMT 2023 using the ACES Challenge Set (Amrhein et al., 2022).
MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue
We use MULTI3NLU++ to benchmark state-of-the-art multilingual models for the NLU tasks of intent detection and slot labelling for TOD systems in the multilingual setting.
ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics
As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level.
Ranked #1 on
Machine Translation
on ACES
Investigating the use of Paraphrase Generation for Question Reformulation in the FRANK QA system
Our two main conclusions are that cleaning of LC-QuAD 2. 0 is required as the errors present can affect evaluation; and that, due to limitations of FRANK's parser, paraphrase generation is not a method which we can rely on to improve the variety of natural language questions that FRANK can answer.
Cross-lingual Inference with A Chinese Entailment Graph
Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples.
Blindness to Modality Helps Entailment Graph Mining
Understanding linguistic modality is widely seen as important for downstream tasks such as Question Answering and Knowledge Graph Population.
Incorporating Temporal Information in Entailment Graph Mining
We present a novel method for injecting temporality into entailment graphs to address the problem of spurious entailments, which may arise from similar but temporally distinct events involving the same pair of entities.
Modality and Negation in Event Extraction
Language provides speakers with a rich system of modality for expressing thoughts about events, without being committed to their actual occurrence.
Multivalent Entailment Graphs for Question Answering
Drawing inferences between open-domain natural language predicates is a necessity for true language understanding.
Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction
We describe the design, the evaluation setup, and the results of the 2016 WMT shared task on cross-lingual pronoun prediction.
A Pronoun Test Suite Evaluation of the English--German MT Systems at WMT 2018
We evaluate the output of 16 English-to-German MT systems with respect to the translation of pronouns in the context of the WMT 2018 competition.
Pronoun Translation in English-French Machine Translation: An Analysis of Error Types
Pronouns are a long-standing challenge in machine translation.
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point
We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite.
What is it? Disambiguating the different readings of the pronoun `it'
In this paper, we address the problem of predicting one of three functions for the English pronoun {`}it{'}: anaphoric, event reference or pleonastic.
PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation
We present PROTEST, a test suite for the evaluation of pronoun translation by MT systems.
ParCor 1.0: A Parallel Pronoun-Coreference Corpus to Support Statistical MT
We present ParCor, a parallel corpus of texts in which pronoun coreference ― reduced coreference in which pronouns are used as referring expressions ― has been annotated.
