Search Results for author:
Found 35 papers, 27 papers with code
WMT20 Document-Level Markable Error Exploration
For an annotation experiment of two phases, we chose Czech and English documents translated by systems submitted to WMT20 News Translation Task.
A Bayesian Optimization Approach to Machine Translation Reranking
Reranking a list of candidates from a machine translation system with an external scoring model and returning the highest-scoring candidate remains a simple and effective method for improving the overall output quality.
Pitfalls and Outlooks in Using COMET
The COMET metric has blazed a trail in the machine translation community, given its strong correlation with human judgements of translation quality.
Distributional Properties of Subword Regularization
Subword regularization, used widely in NLP, improves model performance by reducing the dependency on exact tokenizations, augmenting the training corpus, and exposing the model to more unique contexts during training.
Preliminary WMT24 Ranking of General MT Systems and LLMs
This is the preliminary ranking of WMT24 General MT systems based on automatic metrics.
How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading
However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied.
AI-Assisted Human Evaluation of Machine Translation
The recently adopted annotation protocol, Error Span Annotation (ESA), has annotators marking erroneous parts of the translation and then assigning a final score.
Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
High-quality Machine Translation (MT) evaluation relies heavily on human judgments.
Interactive Analysis of LLMs using Meaningful Counterfactuals
First, the generated textual counterfactuals should be meaningful and readable to users and thus can be mentally compared to draw conclusions.
Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains
We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain.
Two Counterexamples to Tokenization and the Noiseless Channel
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), R\'enyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R\'enyi efficiency of the unigram distribution should be chosen.
AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails
Large Language Models (LLMs) have found several use cases in education, ranging from automatic question generation to essay evaluation.
Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
On the machine translation task, we explore (1) whether the choice of the vocabulary plays a role in model stealing scenarios and (2) if it is possible to extract the victim's vocabulary.
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Ten years ago a single metric, BLEU, governed progress in machine translation research.
Quality and Quantity of Machine Translation References for Automatic Metrics
Automatic machine translation metrics typically rely on human translations to determine the quality of system translations.
Evaluating Optimal Reference Translations
The overall translation quality reached by current machine translation (MT) systems for high-resourced language pairs is remarkably good.
RELIC: Investigating Large Language Model Responses using Self-Consistency
To address this challenge, we propose an interactive system that helps users gain insight into the reliability of the generated text.
A Diachronic Perspective on User Trust in AI under Uncertainty
In a human-AI collaboration, users build a mental model of the AI system based on its reliability and how it presents its decision, e. g. its presentation of system confidence and an explanation of the output.
A Formal Perspective on Byte-Pair Encoding
Via submodular functions, we prove that the iterative greedy version is a $\frac{1}{{\sigma(\boldsymbol{\mu}^\star)}}(1-e^{-{\sigma(\boldsymbol{\mu}^\star)}})$-approximation of an optimal merge sequence, where ${\sigma(\boldsymbol{\mu}^\star)}$ is the total backward curvature with respect to the optimal merge sequence $\boldsymbol{\mu}^\star$.
Revisiting Automated Topic Model Evaluation with Large Language Models
Topic models are used to make sense of large text collections.
Enhancing Textbooks with Visuals from the Web for Improved Learning
We release the dataset of textbooks with an associated image bank to inspire further research in this intersectional area of computer vision and NLP for education.
PWESuite: Phonetic Word Embeddings and Tasks They Facilitate
Mapping words into a fixed-dimensional vector space is the backbone of modern NLP.
Multimodal Shannon Game with Images
To investigate the impact of multimodal information in this game, we use human participants and a language model (LM, GPT-2).
Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference
Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference.
Sentence Ambiguity, Grammaticality and Complexity Probes
It is unclear whether, how and where large pre-trained language models capture subtle linguistic traits like ambiguity, grammaticality and sentence complexity.
Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models
We present an LSTM-based autoregressive language model which uses prefix embeddings (from a pretrained masked language model) via fusion (e. g. concatenation) to obtain a richer context representation for language modelling.
EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios
We present the Eyetracked Multi-Modal Translation (EMMT) corpus, a dataset containing monocular eye movement recordings, audio and 4-electrode electroencephalogram (EEG) data of 43 participants.
Knowledge Base Index Compression via Dimensionality and Precision Reduction
Finally, we show that it is possible to combine PCA with using 1bit per dimension.
Artefact Retrieval: Overview of NLP Models with Knowledge Base Access
Many NLP models gain performance by having access to a knowledge base.
Neural Machine Translation Quality and Post-Editing Performance
We test the natural expectation that using MT in professional translation saves human processing time.
Backtranslation Feedback Improves User Confidence in MT, Not Quality
Translating text into a language unknown to the text's author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility.
Sampling and Filtering of Neural Machine Translation Distillation Data
In most of neural machine translation distillation or stealing scenarios, the goal is to preserve the performance of the target model (teacher).
Leveraging Neural Machine Translation for Word Alignment
The most common tools for word-alignment rely on a large amount of parallel sentences, which are then usually processed according to one of the IBM model algorithms.
Outbound Translation User Interface Ptakopet: A Pilot Study
It is not uncommon for Internet users to have to produce a text in a foreign language they have very little knowledge of and are unable to verify the translation quality.
