Search Results for author:
Found 12 papers, 6 papers with code
How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?
While machine translation evaluation has been studied primarily for high-resource languages, there has been a recent interest in evaluation for low-resource languages due to the increasing availability of data and models.
Closing the Gap in the Trade-off between Fair Representations and Accuracy
The rapid developments of various machine learning models and their deployments in several applications has led to discussions around the importance of looking beyond the accuracies of these models.
BiPhone: Modeling Inter Language Phonetic Influences in Text
To the best of our knowledge, FunGLUE is the first benchmark to introduce L1-L2 interactions in text.
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages
In this paper, we fill this gap by creating an MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, and use it to establish correlations between annotator scores and scores obtained using existing automatic metrics.
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on.
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e. g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc.
A Tutorial on Evaluation Metrics used in Natural Language Generation
(iv) What are the criticisms and shortcomings of existing metrics?
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining
However, no such data is publicly available, and hence existing models are usually trained using a single relevant response and multiple randomly selected responses from other contexts (random negatives).
A Survey of Evaluation Metrics Used for NLG Systems
The expanding number of NLG models and the shortcomings of the current metrics has led to a rapid surge in the number of evaluation metrics proposed since 2014.
Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
We believe that the non-adversarial dataset created as a part of this work would complement the research on adversarial evaluation and give a more realistic assessment of the ability of RC models.
ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
Specifically, it has gates which decide whether an option can be eliminated given the passage, question pair and if so it tries to make the passage representation orthogonal to this eliminated option (akin to ignoring portions of the passage corresponding to the eliminated option).
Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
ADEM(Lowe et al. 2017) formulated the automatic evaluation of dialogue systems as a learning problem and showed that such a model was able to predict responses which correlate significantly with human judgements, both at utterance and system level.
