Search Results for author:
Found 31 papers, 12 papers with code
Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation
NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU.
Gradations of Error Severity in Automatic Image Descriptions
Because our manipulated descriptions form minimal pairs with the reference descriptions, we are able to assess the impact of different kinds of errors on the perceived quality of the descriptions.
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions
Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility.
How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE-domain
While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce.
Open Dutch WordNet
We describe Open Dutch WordNet, which has been derived from the Cornetto database, the Princeton WordNet and open source resources.
WordNet-based similarity metrics for adjectives
Le and Fokkens (2015) recently showed that taxonomy-based approaches are more reliable than corpus-based approaches in estimating human similarity ratings.
Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations
This review gives an extensive overview of evaluation methods for task-oriented dialogue systems, paying special attention to practical applications of dialogue systems, for example for customer service.
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible.
Evaluating NLG systems: A brief introduction
This year the International Conference on Natural Language Generation (INLG) will feature an award for the paper with the best evaluation.
Implicit causality in GPT-2: a case study
This case study investigates the extent to which a language model (GPT-2) is able to capture native speakers' intuitions about implicit causality in a sentence completion task.
Underreporting of errors in NLG output, and what to do about it
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make.
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
By applying this framework to the GEM generation benchmark, we propose an evaluation suite made of 80 challenge sets, demonstrate the kinds of analyses that it enables and shed light onto the limits of current generation models.
Preregistering NLP Research
Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study.
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
On the use of human reference data for evaluating automatic image descriptions
Automatic image description systems are commonly trained and evaluated using crowdsourced, human-generated image descriptions.
On task effects in NLG corpus elicitation: a replication study using mixed effects modeling
Task effects in NLG corpus elicitation recently started to receive more attention, but are usually not modeled statistically.
Best practices for the human evaluation of automatically generated text
Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated.
Neural data-to-text generation: A comparison between pipeline and end-to-end architectures
In contrast, recent neural models for data-to-text generation have been proposed as end-to-end approaches, where the non-linguistic input is rendered in natural language with much less explicit intermediate representations in-between.
Ranked #8 on
Data-to-Text Generation
on WebNLG Full
Talking about other people: an endless range of possibilities
This taxonomy serves as a reference point to think about how other people should be described, and can be used to classify and compute statistics about labels applied to people.
Measuring the Diversity of Automatic Image Descriptions
Automatic image description systems typically produce generic sentences that only make use of a small subset of the vocabulary available to them.
Varying image description tasks: spoken versus written descriptions
Automatic image description systems are commonly trained and evaluated on written image descriptions.
DIDEC: The Dutch Image Description and Eye-tracking Corpus
We present a corpus of spoken Dutch image descriptions, paired with two sets of eye-tracking data: Free viewing, where participants look at images without any particular purpose, and Description viewing, where we track eye movements while participants produce spoken descriptions of the images they are viewing.
Cross-linguistic differences and similarities in image descriptions
Automatic image description systems are commonly trained and evaluated on large image description datasets.
Room for improvement in automatic image description: an error analysis
In recent years we have seen rapid and significant progress in automatic image description but what are the open problems in this area?
Pragmatic descriptions of perceptual stimuli
This research proposal discusses pragmatic factors in image description, arguing that current automatic image description systems do not take these factors into account.
Building a Dictionary of Affixal Negations
This paper discusses the need for a dictionary of affixal negations and regular antonyms to facilitate their automatic detection in text.
Pragmatic factors in image description: the case of negations
We provide a qualitative analysis of the descriptions containing negations (no, not, n't, nobody, etc) in the Flickr30K corpus, and a categorization of negation uses.
Stereotyping and Bias in the Flickr30K Dataset
An untested assumption behind the crowdsourced descriptions of the images in the Flickr30K dataset (Young et al., 2014) is that they "focus only on the information that can be obtained from the image alone" (Hodosh et al., 2013, p. 859).
The VU Sound Corpus: Adding More Fine-grained Annotations to the Freesound Database
The main goal of this study is to find out (i) whether it is feasible to collect keywords for a large collection of sounds through crowdsourcing, and (ii) how people talk about sounds, and what information they can infer from hearing a sound in isolation.
Detecting and ordering adjectival scalemates
This paper presents a pattern-based method that can be used to infer adjectival scales, such as <lukewarm, warm, hot>, from a corpus.
