Search Results for author:
Found 8 papers, 1 papers with code
Opening up Minds with Argumentative Dialogues
We show that while both models perform closely in terms of opening up minds, the argument-based model is significantly better on other dialogue properties such as engagement and clarity.
Identifying Annotator Bias: A new IRT-based method for bias identification
Our interpretation of IRT offers an original bias identification method that can be used to compare annotators{'} bias and characterise annotation disagreement.
Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions
We take a close look at a recent dataset of TED-talks annotated with the questions they implicitly evoke, TED-Q (Westera et al., 2020).
Aligning Intraobserver Agreement by Transitivity
Annotation reproducibility and accuracy rely on good consistency within annotators.
Agreement is overrated: A plea for correlation to assess human evaluation reliability
Following Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b), such phenomena can be explained in terms of irreducible human language variability.
The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations
Rating and Likert scales are widely used in evaluation experiments to measure the quality of Natural Language Generation (NLG) systems.
Evaluation methodologies in Automatic Question Generation 2013-2018
In the last few years Automatic Question Generation (AQG) has attracted increasing interest.
Rethinking the Agreement in Human Evaluation Tasks
For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care.
