no code implementations • 16 Jan 2023 • Youmna Farag, Charlotte O. Brand, Jacopo Amidei, Paul Piwek, Tom Stafford, Svetlana Stoyanchev, Andreas Vlachos
We show that while both models perform closely in terms of opening up minds, the argument-based model is significantly better on other dialogue properties such as engagement and clarity.
no code implementations • COLING 2020 • Jacopo Amidei, Paul Piwek, Alistair Willis
Our interpretation of IRT offers an original bias identification method that can be used to compare annotators{'} bias and characterise annotation disagreement.
1 code implementation • COLING 2020 • Matthijs Westera, Jacopo Amidei, Laia Mayol
We take a close look at a recent dataset of TED-talks annotated with the questions they implicitly evoke, TED-Q (Westera et al., 2020).
no code implementations • 29 Sep 2020 • Jacopo Amidei
Annotation reproducibility and accuracy rely on good consistency within annotators.
no code implementations • WS 2019 • Jacopo Amidei, Paul Piwek, Alistair Willis
Following Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b), such phenomena can be explained in terms of irreducible human language variability.
no code implementations • WS 2019 • Jacopo Amidei, Paul Piwek, Alistair Willis
Rating and Likert scales are widely used in evaluation experiments to measure the quality of Natural Language Generation (NLG) systems.
no code implementations • WS 2018 • Jacopo Amidei, Paul Piwek, Alistair Willis
In the last few years Automatic Question Generation (AQG) has attracted increasing interest.
no code implementations • COLING 2018 • Jacopo Amidei, Paul Piwek, Alistair Willis
For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care.