no code implementations • LREC 2022 • Anaëlle Baledent, Yann Mathet, Antoine Widlöcher, Christophe Couronne, Jean-Luc Manguin
That inter-annotator agreement measure, which is required to check the reliability of data and the reproducibility of an annotation task, and thus to establish a gold standard, is another thorny problem.
no code implementations • CL 2017 • Yann Mathet
Agreement on unitizing, where several annotators freely put units of various sizes and categories on a continuum, is difficult to assess because of the simultaneaous discrepancies in positioning and categorizing.