“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation

no code implementations • ACL (EvalNLGEval, INLG) 2020 • Stephanie Schoch, Diyi Yang, Yangfeng Ji

Despite recent efforts reviewing current human evaluation practices for natural language generation (NLG) research, the lack of reported question wording and potential for framing effects or cognitive biases influencing results has been widely overlooked.

Text Generation

Contextualizing Variation in Text Style Transfer Datasets

no code implementations • INLG (ACL) 2021 • Stephanie Schoch, Wanyu Du, Yangfeng Ji

Text style transfer involves rewriting the content of a source sentence in a target style.

Style Transfer Text Style Transfer

Underreporting of errors in NLG output, and what to do about it

no code implementations • INLG (ACL) 2021 • Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, Luou Wen

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make.

Text Generation