EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems

The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues remain largely unaddressed. This is despite emotions and dialogue success having equally important roles in a natural system. Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks. To lay a foundation for studies on emotions in task-oriented dialogues, we introduce EmoWOZ, a large-scale manually emotion-annotated corpus of task-oriented dialogues. EmoWOZ is based on MultiWOZ, a multi-domain task-oriented dialogue dataset. It contains more than 11K dialogues with more than 83K emotion annotations of user utterances. In addition to Wizard-of-Oz dialogues from MultiWOZ, we collect human-machine dialogues within the same set of domains to sufficiently cover the space of various emotions that can happen during the lifetime of a data-driven dialogue system. To the best of our knowledge, this is the first large-scale open-source corpus of its kind. We propose a novel emotion labelling scheme, which is tailored to task-oriented dialogues. We report a set of experimental results to show the usability of this corpus for emotion recognition and state tracking in task-oriented dialogues.

PDF Abstract LREC 2022 PDF LREC 2022 Abstract

Datasets


Introduced in the Paper:

EmoWOZ

Used in the Paper:

MultiWOZ MELD EmoryNLP
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Emotion Recognition in Conversation EmoWoz ContextBERT Weighted F1 79.7 # 1
Macro F1 54.3 # 2
Emotion Recognition in Conversation EmoWoz COSMIC Weighted F1 77.1 # 2
Macro F1 56.3 # 1
Emotion Recognition in Conversation EmoWoz DialogueRNN-BERT Weighted F1 75.5 # 3
Macro F1 52.1 # 3
Emotion Recognition in Conversation EmoWoz DialogueRNN-GloVe Weighted F1 74.6 # 4
Macro F1 40.1 # 5
Emotion Recognition in Conversation EmoWoz BERT Weighted F1 73.5 # 5
Macro F1 50.1 # 4

Methods


No methods listed for this paper. Add relevant methods here