Effective Sequence-to-Sequence Dialogue State Tracking

Sequence-to-sequence models have been applied to a wide variety of NLP tasks, but how to properly use them for dialogue state tracking has not been systematically investigated. In this paper, we study this problem from the perspectives of pre-training objectives as well as the formats of context representations. We demonstrate that the choice of pre-training objective makes a significant difference to the state tracking quality. In particular, we find that masked span prediction is more effective than auto-regressive language modeling. We also explore using Pegasus, a span prediction-based pre-training objective for text summarization, for the state tracking model. We found that pre-training for the seemingly distant summarization task works surprisingly well for dialogue state tracking. In addition, we found that while recurrent state context representation works also reasonably well, the model may have a hard time recovering from earlier mistakes. We conducted experiments on the MultiWOZ 2.1-2.4, WOZ 2.0, and DSTC2 datasets with consistent observations.

PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Multi-domain Dialogue State Tracking MULTIWOZ 2.1 T5 (span) Joint Acc 52.8 # 16
Multi-domain Dialogue State Tracking MULTIWOZ 2.1 Pegasus Joint Acc 54.4 # 11
Multi-domain Dialogue State Tracking MULTIWOZ 2.2 T5 (span) Joint Acc 57.6 # 1
Multi-domain Dialogue State Tracking MULTIWOZ 2.2 Pegasus Joint Acc 56.6 # 3
Multi-domain Dialogue State Tracking MULTIWOZ 2.3 T5 (span) Joint Acc 59.3 # 2
Multi-domain Dialogue State Tracking MULTIWOZ 2.3 Pegasus Joint Acc 60.2 # 1
Multi-domain Dialogue State Tracking MULTIWOZ 2.4 Pegasus Joint Acc 66.6 # 2
Multi-domain Dialogue State Tracking MULTIWOZ 2.4 T5 (span) Joint Acc 67.1 # 1
Dialogue State Tracking Second dialogue state tracking challenge T5 (span) Joint 73.6 # 4
Dialogue State Tracking Wizard-of-Oz T5 (span) Joint 91 # 3

Methods


No methods listed for this paper. Add relevant methods here