Deep Open Intent Classification with Adaptive Decision Boundary

18 Dec 2020  ยท  Hanlei Zhang, Hua Xu, Ting-En Lin ยท

Open intent classification is a challenging task in dialogue systems. On the one hand, it should ensure the quality of known intent identification. On the other hand, it needs to detect the open (unknown) intent without prior knowledge. Current models are limited in finding the appropriate decision boundary to balance the performances of both known intents and the open intent. In this paper, we propose a post-processing method to learn the adaptive decision boundary (ADB) for open intent classification. We first utilize the labeled known intent samples to pre-train the model. Then, we automatically learn the adaptive spherical decision boundary for each known class with the aid of well-trained features. Specifically, we propose a new loss function to balance both the empirical risk and the open space risk. Our method does not need open intent samples and is free from modifying the model architecture. Moreover, our approach is surprisingly insensitive with less labeled data and fewer known intents. Extensive experiments on three benchmark datasets show that our method yields significant improvements compared with the state-of-the-art methods. The codes are released at https://github.com/thuiar/Adaptive-Decision-Boundary.

PDF Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Open Intent Detection BANKING77 (25%known) ADB 1:1 Accuracy 78.85 # 2
F1-score 71.62 # 2
Open Intent Detection BANKING-77 (50% known) ADB 1:1 Accuracy 78.86 # 2
F1-score 80.90 # 2
Open Intent Detection BANKING-77 (75% known) ADB 1:1 Accuracy 81.08 # 2
F1-score 85.96 # 2
Open Intent Detection OOS(25%known) ADB 1:1 Accuracy 87.59 # 2
F1-score 77.19 # 2
Open Intent Detection OOS(50%known) ADB 1:1 Accuracy 86.54 # 2
F1-score 85.05 # 2
Open Intent Detection OOS(75%known) ADB 1:1 Accuracy 86.32 # 2
F1-score 88.53 # 2
Open Intent Detection StackOverFlow(25%known) ADB 1:1 Accuracy 86.72 # 1
F1-score 80.83 # 1
Open Intent Detection StackOverFlow(50%known) ADB 1:1 Accuracy 86.40 # 1
F1-score 85.83 # 1
Open Intent Detection StackOverFlow(75%known) ADB 1:1 Accuracy 82.78 # 1
F1-score 85.99 # 1

Methods


No methods listed for this paper. Add relevant methods here