Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

1 Jun 2023  ·  Erik Arakelyan, Arnav Arora, Isabelle Augenstein ·

Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose $\textbf{T}$opic $\textbf{E}$fficient $\textbf{St}$anc$\textbf{E}$ $\textbf{D}$etection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier. We evaluate the method on an existing benchmark of $16$ datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of $3.5$ F1 points increase in-domain, and is more generalizable with an averaged increase of $10.2$ F1 on out-of-domain evaluation while using $\leq10\%$ of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Stance Detection ARC (AI2 Reasoning Challenge) TESTED F1 64.82 # 1
Stance Detection argmin TESTED F1 62.79 # 1
Stance Detection emergent TESTED F1 82.1 # 1
Stance Detection FNC-1 TESTED F1 83.17 # 1
Stance Detection iac1 TESTED F1 56.97 # 1
Stance Detection ibmcs TESTED F1 88.06 # 1
Stance Detection mtsd TESTED F1 63.96 # 1
Stance Detection Perspectrum TESTED F1 83.11 # 1
Stance Detection poldeb TESTED F1 52.76 # 1
Stance Detection RumourEval TESTED F1 66.58 # 1
Stance Detection SCD TESTED F1 64.71 # 1
Stance Detection SemEval 2019 TESTED F1 58.72 # 1
Stance Detection Snopes TESTED F1 78.61 # 1
Stance Detection VAST TESTED F1 57.47 # 1
Stance Detection wtwt TESTED F1 70.98 # 1