Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

22 Apr 2019  ·  Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Jason Weston ·

The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Conversational Response Selection Douban Poly-encoder MAP 0.608 # 7
MRR 0.650 # 7
P@1 0.475 # 7
R10@1 0.299 # 7
R10@2 0.494 # 5
R10@5 0.822 # 8
Conversational Response Selection DSTC7 Ubuntu Bi-encoder (v2) 1-of-100 Accuracy 70.9% # 2
Conversational Response Selection DSTC7 Ubuntu Bi-encoder 1-of-100 Accuracy 66.3% # 3
Conversational Response Selection RRS Ranking Test Poly-encoder NDCG@3 0.679 # 1
NDCG@5 0.765 # 1
Conversational Response Selection Ubuntu Dialogue (v1, Ranking) Poly-encoder R10@1 0.882 # 4
R10@2 0.949 # 2
R10@5 0.990 # 2