Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset

3 Dec 2021  Â·  MatÄ›j Kocián, Jakub Náplava, Daniel Ĺ tancl, VladimĂ­r Kadlec ·

Web search engines focus on serving highly relevant results within hundreds of milliseconds. Pre-trained language transformer models such as BERT are therefore hard to use in this scenario due to their high computational demands. We present our real-time approach to the document ranking problem leveraging a BERT-based siamese architecture. The model is already deployed in a commercial search engine and it improves production performance by more than 3%. For further research and evaluation, we release DaReCzech, a unique data set of 1.6 million Czech user query-document pairs with manually assigned relevance levels. We also release Small-E-Czech, an Electra-small language model pre-trained on a large Czech corpus. We believe this data will support endeavours both of search relevance and multilingual-focused research communities.

PDF Abstract

Datasets


Introduced in the Paper:

DaReCzech

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Document Ranking DaReCzech Query-doc RobeCzech (Roberta-base) P@10 46.73 # 1
Document Ranking DaReCzech Query-doc Small-E-Czech (Electra-small) P@10 46.30 # 2
Document Ranking DaReCzech Siamese Small-E-Czech (Electra-small) P@10 45.26 # 3

Methods