RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models (, and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.

PDF Abstract EMNLP 2020 PDF EMNLP 2020 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering DaNetQA Baseline TF-IDF1.1 Accuracy 0.621 # 16
Question Answering DaNetQA Human Benchmark Accuracy 0.915 # 2
Natural Language Inference LiDiRus Baseline TF-IDF1.1 MCC 0.06 # 17
Natural Language Inference LiDiRus Human Benchmark MCC 0.626 # 1
Reading Comprehension MuSeRC Baseline TF-IDF1.1 Average F1 0.587 # 20
EM 0.242 # 17
Reading Comprehension MuSeRC Human Benchmark Average F1 0.806 # 5
EM 0.42 # 8
Common Sense Reasoning PARus Baseline TF-IDF1.1 Accuracy 0.486 # 19
Common Sense Reasoning PARus Human Benchmark Accuracy 0.982 # 1
Natural Language Inference RCB Baseline TF-IDF1.1 Average F1 0.301 # 21
Accuracy 0.441 # 19
Natural Language Inference RCB Human Benchmark Average F1 0.68 # 1
Accuracy 0.702 # 1
Common Sense Reasoning RuCoS Baseline TF-IDF1.1 Average F1 0.26 # 15
EM 0.252 # 16
Common Sense Reasoning RuCoS Human Benchmark Average F1 0.93 # 1
EM 0.89 # 2
Word Sense Disambiguation RUSSE Baseline TF-IDF1.1 Accuracy 0.57 # 19
Word Sense Disambiguation RUSSE Human Benchmark Accuracy 0.805 # 1
Common Sense Reasoning RWSD Human Benchmark Accuracy 0.84 # 22
Common Sense Reasoning RWSD Baseline TF-IDF1.1 Accuracy 0.662 # 6
Natural Language Inference TERRa Human Benchmark Accuracy 0.92 # 1
Natural Language Inference TERRa Baseline TF-IDF1.1 Accuracy 0.471 # 22


No methods listed for this paper. Add relevant methods here