MERA: A Comprehensive LLM Evaluation in Russian

no code implementations • 9 Jan 2024 • Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Denis Dimitrov, Alexander Panchenko, Sergei Markov

To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language.

RuCLIP -- new models and experiments: a technical report

1 code implementation • 22 Feb 2022 • Alex Shonenkov, Andrey Kuznetsov, Denis Dimitrov, Tatyana Shavrina, Daniil Chesakov, Anastasia Maltseva, Alena Fenogenova, Igor Pavlov, Anton Emelyanov, Sergey Markov, Daria Bakshandaeva, Vera Shybaeva, Andrey Chertok

In the report we propose six new implementations of ruCLIP model trained on our 240M pairs.

Translation

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models

no code implementations • 15 Feb 2022 • Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Tatiana Shavrina, Anton Emelyanov, Denis Shevelev, Alexandr Kukushkin, Valentin Malykh, Ekaterina Artemova

In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of language understanding tasks.

Common Sense Reasoning Reading Comprehension

RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

2 code implementations • EMNLP 2020 • Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, Andrey Evlampiev

In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE.

Common Sense Reasoning Lexical Entailment +5

Humans Keep It One Hundred: an Overview of AI Journey

1 code implementation • LREC 2020 • Tatiana Shavrina, Anton Emelyanov, Alena Fenogenova, Vadim Fomin, Vladislav Mikhailov, Andrey Evlampiev, Valentin Malykh, Vladimir Larin, Alex Natekin, Aleks Vatulin, R, Peter Romov, Daniil Anastasiev, Nikolai Zinov, Andrey Chertok

Artificial General Intelligence (AGI) is showing growing performance in numerous applications - beating human performance in Chess and Go, using knowledge bases and text sources to answer questions (SQuAD) and even pass human examination (Aristo project).

Text Generation