Search Results for author:
Found 53 papers, 33 papers with code
Razmecheno: Named Entity Recognition from Digital Archive of Diaries “Prozhito”
The labelling is carried out on the crowdsourcing platfrom Yandex. Toloka in two stages.
Supervised and Unsupervised Evaluation of Synthetic Code-Switching
Code-switching (CS) is a phenomenon of mixing words and phrases from multiple languages within a single sentence or conversation.
Single Example Can Improve Zero-Shot Data Generation
We explore two approaches to the generation of task-oriented utterances: in the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training.
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs
Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions.
Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop
Training and deploying machine learning models relies on a large amount of human-annotated data.
Beemo: Benchmark of Expert-edited Machine-generated Outputs
The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains.
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection
Category (iii) aims to detect attempts to obfuscate the fact that a text was machine-generated, while category (iv) looks for cases where the LLM was used to polish a human-written text, which is typically acceptable in academic writing, but not in education.
Papilusion at DAGPap24: Paper or Illusion? Detecting AI-generated Scientific Papers
This paper presents Papilusion, an AI-generated scientific text detector developed within the DAGPap24 shared task on detecting automatically generated scientific papers.
RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models.
AIpom at SemEval-2024 Task 8: Detecting AI-produced Outputs in M4
This paper describes AIpom, a system designed to detect a boundary between human-written and machine-generated text (SemEval-2024 Task 8, Subtask C: Human-Machine Mixed Text Detection).
RuBia: A Russian Language Bias Detection Dataset
To illustrate the dataset's purpose, we conduct a diagnostic evaluation of state-of-the-art or near-state-of-the-art LLMs and discuss the LLMs' predisposition to social biases.
Sebastian, Basti, Wastl?! Recognizing Named Entities in Bavarian Dialectal Data
Named Entity Recognition (NER) is a fundamental task to extract key information from texts, but annotated resources are scarce for dialects.
Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties
Inspired by prior work on English varieties, we craft and manually evaluate perturbation rules that transform German sentences into colloquial forms and use them to synthesize test sets in four ToD datasets.
LUNA: A Framework for Language Understanding and Naturalness Assessment
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text.
Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?
Our results show that the choice of the right AED method and model size is indeed crucial and derive practical recommendations for how to use AED methods to clean instruction-tuning data.
Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data
Transferring information retrieval (IR) models from a high-resource language (typically English) to other languages in a zero-shot fashion has become a widely adopted approach.
Low-resource Bilingual Dialect Lexicon Induction with Large Language Models
Bilingual word lexicons are crucial tools for multilingual natural language understanding and machine translation tasks, as they facilitate the mapping of words in one language to their synonyms in another language.
Bilingual Lexicon Induction
Natural Language Understanding
+4
Can BERT eat RuCoLA? Topological Data Analysis to Explain
Our results contribute to understanding the behavior of monolingual LMs in the acceptability classification task, provide insights into the functional roles of attention heads, and highlight the advantages of TDA-based approaches for analyzing LMs.
Ranked #1 on
Linguistic Acceptability
on RuCoLA
TAPE: Assessing Few-shot Russian Language Understanding
Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes.
Ranked #1 on
Ethics
on Ethics (per ethics)
RuCoLA: Russian Corpus of Linguistic Acceptability
Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers.
Ranked #2 on
Linguistic Acceptability
on ItaCoLA
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation capabilities from multiple perspectives.
Template-based Approach to Zero-shot Intent Recognition
We outperform previous state-of-the-art f1-measure by up to 16\% for unseen intents, using intent labels and user utterances and without accessing external sources (such as knowledge bases).
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
The first task is framed as a binary classification problem.
RuNNE-2022 Shared Task: Recognizing Nested Named Entities
In the test set the frequency of all entity types is even.
Acceptability Judgements via Examining the Topology of Attention Maps
The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP.
Ranked #1 on
Linguistic Acceptability
on ItaCoLA
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of language understanding tasks.
Razmecheno: Named Entity Recognition from Digital Archive of Diaries "Prozhito"
The labelling is carried out on the crowdsourcing platfrom Yandex. Toloka in two stages.
Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models
This is in line with the common understanding of how multilingual models conduct transferring between languages
Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models across many NLP tasks.
Artificial Text Detection via Examining the Topology of Attention Maps
The impressive capabilities of recent generative models to create texts that are challenging to distinguish from the human-written ones can be misused for generating fake news, product reviews, and even abusive content.
NEREL: A Russian Dataset with Nested Named Entities, Relations and Events
In this paper, we present NEREL, a Russian dataset for named entity recognition and relation extraction.
A Single Example Can Improve Zero-Shot Data Generation
In the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training.
A Differentiable Language Model Adversarial Attack on Text Classifiers
Moreover, due to the usage of the fine-tuned language model, the generated adversarial examples are hard to detect, thus current models are not robust.
MOROCCO: Model Resource Comparison Framework
The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environments is often prohibitively expensive.
Teaching a Massive Open Online Course on Natural Language Processing
This paper presents a new Massive Open Online Course on Natural Language Processing, targeted at non-English speaking students.
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings.
RuSentEval: Linguistic Source, Encoder Force!
The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language.
Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates
Annotating training data for sequence tagging of texts is usually very time-consuming.
Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection
In turn, the Mahalanobis distance captures this disparity easily.
SumTitles: a Summarization Dataset with Low Extractiveness
The existing dialogue summarization corpora are significantly extractive.
RuREBus: a Case Study of Joint Named Entity Recognition and Relation Extraction from e-Government Domain
We show-case an application of information extraction methods, such as named entity recognition (NER) and relation extraction (RE) to a novel corpus, consisting of documents, issued by a state agency.
RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE.
Ranked #1 on
Word Sense Disambiguation
on RUSSE
ELMo and BERT in semantic change detection for Russian
We study the effectiveness of contextualized embeddings for the task of diachronic semantic change detection for Russian language data.
DaNetQA: a yes/no Question Answering Dataset for the Russian Language
The task is to take both the question and a paragraph as input and come up with a yes/no answer, i. e. to produce a binary output.
So What's the Plan? Mining Strategic Planning Documents
In this paper we present a corpus of Russian strategic planning documents, RuREBus.
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
In this work, we step outside the computer vision domain by leveraging the language modeling task, which is the core of natural language processing (NLP).
Data-driven models and computational tools for neurolinguistics: a language technology perspective
In this paper, our focus is the connection and influence of language technologies on the research in neurolinguistics.
A Joint Approach to Compound Splitting and Idiomatic Compound Detection
Applications such as machine translation, speech recognition, and information retrieval require efficient handling of noun compounds as they are one of the possible sources for out-of-vocabulary (OOV) words.
Word Sense Disambiguation for 158 Languages using Word Embeddings Only
We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings by Grave et al. (2018), enabling WSD in these languages.
Char-RNN and Active Learning for Hashtag Segmentation
We explore the abilities of character recurrent neural network (char-RNN) for hashtag segmentation.
Sentence Embeddings for Russian NLU
We investigate the performance of sentence embeddings models on several tasks for the Russian language.
A Dataset for Noun Compositionality Detection for a Slavic Language
This paper presents the first gold-standard resource for Russian annotated with compositionality information of noun compounds.
Multilingual Named Entity Recognition Using Pretrained Embeddings, Attention Mechanism and NCRF
In this paper we tackle multilingual named entity recognition task.
