Browse > Natural Language Processing > Lexical Normalization

Lexical Normalization

3 papers with code · Natural Language Processing

Lexical normalization is the task of translating/transforming a non standard text to a standard register.

Example:

new pix comming tomoroe
new pictures coming tomorrow

Datasets usually consists of tweets, since these naturally contain a fair amount of these phenomena.

For lexical normalization, only replacements on the word-level are annotated. Some corpora include annotation for 1-N and N-1 replacements. However, word insertion/deletion and reordering is not part of the task.

Leaderboards

Greatest papers with code

Adapting Sequence to Sequence models for Text Normalization in Social Media

12 Apr 2019Isminoula/TextNormSeq2Seq

Social media offer an abundant source of valuable raw data, however informal writing can quickly become a bottleneck for many natural language processing (NLP) tasks.

LEXICAL NORMALIZATION

MoNoise: Modeling Noise Using a Modular Normalization System

10 Oct 2017wesselreijngoud/masterthesis2019

We show that MoNoise beats the state-of-the-art on different normalization benchmarks for English and Dutch, which all define the task of normalization slightly different.

LEXICAL NORMALIZATION SPELLING CORRECTION WORD EMBEDDINGS

A Multi-cascaded Deep Model for Bilingual SMS Classification

29 Nov 2019haroonshakeel/bilingual_sms_classification

Our model achieves high accuracy for classification on this dataset and outperforms the previous model for multilingual text classification, highlighting language independence of McM.

LEXICAL NORMALIZATION MULTILINGUAL TEXT CLASSIFICATION TEXT CLASSIFICATION TRANSLITERATION