SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

EACL (DravidianLangTech) 2021 · Sai Muralidhar Jayanthi, Akshat Gupta ·

In this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.

PDF Abstract EACL (DravidianLangTech) 2021 PDF EACL (DravidianLangTech) 2021 Abstract

Code

Add Remove Mark official

murali1996/eacl2021-OffensEval-Drav… official

0

Tasks

Add Remove

Language Identification

Language Modelling

Masked Language Modeling

Datasets

Add Datasets introduced or used in this paper

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods

Add Remove

Adam • Attention Dropout • BERT • Dense Connections • Dropout • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • mBERT • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece