UzBERT: pretraining a BERT model for Uzbek
Pretrained language models based on the Transformer architecture have achieved state-of-the-art results in various natural language processing tasks such as part-of-speech tagging, named entity recognition, and question answering. However, no such monolingual model for the Uzbek language is publicly available. In this paper, we introduce UzBERT, a pretrained Uzbek language model based on the BERT architecture. Our model greatly outperforms multilingual BERT on masked language model accuracy. We make the model publicly available under the MIT open-source license.
PDF Abstract
Datasets
Add Datasets
introduced or used in this paper
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
Absolute Position Encodings •
Adam •
Attention Dropout •
BERT •
BPE •
Dense Connections •
Dropout •
GELU •
Label Smoothing •
Layer Normalization •
Linear Layer •
Linear Warmup With Linear Decay •
Multi-Head Attention •
Position-Wise Feed-Forward Layer •
Residual Connection •
Scaled Dot-Product Attention •
Softmax •
Transformer •
Weight Decay •
WordPiece
