Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress
With the advent of Transformer, which was used in translation models in 2017, attention-based architectures began to attract attention. Furthermore, after the emergence of BERT, which strengthened the NLU-specific encoder part, which is a part of the Transformer, and the GPT architecture, which strengthened the NLG-specific decoder part, various methodologies, data, and models for learning the Pretrained Language Model began to appear. Furthermore, in the past three years, various Pretrained Language Models specialized for Korean have appeared. In this paper, we intend to numerically and qualitatively compare and analyze various Korean PLMs released to the public.
PDF Abstract
KLUE
KorSTS
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
Absolute Position Encodings •
Adam •
Attention Dropout •
BERT •
BPE •
Cosine Annealing •
Dense Connections •
Discriminative Fine-Tuning •
Dropout •
GELU •
GPT •
Label Smoothing •
Layer Normalization •
Linear Layer •
Linear Warmup With Cosine Annealing •
Linear Warmup With Linear Decay •
Multi-Head Attention •
Position-Wise Feed-Forward Layer •
Residual Connection •
Scaled Dot-Product Attention •
Softmax •
Transformer •
Weight Decay •
WordPiece
