ERNIE: Enhanced Language Representation with Informative Entities

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding... (read more)

PDF Abstract ACL 2019 PDF ACL 2019 Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Linguistic Acceptability CoLA ERNIE Accuracy 52.3% # 19
Relation Extraction FewRel ERNIE F1 88.32 # 1
Precision 88.49 # 1
Recall 88.44 # 1
Entity Linking FIGER ERNIE Accuracy 57.19 # 1
Macro F1 76.51 # 1
Micro F1 73.39 # 1
Semantic Textual Similarity MRPC ERNIE Accuracy 88.2% # 14
Natural Language Inference MultiNLI ERNIE Matched 84.0 # 20
Mismatched 83.2 # 16
Entity Typing Open Entity ERNIE F1 75.56 # 3
Precision 78.42 # 3
Recall 72.9 # 3
Natural Language Inference QNLI ERNIE Accuracy 91.3% # 19
Paraphrase Identification Quora Question Pairs ERNIE F1 71.2 # 9
Natural Language Inference RTE ERNIE Accuracy 68.8% # 21
Sentiment Analysis SST-2 Binary classification ERNIE Accuracy 93.5 # 25
Semantic Textual Similarity STS Benchmark ERNIE Pearson Correlation 0.832 # 20
Relation Extraction TACRED ERNIE F1 67.97 # 16

Methods used in the Paper


METHOD TYPE
Residual Connection
Skip Connections
Attention Dropout
Regularization
Linear Warmup With Linear Decay
Learning Rate Schedules
Weight Decay
Regularization
GELU
Activation Functions
Dense Connections
Feedforward Networks
Adam
Stochastic Optimization
WordPiece
Subword Segmentation
Softmax
Output Functions
Dropout
Regularization
Multi-Head Attention
Attention Modules
Layer Normalization
Normalization
Scaled Dot-Product Attention
Attention Mechanisms
BERT
Language Models