Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER

29 Aug 2019  ·  Peng-Hsuan Li, Tsu-Jui Fu, Wei-Yun Ma ·

BiLSTM has been prevalently used as a core module for NER in a sequence-labeling setup. State-of-the-art approaches use BiLSTM with additional resources such as gazetteers, language-modeling, or multi-task supervision to further improve NER. This paper instead takes a step back and focuses on analyzing problems of BiLSTM itself and how exactly self-attention can bring improvements. We formally show the limitation of (CRF-)BiLSTM in modeling cross-context patterns for each word -- the XOR limitation. Then, we show that two types of simple cross-structures -- self-attention and Cross-BiLSTM -- can effectively remedy the problem. We test the practical impacts of the deficiency on real-world NER datasets, OntoNotes 5.0 and WNUT 2017, with clear and consistent improvements over the baseline, up to 8.7% on some of the multi-token entity mentions. We give in-depth analyses of the improvements across several aspects of NER, especially the identification of multi-token mentions. This study should lay a sound foundation for future improvements on sequence-labeling NER. (Source codes: https://github.com/jacobvsdanniel/cross-ner)

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Named Entity Recognition (NER) Ontonotes v5 (English) Att-BiLSTM-CNN F1 88.4 # 20
Precision 88.71 # 2
Recall 88.11 # 3
Named Entity Recognition (NER) WNUT 2017 Cross-BiLSTM-CNN F1 42.85 # 19
Precision 58.28 # 1
Recall 33.92 # 1

Methods