K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.

PDF Abstract Findings (ACL) 2021 PDF Findings (ACL) 2021 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Entity Typing Open Entity K-Adapter ( fac-adapter ) F1 77.6916 # 1
Precision 79.6712 # 1
Recall 75.8081 # 2
Entity Typing Open Entity K-Adapter ( fac-adapter + lin-adapter ) F1 77.6127 # 2
Precision 78.9956 # 2
Recall 76.2774 # 1
Relation Classification TACRED K-Adapter F1 72.0 # 13
Relation Classification TACRED RoBERTa F1 71.3 # 8
Relation Extraction TACRED K-ADAPTER (F+L) F1 72.04 # 14
F1 (1% Few-Shot) 13.8 # 5
F1 (5% Few-Shot) 45.1 # 4
F1 (10% Few-Shot) 56.0 # 4