HittER: Hierarchical Transformers for Knowledge Graph Embeddings

This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.

PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract

Results from the Paper

 Ranked #1 on Link Prediction on FB15k-237 (Hit@10 metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Link Prediction FB15k-237 HittER MRR 0.373 # 7
Hits@3 0.409 # 6
Hit@10 0.558 # 1
Hit@1 0.279 # 1
Link Prediction WN18RR HittER MRR 0.503 # 6
Hits@10 0.584 # 18
Hits@3 0.516 # 11
Hits@1 0.462 # 6