JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.

PDF Abstract Findings (ACL) 2021 PDF Findings (ACL) 2021 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
KG-to-Text Generation PathQuestion JointGT (T5) BLEU 60.45 # 4
METEOR 45.38 # 3
ROUGE 77.59 # 4
KG-to-Text Generation PathQuestion JointGT (BART) BLEU 65.89 # 1
METEOR 48.25 # 1
ROUGE 78.87 # 1
KG-to-Text Generation PathQuestion BART BLEU 63.74 # 2
METEOR 47.23 # 2
ROUGE 77.76 # 2
KG-to-Text Generation PathQuestion T5 BLEU 58.95 # 5
METEOR 44.72 # 4
ROUGE 76.58 # 5
KG-to-Text Generation WebNLG 2.0 (Constrained) T5 BLEU 58.66 # 2
METEOR 46.04 # 2
ROUGE 73.06 # 2
KG-to-Text Generation WebNLG 2.0 (Constrained) JointGT (T5) BLEU 61.01 # 1
METEOR 46.32 # 1
ROUGE 73.57 # 1
KG-to-Text Generation WebNLG 2.0 (Constrained) JointGT (BART) BLEU 58.55 # 3
METEOR 45.01 # 3
ROUGE 72.31 # 3
KG-to-Text Generation WebNLG 2.0 (Constrained) BART BLEU 56.65 # 4
METEOR 44.51 # 4
ROUGE 70.94 # 4
KG-to-Text Generation WebNLG 2.0 (Unconstrained) BART BLEU 64.55 # 3
METEOR 46.51 # 4
ROUGE 75.13 # 3
KG-to-Text Generation WebNLG 2.0 (Unconstrained) T5 BLEU 64.42 # 4
METEOR 46.58 # 3
ROUGE 74.77 # 4
KG-to-Text Generation WebNLG 2.0 (Unconstrained) JointGT (BART) BLEU 65.92 # 2
METEOR 47.15 # 2
ROUGE 76.10 # 1
KG-to-Text Generation WebNLG 2.0 (Unconstrained) JointGT (T5) BLEU 66.14 # 1
METEOR 47.25 # 1
ROUGE 75.91 # 2
KG-to-Text Generation WebQuestions JointGT (BART) BLEU 30.02 # 1
METEOR 32.05 # 1
ROUGE 55.6 # 1
KG-to-Text Generation WebQuestions JointGT (T5) BLEU 28.95 # 4
METEOR 31.29 # 3
ROUGE 54.47 # 5
KG-to-Text Generation WebQuestions T5 BLEU 28.78 # 5
METEOR 30.55 # 5
ROUGE 55.12 # 4
KG-to-Text Generation WebQuestions BART BLEU 29.61 # 2
METEOR 31.48 # 2
ROUGE 55.42 # 3

Methods