GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation

Recent improvements in KG-to-text generation are due to additional auxiliary pre-training tasks designed to give the fine-tune task a boost in performance. These tasks require extensive computational resources while only suggesting marginal improvements. Here, we demonstrate that by fusing graph-aware elements into existing pre-trained language models, we are able to outperform state-of-the-art models and close the gap imposed by additional pre-training tasks. We do so by proposing a mask structure to capture neighborhood information and a novel type encoder that adds a bias to the graph-attention weights depending on the connection type. Experiments on two KG-to-text benchmark datasets show our models are competitive while involving fewer parameters and no additional pre-training tasks. By formulating the problem as a framework, we can interchange the various proposed components and begin interpreting KG-to-text generative models based on the topological and type information found in a graph.

PDF Abstract COLING 2022 PDF COLING 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
KG-to-Text Generation EventNarrative BART BLEU 31.38 # 3
METEOR 26.68 # 4
ROUGE 62.65 # 5
BertScore 93.12 # 3
KG-to-Text Generation EventNarrative T5 BLEU 12.8 # 7
METEOR 22.77 # 7
ROUGE 52.06 # 7
BertScore 89.59 # 6
KG-to-Text Generation EventNarrative GAP - Me,r+γ BLEU 35.08 # 1
METEOR 27.5 # 2
ROUGE 64.28 # 3
BertScore 93.38 # 2
KG-to-Text Generation EventNarrative GAP - Me,re BLEU 34.02 # 2
METEOR 26.93 # 3
ROUGE 62.9 # 4
KG-to-Text Generation EventNarrative JointGT BLEU 31.19 # 5
METEOR 26.58 # 6
ROUGE 64.91 # 2
BertScore 93.68 # 1
KG-to-Text Generation WebNLG 2.0 (Unconstrained) GAP - Me,r+γ BLEU 66.2 # 1
ROUGE 76.36 # 1
KG-to-Text Generation WebNLG 2.0 (Unconstrained) GAP - Me,re ROUGE 76.22 # 2
KG-to-Text Generation WebNLG 2.0 (Unconstrained) JointGT (BART) - w/ JointGTPretrain BLEU 65.92 # 3
METEOR 47.15 # 2
ROUGE 76.1 # 3
KG-to-Text Generation WebNLG 2.0 (Unconstrained) JointGT (BART) - w/ BARTPretrain BLEU 64.6 # 5
METEOR 46.77 # 4
ROUGE 75.74 # 6
KG-to-Text Generation WebNLG 2.0 (Unconstrained) KGPT w/o pretrain BLEU 62.3 # 9
METEOR 44.33 # 8
ROUGE 73 # 10
KG-to-Text Generation WebNLG 2.0 (Unconstrained) GCN BLEU 60.8 # 12
METEOR 42.76 # 10
ROUGE 71.13 # 12

Methods


No methods listed for this paper. Add relevant methods here