Learning on Large-scale Text-attributed Graphs via Variational Inference

26 Oct 2022  ·  Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, Jian Tang ·

This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description. An ideal solution for such a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs). However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by training large language models and GNNs together. In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational Expectation-Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such a procedure allows training the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach.

PDF Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Node Property Prediction ogbn-arxiv GLEM+RevGAT Test Accuracy 0.7694 ± 0.0025 # 5
Validation Accuracy 0.7746 ± 0.0018 # 5
Number of params 140469624 # 4
Ext. data Yes # 1
Node Property Prediction ogbn-papers100M GLEM+GIANT+GAMLP Test Accuracy 0.7037 ± 0.0002 # 1
Validation Accuracy 0.7354 ± 0.0001 # 1
Number of params 154775375 # 2
Ext. data Yes # 1
Node Property Prediction ogbn-products GLEM+EnGCN Test Accuracy 0.9014 ± 0.0012 # 1
Validation Accuracy 0.9370 ± 0.0004 # 6
Number of params 139633805 # 3
Ext. data Yes # 1
Node Property Prediction ogbn-products GLEM+GIANT+SAGN+SCR Test Accuracy 0.8737 ± 0.0006 # 3
Validation Accuracy 0.9400 ± 0.0003 # 2
Number of params 139792525 # 2
Ext. data Yes # 1


No methods listed for this paper. Add relevant methods here