Combining Label Propagation and Simple Models Out-performs Graph Neural Networks

Graph Neural Networks (GNNs) are the predominant technique for learning over graphs. However, there is relatively little understanding of why GNNs are successful in practice and whether they are necessary for good performance. Here, we show that for many standard transductive node classification benchmarks, we can exceed or match the performance of state-of-the-art GNNs by combining shallow models that ignore the graph structure with two simple post-processing steps that exploit correlation in the label structure: (i) an "error correlation" that spreads residual errors in training data to correct errors in test data and (ii) a "prediction correlation" that smooths the predictions on the test data. We call this overall procedure Correct and Smooth (C&S), and the post-processing steps are implemented via simple modifications to standard label propagation techniques from early graph-based semi-supervised learning methods. Our approach exceeds or nearly matches the performance of state-of-the-art GNNs on a wide variety of benchmarks, with just a small fraction of the parameters and orders of magnitude faster runtime. For instance, we exceed the best known GNN performance on the OGB-Products dataset with 137 times fewer parameters and greater than 100 times less training time. The performance of our methods highlights how directly incorporating label information into the learning algorithm (as was done in traditional techniques) yields easy and substantial performance gains. We can also incorporate our techniques into big GNN models, providing modest gains. Our code for the OGB results is at https://github.com/Chillee/CorrectAndSmooth.

PDF Abstract ICLR 2021 PDF ICLR 2021 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Node Property Prediction ogbn-arxiv GCN_res + C&S Test Accuracy 0.7297 ± 0.0022 # 30
Validation Accuracy 0.7423 ± 0.0014 # 29
Number of params 155824 # 39
Ext. data No # 1
Node Property Prediction ogbn-arxiv MLP + C&S Test Accuracy 0.7312 ± 0.0012 # 26
Validation Accuracy 0.7391 ± 0.0015 # 31
Number of params 175656 # 38
Ext. data No # 1
Node Property Prediction ogbn-arxiv GCN_res + C&S_v2 Test Accuracy 0.7313 ± 0.0017 # 25
Validation Accuracy 0.7445 ± 0.0011 # 26
Number of params 155824 # 39
Ext. data No # 1
Node Property Prediction ogbn-arxiv GAT + C&S Test Accuracy 0.7386 ± 0.0014 # 19
Validation Accuracy 0.7484 ± 0.0007 # 20
Number of params 1567000 # 14
Ext. data No # 1
Node Property Prediction ogbn-arxiv Plain Linear + C&S Test Accuracy 0.7126 ± 0.0001 # 51
Validation Accuracy 0.7300 ± 0.0001 # 44
Number of params 5160 # 54
Ext. data No # 1
Node Property Prediction ogbn-arxiv GAT(norm.adj.)+label reuse+C&S Test Accuracy 0.7395 ± 0.0012 # 17
Validation Accuracy 0.7519 ± 0.0008 # 8
Number of params 1441580 # 21
Ext. data No # 1
Node Property Prediction ogbn-arxiv Linear + C&S Test Accuracy 0.7222 ± 0.0002 # 37
Validation Accuracy 0.7368 ± 0.0004 # 35
Number of params 15400 # 53
Ext. data No # 1
Node Property Prediction ogbn-products Linear + C&S Test Accuracy 0.8301 ± 0.0001 # 18
Validation Accuracy 0.9134 ± 0.0001 # 35
Number of params 10763 # 43
Ext. data No # 1
Node Property Prediction ogbn-products GraphSAGE w/NS + C&S Test Accuracy 0.8041 ± 0.0022 # 30
Validation Accuracy 0.9238 ± 0.0007 # 21
Number of params 207919 # 31
Ext. data No # 1
Node Property Prediction ogbn-products GAT w/NS + C&S Test Accuracy 0.8092 ± 0.0037 # 27
Validation Accuracy 0.9263 ± 0.0008 # 19
Number of params 753622 # 21
Ext. data No # 1
Node Property Prediction ogbn-products Plain Linear + C&S Test Accuracy 0.8254 ± 0.0003 # 20
Validation Accuracy 0.9103 ± 0.0001 # 37
Number of params 4747 # 44
Ext. data No # 1
Node Property Prediction ogbn-products MLP + C&S Test Accuracy 0.8418 ± 0.0007 # 15
Validation Accuracy 0.9147 ± 0.0009 # 34
Number of params 96247 # 42
Ext. data No # 1
Node Property Prediction ogbn-products Spec-MLP-Wide + C&S Test Accuracy 0.8451 ± 0.0006 # 12
Validation Accuracy 0.9132 ± 0.0010 # 36
Number of params 406063 # 26
Ext. data No # 1

Methods


No methods listed for this paper. Add relevant methods here