Robust Optimization as Data Augmentation for Large-scale Graphs

19 Oct 2020  ·  Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, Tom Goldstein ·

Data augmentation helps neural networks generalize better by enlarging the training set, but it remains an open question how to effectively augment graph data to enhance the performance of GNNs (Graph Neural Networks). While most existing graph regularizers focus on manipulating graph topological structures by adding/removing edges, we offer a method to augment node features for better performance. We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training. By making the model invariant to small fluctuations in input data, our method helps models generalize to out-of-distribution samples and boosts model performance at test time. FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks. FLAG is also highly flexible and scalable, and is deployable with arbitrary GNN backbones and large-scale datasets. We demonstrate the efficacy and stability of our method through extensive experiments and ablation studies. We also provide intuitive observations for a deeper understanding of our method.

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Graph Property Prediction ogbg-molhiv GCN+FLAG Test ROC-AUC 0.7683 ± 0.0102 # 31
Validation ROC-AUC 0.8176 ± 0.0087 # 29
Number of params 527701 # 18
Ext. data No # 1
Graph Property Prediction ogbg-molhiv GIN+virtual node+FLAG Test ROC-AUC 0.7748 ± 0.0096 # 28
Validation ROC-AUC 0.8438 ± 0.0128 # 6
Number of params 3336306 # 8
Ext. data No # 1
Graph Property Prediction ogbg-molhiv DeeperGCN+FLAG Test ROC-AUC 0.7942 ± 0.0120 # 18
Validation ROC-AUC 0.8425 ± 0.0061 # 8
Number of params 531976 # 16
Ext. data No # 1
Graph Property Prediction ogbg-molhiv GIN+FLAG Test ROC-AUC 0.7654 ± 0.0114 # 32
Validation ROC-AUC 0.8225 ± 0.0155 # 26
Number of params 1885206 # 12
Ext. data No # 1
Graph Property Prediction ogbg-molpcba GIN+FLAG Test AP 0.2395 ± 0.0040 # 23
Validation AP 0.2451 ± 0.0042 # 23
Number of params 1923433 # 21
Ext. data No # 1
Graph Property Prediction ogbg-molpcba GCN+virtual node+FLAG Test AP 0.2483 ± 0.0037 # 21
Validation AP 0.2556 ± 0.0040 # 21
Number of params 2017028 # 19
Ext. data No # 1
Graph Property Prediction ogbg-molpcba GIN+virtual node+FLAG Test AP 0.2834 ± 0.0038 # 17
Validation AP 0.2912 ± 0.0026 # 19
Number of params 3374533 # 16
Ext. data No # 1
Graph Property Prediction ogbg-molpcba DeeperGCN+virtual node+FLAG Test AP 0.2842 ± 0.0043 # 15
Validation AP 0.2952 ± 0.0029 # 15
Number of params 5550208 # 13
Ext. data No # 1
Graph Property Prediction ogbg-molpcba GCN+FLAG Test AP 0.2116 ± 0.0017 # 26
Validation AP 0.2150 ± 0.0022 # 27
Number of params 565928 # 25
Ext. data No # 1
Graph Property Prediction ogbg-ppa GIN+FLAG Test Accuracy 0.6905 ± 0.0092 # 10
Validation Accuracy 0.6465 ± 0.0070 # 13
Number of params 1836942 # 10
Ext. data No # 1
Graph Property Prediction ogbg-ppa DeeperGCN+FLAG Test Accuracy 0.7752 ± 0.0069 # 5
Validation Accuracy 0.7484 ± 0.0052 # 5
Number of params 2336421 # 6
Ext. data No # 1
Graph Property Prediction ogbg-ppa GIN+virtual node+FLAG Test Accuracy 0.7245 ± 0.0114 # 7
Validation Accuracy 0.6789 ± 0.0079 # 7
Number of params 3288042 # 4
Ext. data No # 1
Graph Property Prediction ogbg-ppa GCN+virtual node+FLAG Test Accuracy 0.6944 ± 0.0052 # 9
Validation Accuracy 0.6638 ± 0.0055 # 9
Number of params 1930537 # 8
Ext. data No # 1
Node Property Prediction ogbn-arxiv MLP+FLAG Test Accuracy 0.5602 ± 0.0019 # 57
Validation Accuracy 0.5817 ± 0.0011 # 54
Number of params 110120 # 46
Ext. data No # 1
Node Property Prediction ogbn-arxiv GCN_res + 8 layers + FLAG Test Accuracy 0.7276 ± 0.0024 # 33
Validation Accuracy 0.7389 ± 0.0012 # 32
Number of params 155824 # 39
Ext. data No # 1
Node Property Prediction ogbn-arxiv DeeperGCN+FLAG Test Accuracy 0.7214 ± 0.0019 # 43
Validation Accuracy 0.7311 ± 0.0009 # 45
Number of params 491176 # 31
Ext. data No # 1
Node Property Prediction ogbn-arxiv GAT+FLAG Test Accuracy 0.7371 ± 0.0013 # 22
Validation Accuracy 0.7496 ± 0.0010 # 19
Number of params 1628440 # 12
Ext. data No # 1
Node Property Prediction ogbn-arxiv GraphSAGE+FLAG Test Accuracy 0.7219 ± 0.0021 # 38
Validation Accuracy 0.7349 ± 0.0009 # 37
Number of params 218664 # 36
Ext. data No # 1
Node Property Prediction ogbn-arxiv GCN+FLAG Test Accuracy 0.7204 ± 0.0020 # 45
Validation Accuracy 0.7330 ± 0.0010 # 41
Number of params 142888 # 43
Ext. data No # 1
Node Property Prediction ogbn-mag R-GCN+FLAG Test Accuracy 0.4737 ± 0.0048 # 14
Validation Accuracy 0.4835 ± 0.0036 # 14
Number of params 154366772 # 5
Ext. data No # 1
Node Property Prediction ogbn-products DeeperGCN+FLAG Test Accuracy 0.8193 ± 0.0031 # 21
Validation Accuracy 0.9221 ± 0.0037 # 25
Number of params 253743 # 29
Ext. data No # 1
Node Property Prediction ogbn-products GAT+FLAG Test Accuracy 0.8176 ± 0.0045 # 22
Validation Accuracy 0.9251 ± 0.0006 # 20
Number of params 751574 # 22
Ext. data No # 1
Node Property Prediction ogbn-products GraphSAGE+FLAG Test Accuracy 0.7936 ± 0.0057 # 34
Validation Accuracy 0.9205 ± 0.0007 # 28
Number of params 206895 # 32
Ext. data No # 1
Node Property Prediction ogbn-products MLP+FLAG Test Accuracy 0.6241 ± 0.0016 # 47
Validation Accuracy 0.7688 ± 0.0014 # 43
Number of params 103727 # 39
Ext. data No # 1
Node Property Prediction ogbn-proteins DeeperGCN+FLAG Test ROC-AUC 0.8596 ± 0.0027 # 10
Validation ROC-AUC 0.9132 ± 0.0022 # 10
Number of params 2374568 # 9
Ext. data No # 1

Methods


No methods listed for this paper. Add relevant methods here