Labeling Trick: A Theory of Using Graph Neural Networks for Multi-Node Representation Learning

In this paper, we provide a theory of using graph neural networks (GNNs) for multi-node representation learning (where we are interested in learning a representation for a set of more than one node, such as link). We know that GNN is designed to learn single-node representations. When we want to learn a node set representation involving multiple nodes, a common practice in previous works is to directly aggregate the single-node representations obtained by a GNN into a joint node set representation. In this paper, we show a fundamental constraint of such an approach, namely the inability to capture the dependence between nodes in the node set, and argue that directly aggregating individual node representations does not lead to an effective joint representation for multiple nodes. Then, we notice that a few previous successful works for multi-node representation learning, including SEAL, Distance Encoding, and ID-GNN, all used node labeling. These methods first label nodes in the graph according to their relationships with the target node set before applying a GNN. Then, the node representations obtained in the labeled graph are aggregated into a node set representation. By investigating their inner mechanisms, we unify these node labeling techniques into a single and most general form -- labeling trick. We prove that with labeling trick a sufficiently expressive GNN learns the most expressive node set representations, thus in principle solves any joint learning tasks over node sets. Experiments on one important two-node representation learning task, link prediction, verified our theory. Our work explains the superior performance of previous node-labeling-based methods, and establishes a theoretical foundation of using GNNs for multi-node representation learning.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Link Property Prediction ogbl-citation2 SEAL Test MRR 0.8767 ± 0.0032 # 8
Validation MRR 0.8757 ± 0.0031 # 8
Number of params 260802 # 15
Ext. data No # 1
Link Property Prediction ogbl-collab SEAL-nofeat (val as input) Test Hits@50 0.6474 ± 0.0043 # 13
Validation Hits@50 0.6495 ± 0.0043 # 13
Number of params 501570 # 17
Ext. data No # 1
Link Property Prediction ogbl-collab SEAL-nofeat Test Hits@50 0.5471 ± 0.0049 # 16
Validation Hits@50 0.6495 ± 0.0043 # 13
Number of params 501570 # 17
Ext. data No # 1
Link Property Prediction ogbl-ddi SEAL Test Hits@20 0.3056 ± 0.0386 # 24
Validation Hits@20 0.2849 ± 0.0269 # 26
Number of params 531138 # 27
Ext. data No # 1
Link Property Prediction ogbl-ppa SEAL Test Hits@100 0.4880 ± 0.0316 # 12
Validation Hits@100 0.5125 ± 0.0252 # 10
Number of params 709122 # 9
Ext. data No # 1

Methods


No methods listed for this paper. Add relevant methods here