Exphormer: Sparse Transformers for Graphs

Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphormer consists of a sparse attention mechanism based on two mechanisms: virtual global nodes and expander graphs, whose mathematical characteristics, such as spectral expansion, pseduorandomness, and sparsity, yield graph transformers with complexity only linear in the size of the graph, while allowing us to prove desirable theoretical properties of the resulting transformer models. We show that incorporating Exphormer into the recently-proposed GraphGPS framework produces models with competitive empirical results on a wide variety of graph datasets, including state-of-the-art results on three datasets. We also show that Exphormer can scale to datasets on larger graphs than shown in previous graph transformer architectures. Code can be found at \url{https://github.com/hamed1375/Exphormer}.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Node Classification Amazon Computers Exphormer Accuracy 91.47±0.17% # 1
Node Classification AMZ Photo Exphormer Accuracy 95.35±0.22% # 1
Graph Classification CIFAR10 100k Exphormer Accuracy (%) 74.754±0.194 # 1
Node Classification CLUSTER Exphormer Accuracy 78.22±0.045 # 3
Node Classification Coauthor CS Exphormer Accuracy 94.93±0.46% # 8
Node Classification Coauthor Physics Exphormer Accuracy 96.89±0.09% # 4
Node Classification COCO-SP Exphormer macro F1 0.343±0.0008 # 2
Graph Classification MalNet-Tiny Exphormer Accuracy 94.02±0.209 # 1
Graph Classification MNIST Exphormer Accuracy 98.414±0.038 # 1
Node Classification PascalVOC-SP Exphormer macro F1 0.396±0.0027 # 2
Node Classification PATTERN Exphormer Accuracy 86.734±0.008 # 4
Link Prediction PCQM-Contact Exphormer MRR 0.3637±0.0020 # 2
Graph Classification Peptides-func Exphormer AP 0.6527±0.0043 # 16
Graph Regression Peptides-struct Exphormer MAE 0.2481±0.0007 # 9