TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Graph Classification	D&D	TokenGT	Accuracy	73.950±3.361	# 48
Molecular Property Prediction	ESOL	TokenGT	RMSE	0.667±0.103	# 12
Molecular Property Prediction	ESOL	TokenGT	R2	0.892±0.036	# 10
Graph Regression	ESR2	TokenGT	R2	0.641±0.000	# 8
Graph Regression	ESR2	TokenGT	RMSE	0.529±0.641	# 8
Graph Regression	F2	TokenGT	R2	0.872±0.000	# 8
Graph Regression	F2	TokenGT	RMSE	0.363±0.872	# 8
Molecular Property Prediction	FreeSolv	TokenGT	RMSE	1.038±0.125	# 8
Molecular Property Prediction	FreeSolv	TokenGT	R2	0.930±0.018	# 8
Graph Classification	IMDb-B	TokenGT	Accuracy	80.250±3.304	# 6
Graph Regression	KIT	TokenGT	R2	0.800±0.000	# 8
Graph Regression	KIT	TokenGT	RMSE	0.486±0.800	# 8
Graph Regression	Lipophilicity	TokenGT	RMSE	0.852±0.023	# 16
Graph Regression	Lipophilicity	TokenGT	R2	0.545±0.024	# 10
Graph Classification	NCI1	TokenGT	Accuracy	76.740±2.054	# 46
Graph Classification	NCI109	TokenGT	Accuracy	72.077±1.883	# 33
Graph Regression	PARP1	TokenGT	R2	0.907±0.000	# 8
Graph Regression	PARP1	TokenGT	RMSE	0.383±0.907	# 8
Graph Regression	PCQM4Mv2-LSC	TokenGT	Validation MAE	0.0910	# 17
Graph Regression	PCQM4Mv2-LSC	TokenGT	Test MAE	0.0919	# 10
Graph Regression	Peptides-struct	TokenGT	MAE	0.2489±0.0013	# 21
Graph Regression	PGR	TokenGT	R2	0.684±0.000	# 5
Graph Regression	PGR	TokenGT	RMSE	0.543±0.684	# 5
Graph Regression	ZINC-full	TokenGT	Test MAE	0.047±0.010	# 13

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-pgr)](https://paperswithcode.com/sota/graph-regression-on-pgr?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-classification-on-imdb-b)](https://paperswithcode.com/sota/graph-classification-on-imdb-b?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-esr2)](https://paperswithcode.com/sota/graph-regression-on-esr2?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-f2)](https://paperswithcode.com/sota/graph-regression-on-f2?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/molecular-property-prediction-on-freesolv)](https://paperswithcode.com/sota/molecular-property-prediction-on-freesolv?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-kit)](https://paperswithcode.com/sota/graph-regression-on-kit?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-parp1)](https://paperswithcode.com/sota/graph-regression-on-parp1?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/molecular-property-prediction-on-esol)](https://paperswithcode.com/sota/molecular-property-prediction-on-esol?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-zinc-full)](https://paperswithcode.com/sota/graph-regression-on-zinc-full?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-lipophilicity)](https://paperswithcode.com/sota/graph-regression-on-lipophilicity?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-pcqm4mv2-lsc)](https://paperswithcode.com/sota/graph-regression-on-pcqm4mv2-lsc?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-regression-on-peptides-struct)](https://paperswithcode.com/sota/graph-regression-on-peptides-struct?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-classification-on-nci109)](https://paperswithcode.com/sota/graph-classification-on-nci109?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-classification-on-nci1)](https://paperswithcode.com/sota/graph-classification-on-nci1?p=pure-transformers-are-powerful-graph-learners)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/pure-transformers-are-powerful-graph-learners/graph-classification-on-dd)](https://paperswithcode.com/sota/graph-classification-on-dd?p=pure-transformers-are-powerful-graph-learners)`

Pure Transformers are Powerful Graph Learners

6 Jul 2022 · Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong ·

We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer. With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN). When trained on a large-scale graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias. Our implementation is available at https://github.com/jw9730/tokengt.

PDF Abstract

Code

Add Remove Mark official

jw9730/tokengt official

342

luis-mueller/wl-transformers

Tasks

Add Remove

Graph Classification

Graph Learning

Graph Regression

Inductive Bias

Molecular Property Prediction

Datasets

IMDB-BINARY NCI1

ZINC

NCI109

Long Range Graph Benchmark (LRGB) ESOL (Estimated SOLubility) FreeSolv (Free Solvation) dockstring

PCQM4Mv2-LSC

Results from the Paper

Edit

Ranked #5 on Graph Regression on PGR

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Graph Classification	D&D	TokenGT	Accuracy	73.950±3.361	# 48	Compare
Molecular Property Prediction	ESOL	TokenGT	RMSE	0.667±0.103	# 12	Compare
Molecular Property Prediction	ESOL	TokenGT	R2	0.892±0.036	# 10	Compare
Graph Regression	ESR2	TokenGT	R2	0.641±0.000	# 8	Compare
Graph Regression	ESR2	TokenGT	RMSE	0.529±0.641	# 8	Compare
Graph Regression	F2	TokenGT	R2	0.872±0.000	# 8	Compare
Graph Regression	F2	TokenGT	RMSE	0.363±0.872	# 8	Compare
Molecular Property Prediction	FreeSolv	TokenGT	RMSE	1.038±0.125	# 8	Compare
Molecular Property Prediction	FreeSolv	TokenGT	R2	0.930±0.018	# 8	Compare
Graph Classification	IMDb-B	TokenGT	Accuracy	80.250±3.304	# 6	Compare
Graph Regression	KIT	TokenGT	R2	0.800±0.000	# 8	Compare
Graph Regression	KIT	TokenGT	RMSE	0.486±0.800	# 8	Compare
Graph Regression	Lipophilicity	TokenGT	RMSE	0.852±0.023	# 16	Compare
Graph Regression	Lipophilicity	TokenGT	R2	0.545±0.024	# 10	Compare
Graph Classification	NCI1	TokenGT	Accuracy	76.740±2.054	# 46	Compare
Graph Classification	NCI109	TokenGT	Accuracy	72.077±1.883	# 33	Compare
Graph Regression	PARP1	TokenGT	R2	0.907±0.000	# 8	Compare
Graph Regression	PARP1	TokenGT	RMSE	0.383±0.907	# 8	Compare
Graph Regression	PCQM4Mv2-LSC	TokenGT	Validation MAE	0.0910	# 17	Compare
Graph Regression	PCQM4Mv2-LSC	TokenGT	Test MAE	0.0919	# 10	Compare
Graph Regression	Peptides-struct	TokenGT	MAE	0.2489±0.0013	# 21	Compare
Graph Regression	PGR	TokenGT	R2	0.684±0.000	# 5	Compare
Graph Regression	PGR	TokenGT	RMSE	0.543±0.684	# 5	Compare
Graph Regression	ZINC-full	TokenGT	Test MAE	0.047±0.010	# 13	Compare

Methods

Add Remove

Absolute Position Encodings • Adam • Attention • BPE • Dense Connections • Dropout • Graph Transformer • Label Smoothing • LapEigen • Laplacian PE • Layer Normalization • Linear Layer • Multi-Head Attention • Position-Wise Feed-Forward Layer • Residual Connection • Softmax • Transformer

Edit Social Preview

Pure Transformers are Powerful Graph Learners

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove