Hopular: Modern Hopfield Networks for Tabular Data

1 Jun 2022  ·  Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, Sepp Hochreiter ·

While Deep Learning excels in structured data as encountered in vision and natural language processing, it failed to meet its expectations on tabular data. For tabular data, Support Vector Machines (SVMs), Random Forests, and Gradient Boosting are the best performing techniques with Gradient Boosting in the lead. Recently, we saw a surge of Deep Learning methods that were tailored to tabular data but still underperform compared to Gradient Boosting on small-sized datasets. We suggest "Hopular", a novel Deep Learning architecture for medium- and small-sized datasets, where each layer is equipped with continuous modern Hopfield networks. The modern Hopfield networks use stored data to identify feature-feature, feature-target, and sample-sample dependencies. Hopular's novelty is that every layer can directly access the original input as well as the whole training set via stored data in the Hopfield networks. Therefore, Hopular can step-wise update its current model and the resulting prediction at every layer like standard iterative learning algorithms. In experiments on small-sized tabular datasets with less than 1,000 samples, Hopular surpasses Gradient Boosting, Random Forests, SVMs, and in particular several Deep Learning methods. In experiments on medium-sized tabular data with about 10,000 samples, Hopular outperforms XGBoost, CatBoost, LightGBM and a state-of-the art Deep Learning method designed for tabular data. Thus, Hopular is a strong alternative to these methods on tabular data.

PDF Abstract


  Add Datasets introduced or used in this paper

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Benchmark
General Classification Shrutime CatBoost Accuracy 86.39 ± 0.04 # 1
General Classification Shrutime LightGBM Accuracy 86.18 ± 0.02 # 2
General Classification Shrutime XGBoost Accuracy 84.58 ± 0.00 # 5
General Classification Shrutime NPTs Accuracy 85.62 ± 0.07 # 4
General Classification Shrutime Hopular Accuracy 86.12 ± 0.09 # 3


No methods listed for this paper. Add relevant methods here