Neural Oblivious Decision Ensembles

Neural Oblivious Decision Ensembles (NODE) is a tabular data architecture that consists of differentiable oblivious decision trees (ODT) that are trained end-to-end by backpropagation.

The core building block is a Neural Oblivious Decision Ensemble (NODE) layer. The layer is composed of $m$ differentiable oblivious decision trees (ODTs) of equal depth $d$. As an input, all $m$ trees get a common vector $x \in \mathbb{R}^{n}$, containing $n$ numeric features. Below we describe a design of a single differentiable ODT.

In its essence, an ODT is a decision table that splits the data along $d$ splitting features and compares each feature to a learned threshold. Then, the tree returns one of the $2^{d}$ possible responses, corresponding to the comparisons result. Therefore, each ODT is completely determined by its splitting features $f \in \mathbb{R}^{d}$, splitting thresholds $b \in \mathbb{R}^{d}$ and a $d$-dimensional tensor of responses $R \in \mathbb{R} \underbrace{2 \times 2 \times 2}_{d}$. In this notation, the tree output is defined as:

$$ h(x)=R\left[\mathbb{1}\left(f_{1}(x)-b_{1}\right), \ldots, \mathbb{1}\left(f_{d}(x)-b_{d}\right)\right] $$ where $\mathbb{1}(\cdot)$ denotes the Heaviside function.