Robust Ensembles of Neural Networks using Itô Processes

1 Jan 2021  ·  Sumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro Velasquez ·

Residual neural networks (ResNets) can be modeled as dynamical systems where the evolution of dynamical systems represents the inference in ResNets. We exploit this connection and the theory of stochastic dynamical systems to construct a novel ensemble of Itô processes as a new deep learning representation that is more robust than classical residual networks. An Itô process obtained by solving a suitably-formulated stochastic differential equation derived from a residual network has a probability density function that is not readily perturbed by small changes in the neural network’s inputs. Our robust stochastic Itô ensemble of neural networks achieve an accuracy of 73.91% on the CIFAR-10 dataset against the PGD attack with ε = 2.0 under the L2 norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.59%. Similarly, our stochastic Itô ensemble of neural networks achieves an accuracy of 79.66% on PGD attack with ε = 16/255 under the L∞ norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.13%. The Itô ensemble trained on ImageNet achieves an accuracy of 28.53% against PGD attacks under the L∞ norm with ε = 16/255 and accuracy of 65.74% under the L2 norm with ε = 3.0, respectively. This significantly improves state-of-the-art accuracy of 5% and 35.16% for Madry’s robustness tool against the same PGD attacks under the L∞ and L2 norms, respectively. Further, our approach achieves these high robustness values without any explicit adversarial training or a significant loss of accuracy on benign inputs.

PDF Abstract
No code implementations yet. Submit your code now

Tasks


Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods


No methods listed for this paper. Add relevant methods here