The Normal Distributions Indistinguishability Spectrum and its Application to Privacy-Preserving Machine Learning

To achieve differential privacy (DP) one typically randomizes the output of the underlying query. In big data analytics, one often uses randomized sketching/aggregation algorithms to make processing high-dimensional data tractable. Intuitively, such machine learning (ML) algorithms should provide some inherent privacy, yet most if not all existing DP mechanisms do not leverage this inherent randomness, resulting in potentially redundant noising. The motivating question of our work is: (How) can we improve the utility of DP mechanisms for randomized ML queries, by leveraging the randomness of the query itself? Towards a (positive) answer, we prove the Normal Distributions Indistinguishability Spectrum Theorem (in short, NDIS Theorem), a theoretical result with far-reaching practical implications. In a nutshell, NDIS is a closed-form analytic computation for the $(\epsilon,\delta)$-indistinguishability-spectrum (in short, $(\epsilon,\delta)$-IS) of two arbitrary (multi-dimensional) normal distributions $X$ and $Y$, i.e., the optimal $\delta$ (for any given $\epsilon$) such that $X$ and $Y$ are ($\epsilon,\delta$)-close according to the DP distance. The NDIS theorem (1) yields efficient estimators for the above IS, and (2) allows us to analyze DP-mechanisms with normally-distributed outputs, as well as more general mechanisms by leveraging their behavior on large inputs. We apply the NDIS theorem to derive DP mechanisms for queries with normally-distributed outputs -- i.e., Gaussian Random Projections (RP) -- and for more general queries -- i.e., Ordinary Least Squares (OLS). Both RP and OLS are highly relevant in data analytics. Our new DP mechanisms achieve superior privacy/utility trade-offs by leveraging the randomness of the underlying algorithms, and identifies, for the first time, the range of $(\epsilon,\delta)$ for which no additional noising is needed.