Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking

In this paper, we study decentralized empirical risk minimization problems, where the goal to minimize a finite-sum of smooth and strongly-convex functions available over a network of nodes. We propose \textbf{\texttt{GT-SAGA}}, a stochastic first-order algorithm based on decentralized stochastic gradient tracking methods (GT) \cite{DSGT_Pu,DSGT_Xin} and a variance-reduction technique called SAGA \cite{SAGA}. We demonstrate various trade-offs and discuss scenarios in which \textbf{\texttt{GT-SAGA}} achieves superior performance (in terms of the number of local gradient computations required) with respect to existing decentralized schemes.

PDF Abstract