Search Results for author:
Found 26 papers, 4 papers with code
From Local SGD to Local Fixed Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed point algorithms.
Acceleration for Compressed Gradient Descent in Distributed Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on sparsification or quantization of communicated messages are becoming increasingly popular.
Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL).
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
We propose a new variant of the Adam optimizer called MicroAdam that specifically minimizes memory overheads, while maintaining theoretical convergence guarantees.
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs.
Federated Learning is Better with Non-Homomorphic Encryption
One of the popular methodologies is employing Homomorphic Encryption (HE) - a breakthrough in privacy-preserving computation from Cryptography.
Adaptive Compression for Communication-Efficient Distributed Training
We propose Adaptive Compressed Gradient Descent (AdaCGD) - a novel optimization algorithm for communication-efficient training of supervised machine learning models with adaptive compression level.
Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition
Stein Variational Gradient Descent (SVGD) is an important alternative to the Langevin-type algorithms for sampling from probability distributions of the form $\pi(x) \propto \exp(-V(x))$.
Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks
We consider the task of minimizing the sum of smooth and strongly convex functions stored in a decentralized manner across the nodes of a communication network whose links are allowed to change in time.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator.
Optimal Client Sampling for Federated Learning
We show that importance can be measured using only the norm of the update and give a formula for optimal client participation.
Variance-Reduced Methods for Machine Learning
Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago.
Adaptive Learning of the Optimal Batch Size of SGD
Recent advances in the theoretical understanding of SGD led to a formula for the optimal batch size minimizing the number of effective data passes, i. e., the number of iterations times the batch size.
Variance Reduced Coordinate Descent with Acceleration: New Method With a Surprising Application to Finite-Sum Problems
We propose an accelerated version of stochastic variance reduced coordinate descent -- ASVRCD.
RSN: Randomized Subspace Newton
We develop a randomized Newton method capable of solving learning problems with huge dimensional feature spaces, which is a common setting in applications such as medical imaging, genomics and seismology.
Natural Compression for Distributed Deep Learning
Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa.
SGD: General Analysis and Improved Rates
By specializing our theorem to different mini-batching strategies, such as sampling with replacement and independent sampling, we derive exact expressions for the stepsize as a function of the mini-batch size.
Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop
A key structural element in both of these methods is the inclusion of an outer loop at the beginning of which a full pass over the training data is made in order to compute the exact gradient, which is then used to construct a variance-reduced estimator of the gradient.
SEGA: Variance Reduction via Gradient Sketching
In each iteration, SEGA updates the current estimate of the gradient through a sketch-and-project operation using the information provided by the latest sketch, and this is subsequently used to compute an unbiased estimate of the true gradient through a random relaxation procedure.
Randomized Block Cubic Newton Method
To this effect we propose and analyze a randomized block cubic Newton (RBCN) method, which in each iteration builds a model of the objective function formed as the sum of the natural models of its three components: a linear model with a quadratic regularizer for the differentiable term, a quadratic model with a cubic regularizer for the twice differentiable term, and perfect (proximal) model for the nonsmooth term.
Weighted Low-Rank Approximation of Matrices and Background Modeling
We primarily study a special a weighted low-rank approximation of matrices and then apply it to solve the background modeling problem.
Online and Batch Supervised Background Estimation via L1 Regression
We propose a surprisingly simple model for supervised video background estimation.
Quartz: Randomized Dual Coordinate Ascent with Arbitrary Sampling
We study the problem of minimizing the average of a large number of smooth convex functions penalized with a strongly convex regularizer.
Matrix Completion under Interval Uncertainty
Matrix completion under interval uncertainty can be cast as matrix completion with element-wise box constraints.
Separable Approximations and Decomposition Methods for the Augmented Lagrangian
In this paper we study decomposition methods based on separable approximations for minimizing the augmented Lagrangian.
