Search Results for author:
Found 17 papers, 3 papers with code
Uncertainty in Language Models: Assessment through Rank-Calibration
Language Models (LMs) have shown promising performance in natural language generation.
Decentralized Bilevel Optimization over Graphs: Loopless Algorithmic Update and Transient Iteration Complexity
In this paper, we introduce a single-loop decentralized SBO (D-SOBA) algorithm and establish its transient iteration complexity, which, for the first time, clarifies the joint influence of network topology and data heterogeneity on decentralized bilevel algorithms.
Stochastic Controlled Averaging for Federated Learning with Communication Compression
The existing approaches either cannot accommodate arbitrary data heterogeneity or partial participation, or require stringent conditions on compression.
Momentum Benefits Non-IID Federated Learning Simply and Provably
When all clients participate in the training process, we demonstrate that incorporating momentum allows FedAvg to converge without relying on the assumption of bounded data heterogeneity even using a constant local learning rate.
Optimal Heterogeneous Collaborative Linear Regression and Contextual Bandits
MOLAR improves the dependence of the estimation error on the data dimension, compared to independent least squares estimates.
Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
Our results reveal that using independent unbiased compression can reduce the total communication cost by a factor of up to $\Theta(\sqrt{\min\{n, \kappa\}})$ when all local smoothness constants are constrained by a common upper bound, where $n$ is the number of workers and $\kappa$ is the condition number of the functions being minimized.
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
In this paper, we investigate the performance limit of distributed stochastic optimization algorithms employing communication compression.
Demystifying Disagreement-on-the-Line in High Dimensions
Evaluating the performance of machine learning models under distribution shift is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain.
Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks
The main difficulties lie in how to gauge the effectiveness when transmitting messages between two nodes via time-varying communications, and how to establish the lower bound when the network size is fixed (which is a prerequisite in stochastic optimization).
Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
While (Lu and Sa, 2021) have recently provided an optimal rate for non-convex stochastic decentralized optimization with weight matrices defined over linear graphs, the optimal rate with general weight matrices remains unclear.
Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
We establish a convergence lower bound for algorithms whether using unbiased or contractive compressors in unidirection or bidirection.
Collaborative Learning of Discrete Distributions under Heterogeneity and Communication Constraints
In modern machine learning, users often have to collaborate to learn the distribution of the data.
T-Cal: An optimal test for the calibration of predictive models
We find that detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions.
An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In this paper, we will improve the convergence analysis and rates of variance reduction under without-replacement sampling orders for composite finite-sum minimization. Our results are in two-folds.
Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
For smooth objective functions, the transient stage (which measures the number of iterations the algorithm has to experience before achieving the linear speedup stage) of D-SGD is on the order of ${\Omega}(n/(1-\beta)^2)$ and $\Omega(n^3/(1-\beta)^4)$ for strongly and generally convex cost functions, respectively, where $1-\beta \in (0, 1)$ is a topology-dependent quantity that approaches $0$ for a large and sparse network.
Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In the highly data-heterogeneous scenario, Prox-DFinito with optimal cyclic sampling can attain a sample-size-independent convergence rate, which, to our knowledge, is the first result that can match with uniform-iid-sampling with variance reduction.
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.
