Search Results for author: Bicheng Ying

Found 14 papers, 3 papers with code

Information Exchange and Learning Dynamics over Weakly-Connected Adaptive Networks

no code implementations4 Dec 2014 Bicheng Ying, Ali H. Sayed

The paper examines the learning mechanism of adaptive agents over weakly-connected graphs and reveals an interesting behavior on how information flows through such topologies.

Clustering

Performance Limits of Stochastic Sub-Gradient Learning, Part I: Single Agent Case

no code implementations24 Nov 2015 Bicheng Ying, Ali H. Sayed

In this work and the supporting Part II, we examine the performance of stochastic sub-gradient learning strategies under weaker conditions than usually considered in the literature.

Denoising

Online Dual Coordinate Ascent Learning

no code implementations24 Feb 2016 Bicheng Ying, Kun Yuan, Ali H. Sayed

The stochastic dual coordinate-ascent (S-DCA) technique is a useful alternative to the traditional stochastic gradient-descent algorithm for solving large-scale optimization problems due to its scalability to large data sets and strong theoretical guarantees.

On the Influence of Momentum Acceleration on Online Learning

no code implementations14 Mar 2016 Kun Yuan, Bicheng Ying, Ali H. Sayed

The article examines in some detail the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size and slow adaptation regime.

Performance Limits of Stochastic Sub-Gradient Learning, Part II: Multi-Agent Case

no code implementations20 Apr 2017 Bicheng Ying, Ali H. Sayed

The analysis in Part I revealed interesting properties for subgradient learning algorithms in the context of stochastic optimization when gradient noise is present.

Stochastic Optimization

Variance-Reduced Stochastic Learning by Networked Agents under Random Reshuffling

no code implementations4 Aug 2017 Kun Yuan, Bicheng Ying, Jiageng Liu, Ali H. Sayed

For such situations, the balanced gradient computation property of AVRG becomes a real advantage in reducing idle time caused by unbalanced local data storage requirements, which is characteristic of other reduced-variance gradient algorithms.

Variance-Reduced Stochastic Learning under Random Reshuffling

no code implementations4 Aug 2017 Bicheng Ying, Kun Yuan, Ali H. Sayed

First, it resolves this open issue and provides the first theoretical guarantee of linear convergence under random reshuffling for SAGA; the argument is also adaptable to other variance-reduced algorithms.

Stochastic Learning under Random Reshuffling with Constant Step-sizes

no code implementations21 Mar 2018 Bicheng Ying, Kun Yuan, Stefan Vlaski, Ali H. Sayed

In empirical risk optimization, it has been observed that stochastic gradient implementations that rely on random reshuffling of the data achieve better performance than implementations that rely on sampling the data uniformly.

Supervised Learning Under Distributed Features

no code implementations29 May 2018 Bicheng Ying, Kun Yuan, Ali H. Sayed

This work studies the problem of learning under both large datasets and large-dimensional feature space scenarios.

On the Influence of Bias-Correction on Distributed Stochastic Optimization

no code implementations26 Mar 2019 Kun Yuan, Sulaiman A. Alghunaim, Bicheng Ying, Ali H. Sayed

It is still unknown {\em whether}, {\em when} and {\em why} these bias-correction methods can outperform their traditional counterparts (such as consensus and diffusion) with noisy gradient and constant step-sizes.

Stochastic Optimization

Communicate Then Adapt: An Effective Decentralized Adaptive Method for Deep Training

no code implementations29 Sep 2021 Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Yingya Zhang, Pan Pan, Wotao Yin

Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.

Exponential Graph is Provably Efficient for Decentralized Deep Training

2 code implementations NeurIPS 2021 Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, Wotao Yin

Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and high-quality training.

BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

2 code implementations8 Nov 2021 Bicheng Ying, Kun Yuan, Hanbin Hu, Yiming Chen, Wotao Yin

On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall $1. 2\times \sim 1. 8\times$ speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce.

DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithm

1 code implementation1 Jun 2023 Lisang Ding, Kexin Jin, Bicheng Ying, Kun Yuan, Wotao Yin

Their communication, governed by the communication topology and gossip weight matrices, facilitates the exchange of model updates.

Cannot find the paper you are looking for? You can Submit a new open access paper.