Search Results for author:
Found 6 papers, 1 papers with code
Why Transformers Need Adam: A Hessian Perspective
SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear.
Federated Learning with Lossy Distributed Source Coding: Analysis and Optimization
We then conduct an FL convergence analysis to connect the aggregation distortion and the FL convergence performance.
The Global Landscape of Neural Networks: An Overview
Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity.
Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
More specifically, we prove that for any multi-layer network with generic input data and non-linear activation functions, sub-optimal local minima can exist, no matter how wide the network is (as long as the last hidden layer has at least two neurons).
On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
Wide networks are often believed to have a nice optimization landscape, but what rigorous results can we prove?
Sparsity Learning Based Multiuser Detection in Grant-Free Massive-Device Multiple Access
In this work, we study the multiuser detection (MUD) problem for a grant-free massive-device multiple access (MaDMA) system, where a large number of single-antenna user devices transmit sporadic data to a multi-antenna base station (BS).
