Search Results for author:
Found 45 papers, 11 papers with code
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
This is due to the computational overhead of the value model, which does not exist in ReMax.
LEMON: Lossless model expansion
Our empirical results demonstrate that LEMON reduces computational costs by 56. 7% for Vision Transformers and 33. 2% for BERT when compared to training from scratch.
How Graph Neural Networks Learn: Lessons from Training Dynamics in Function Space
A long-standing goal in deep learning has been to characterize the learning behavior of black-box models in a more interpretable manner.
AceGPT, Localizing Large Language Models in Arabic
This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models.
Restricted Generative Projection for One-Class Classification and Anomaly Detection
The core idea is to learn a mapping to transform the unknown distribution of training (normal) data to a known target distribution.
NTK-SAP: Improving neural network pruning by aligning training dynamics
Recent advances in neural tangent kernel (NTK) theory suggest that the training dynamics of large enough neural networks is closely related to the spectrum of the NTK.
Balanced Training for Sparse GANs
We also introduce a new method called balanced dynamic sparse training (ADAPT), which seeks to control the BR during GAN training to achieve a good trade-off between performance and computational cost.
Invariant Layers for Graphs with Nodes of Different Types
We further narrow the invariant network design space by addressing a question about the sizes of tensor layers necessary for function approximation on graph data.
Adversarial Rademacher Complexity of Deep Neural Networks
Specifically, we provide the first bound of adversarial Rademacher complexity of deep neural networks.
DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data
In contrast, we propose a Discriminator gradIent Gap regularized GAN (DigGAN) formulation which can be added to any existing GAN.
When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work
Third, we consider a constrained optimization formulation where the feasible region is the nice local region, and prove that every KKT point is a nearly global minimizer.
Stability Analysis and Generalization Bounds of Adversarial Training
In adversarial machine learning, deep neural networks can fit the adversarial examples on the training dataset but have poor generalization ability on the test set.
Adam Can Converge Without Any Modification On Update Rules
We point out there is a mismatch between the settings of theory and practice: Reddi et al. 2018 pick the problem after picking the hyperparameters of Adam, i. e., $(\beta_1, \beta_2)$; while practical applications often fix the problem first and then tune $(\beta_1, \beta_2)$.
Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning
We show that the performance of MetaNTK-NAS is comparable or better than the state-of-the-art NAS method designed for few-shot learning while enjoying more than 100x speedup.
Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data
In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss.
Towards Understanding the Impact of Model Size on Differential Private Classification
Then we propose a feature selection method to reduce the size of the model, based on a new metric which trades off the classification accuracy and privacy preserving.
Portfolio analysis with mean-CVaR and mean-CVaR-skewness criteria based on mean-variance mixture models
In this note, we give approximate closed form expressions for VaR and CVaR of portfolios of returns with NMVM distributions.
Federated Semi-Supervised Learning with Class Distribution Mismatch
Similar to that encountered in federated supervised learning, class distribution of labeled/unlabeled data could be non-i. i. d.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Larger Model Causes Lower Classification Accuracy Under Differential Privacy: Reason and Solution
Differential privacy (DP) is an essential technique for privacy-preserving, which works by adding random noise to the data.
Achieving Small Test Error in Mildly Overparameterized Neural Networks
Recent theoretical works on over-parameterized neural nets have focused on two aspects: optimization and generalization.
On the Landscape of Sparse Linear Networks
Network pruning, or sparse network has a long history and practical significance in modern applications.
Precondition Layer and Its Use for GANs
We use this PC-layer in two ways: 1) fixed preconditioning (FPC) adds a fixed PC-layer to all layers, and 2) adaptive preconditioning (APC) adaptively controls the strength of preconditioning.
On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method
The Barzilai-Borwein (BB) method has demonstrated great empirical success in nonlinear optimization.
RMSprop can converge with proper hyper-parameter
Removing this assumption allows us to establish a phase transition from divergence to non-divergence for RMSProp.
Towards a Better Global Loss Landscape of GANs
We also perform experiments to support our theory that RpGAN has a better landscape than separable-GAN.
A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems
We prove that the stabilized GDA algorithm can achieve an $O(1/\epsilon^2)$ iteration complexity for minimizing the pointwise maximum of a finite collection of nonconvex functions.
On the Landscape of One-hidden-layer Sparse Networks and Beyond
We show that linear networks can have no spurious valleys under special sparse structures, and non-linear networks could also admit no spurious valleys under a wide final layer.
The Global Landscape of Neural Networks: An Overview
Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity.
Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
Gradient-based meta-learning (GBML) with deep neural nets (DNNs) has become a popular approach for few-shot learning.
Distilling Object Detectors with Task Adaptive Regularization
Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization.
DEED: A General Quantization Scheme for Communication Efficiency in Bits
In the infrequent communication setting, DEED combined with Federated averaging requires a smaller total number of bits than Federated Averaging.
Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
More specifically, for a large class of over-parameterized deep neural networks with appropriate regularizers, the loss function has no bad local minima and no decreasing paths to infinity.
Optimization for deep learning: theory and algorithms
When and why can a neural network be successfully trained?
Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
More specifically, we prove that for any multi-layer network with generic input data and non-linear activation functions, sub-optimal local minima can exist, no matter how wide the network is (as long as the last hidden layer has at least two neurons).
Understanding Limitation of Two Symmetrized Orders by Worst-case Complexity
Recently, Coordinate Descent (CD) with cyclic order was shown to be $O(n^2)$ times slower than randomized versions in the worst-case.
CP-GAN: Towards a Better Global Landscape of GANs
In this work, we perform a global analysis of GANs from two perspectives: the global landscape of the outer-optimization problem and the global behavior of the gradient descent dynamics.
Off-road Autonomous Vehicles Traversability Analysis and Trajectory Planning Based on Deep Inverse Reinforcement Learning
Different cost functions of traversability analysis are learned and tested at various scenes of capability in guiding the trajectory planning of different behaviors.
Max-Sliced Wasserstein Distance and its use for GANs
Generative adversarial nets (GANs) and variational auto-encoders have significantly improved our distribution modeling capabilities, showing promise for dataset augmentation, image-to-image translation and feature learning.
On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
Wide networks are often believed to have a nice optimization landscape, but what rigorous results can we prove?
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
We prove that under our derived conditions, these methods can achieve the convergence rate of order $O(\log{T}/\sqrt{T})$ for nonconvex stochastic optimization.
Adding One Neuron Can Eliminate All Bad Local Minima
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima.
Understanding the Loss Surface of Neural Networks for Binary Classification
Here we focus on the training performance of single-layered neural networks for binary classification, and provide conditions under which the training error is zero at all local minima of a smooth hinge loss function.
Training Language Models Using Target-Propagation
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps.
Guaranteed Matrix Completion via Non-convex Factorization
In this paper, we establish a theoretical guarantee for the factorization formulation to correctly recover the underlying low-rank matrix.
