Search Results for author:
Found 61 papers, 19 papers with code
When GNNs meet symmetry in ILPs: an orbit-based feature augmentation approach
In this work, we investigate the properties of permutation equivariance and invariance in GNNs, particularly in relation to the inherent symmetry of ILP formulations.
RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques
In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs.
A novel multi-agent dynamic portfolio optimization learning system based on hierarchical deep reinforcement learning
As a result, the DRL agents cannot explore the dynamic portfolio optimization policy to improve the risk-adjusted profitability in the training process.
Enabling Scalable Oversight via Self-Evolving Critic
Despite their remarkable performance, the development of Large Language Models (LLMs) faces a critical challenge in scalable oversight: providing effective feedback for tasks where human evaluation is difficult or where LLMs outperform humans.
Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3. 5, due to a predominant focus on mainstream languages (e. g., English and Chinese).
An Efficient Unsupervised Framework for Convex Quadratic Programs via Deep Unrolling
Unlike the standard learning-to-optimize framework that requires optimization solutions generated by solvers, our unsupervised method adjusts the network weights directly from the evaluation of the primal-dual gap.
Position: On-Premises LLM Deployment Demands a Middle Path: Preserving Privacy Without Sacrificing Model Confidentiality
Current LLM customization typically relies on two deployment strategies: closed-source APIs, which require users to upload private data to external servers, and open-weight models, which allow local fine-tuning but pose misuse risks.
Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity
For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects.
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
This could avoid impairing the model performance on the fine-tuning tasks.
Adam-mini: Use Fewer Learning Rates To Gain More
Adam-mini reduces memory by cutting down the learning rate resources in Adam (i. e., $1/\sqrt{v}$).
Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization
We aim to construct a new cover that possesses two properties: 1) compatibility with adversarial examples, and 2) precision comparable to covers used in standard settings.
PDHG-Unrolled Learning-to-Optimize Method for Large-Scale Linear Programming
In this work, we propose an FOM-unrolled neural network (NN) called PDHG-Net, and propose a two-stage L2O method to solve large-scale LP problems.
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Why Transformers Need Adam: A Hessian Perspective
SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear.
Combining Transformer based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization
However, typical DRL agents for portfolio optimization cannot learn a policy that is aware of the dynamic correlation between portfolio asset returns.
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO.
LEMON: Lossless model expansion
Our empirical results demonstrate that LEMON reduces computational costs by 56. 7% for Vision Transformers and 33. 2% for BERT when compared to training from scratch.
How Graph Neural Networks Learn: Lessons from Training Dynamics
In particular, we find that the gradient descent optimization of GNNs implicitly leverages the graph structure to update the learned function, as can be quantified by a phenomenon which we call \emph{kernel-graph alignment}.
AceGPT, Localizing Large Language Models in Arabic
This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models.
Restricted Generative Projection for One-Class Classification and Anomaly Detection
The core idea is to learn a mapping to transform the unknown distribution of training (normal) data to a known target distribution.
NTK-SAP: Improving neural network pruning by aligning training dynamics
Recent advances in neural tangent kernel (NTK) theory suggest that the training dynamics of large enough neural networks is closely related to the spectrum of the NTK.
Balanced Training for Sparse GANs
We also introduce a new method called balanced dynamic sparse training (ADAPT), which seeks to control the BR during GAN training to achieve a good trade-off between performance and computational cost.
Invariant Layers for Graphs with Nodes of Different Types
We further narrow the invariant network design space by addressing a question about the sizes of tensor layers necessary for function approximation on graph data.
Adversarial Rademacher Complexity of Deep Neural Networks
Specifically, we provide the first bound of adversarial Rademacher complexity of deep neural networks.
DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data
In contrast, we propose a Discriminator gradIent Gap regularized GAN (DigGAN) formulation which can be added to any existing GAN.
When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work
Third, we consider a constrained optimization formulation where the feasible region is the nice local region, and prove that every KKT point is a nearly global minimizer.
Stability Analysis and Generalization Bounds of Adversarial Training
In adversarial machine learning, deep neural networks can fit the adversarial examples on the training dataset but have poor generalization ability on the test set.
Provable Adaptivity of Adam under Non-uniform Smoothness
We present the first convergence analysis of RR Adam without the bounded smoothness assumption.
Adam Can Converge Without Any Modification On Update Rules
We point out there is a mismatch between the settings of theory and practice: Reddi et al. 2018 pick the problem after picking the hyperparameters of Adam, i. e., $(\beta_1, \beta_2)$; while practical applications often fix the problem first and then tune $(\beta_1, \beta_2)$.
Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning
We show that the performance of MetaNTK-NAS is comparable or better than the state-of-the-art NAS method designed for few-shot learning while enjoying more than 100x speedup.
Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data
In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss.
Towards Understanding the Impact of Model Size on Differential Private Classification
Then we propose a feature selection method to reduce the size of the model, based on a new metric which trades off the classification accuracy and privacy preserving.
Portfolio analysis with mean-CVaR and mean-CVaR-skewness criteria based on mean-variance mixture models
In this note, we give approximate closed form expressions for VaR and CVaR of portfolios of returns with NMVM distributions.
Federated Semi-Supervised Learning with Class Distribution Mismatch
Similar to that encountered in federated supervised learning, class distribution of labeled/unlabeled data could be non-i. i. d.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Larger Model Causes Lower Classification Accuracy Under Differential Privacy: Reason and Solution
Differential privacy (DP) is an essential technique for privacy-preserving, which works by adding random noise to the data.
Achieving Small Test Error in Mildly Overparameterized Neural Networks
Recent theoretical works on over-parameterized neural nets have focused on two aspects: optimization and generalization.
Precondition Layer and Its Use for GANs
We use this PC-layer in two ways: 1) fixed preconditioning (FPC) adds a fixed PC-layer to all layers, and 2) adaptive preconditioning (APC) adaptively controls the strength of preconditioning.
On the Landscape of Sparse Linear Networks
Network pruning, or sparse network has a long history and practical significance in modern applications.
RMSprop can converge with proper hyper-parameter
Removing this assumption allows us to establish a phase transition from divergence to non-divergence for RMSProp.
On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method
The Barzilai-Borwein (BB) method has demonstrated great empirical success in nonlinear optimization.
Towards a Better Global Loss Landscape of GANs
We also perform experiments to support our theory that RpGAN has a better landscape than separable-GAN.
A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems
We prove that the stabilized GDA algorithm can achieve an $O(1/\epsilon^2)$ iteration complexity for minimizing the pointwise maximum of a finite collection of nonconvex functions.
On the Landscape of One-hidden-layer Sparse Networks and Beyond
We show that linear networks can have no spurious valleys under special sparse structures, and non-linear networks could also admit no spurious valleys under a wide final layer.
The Global Landscape of Neural Networks: An Overview
Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity.
Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
Gradient-based meta-learning (GBML) with deep neural nets (DNNs) has become a popular approach for few-shot learning.
Distilling Object Detectors with Task Adaptive Regularization
Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization.
DEED: A General Quantization Scheme for Communication Efficiency in Bits
In the infrequent communication setting, DEED combined with Federated averaging requires a smaller total number of bits than Federated Averaging.
Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
More specifically, for a large class of over-parameterized deep neural networks with appropriate regularizers, the loss function has no bad local minima and no decreasing paths to infinity.
Optimization for deep learning: theory and algorithms
When and why can a neural network be successfully trained?
Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
More specifically, we prove that for any multi-layer network with generic input data and non-linear activation functions, sub-optimal local minima can exist, no matter how wide the network is (as long as the last hidden layer has at least two neurons).
Understanding Limitation of Two Symmetrized Orders by Worst-case Complexity
Recently, Coordinate Descent (CD) with cyclic order was shown to be $O(n^2)$ times slower than randomized versions in the worst-case.
CP-GAN: Towards a Better Global Landscape of GANs
In this work, we perform a global analysis of GANs from two perspectives: the global landscape of the outer-optimization problem and the global behavior of the gradient descent dynamics.
Off-road Autonomous Vehicles Traversability Analysis and Trajectory Planning Based on Deep Inverse Reinforcement Learning
Different cost functions of traversability analysis are learned and tested at various scenes of capability in guiding the trajectory planning of different behaviors.
Max-Sliced Wasserstein Distance and its use for GANs
Generative adversarial nets (GANs) and variational auto-encoders have significantly improved our distribution modeling capabilities, showing promise for dataset augmentation, image-to-image translation and feature learning.
On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
Wide networks are often believed to have a nice optimization landscape, but what rigorous results can we prove?
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
We prove that under our derived conditions, these methods can achieve the convergence rate of order $O(\log{T}/\sqrt{T})$ for nonconvex stochastic optimization.
Adding One Neuron Can Eliminate All Bad Local Minima
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima.
Understanding the Loss Surface of Neural Networks for Binary Classification
Here we focus on the training performance of single-layered neural networks for binary classification, and provide conditions under which the training error is zero at all local minima of a smooth hinge loss function.
Training Language Models Using Target-Propagation
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps.
Guaranteed Matrix Completion via Non-convex Factorization
In this paper, we establish a theoretical guarantee for the factorization formulation to correctly recover the underlying low-rank matrix.
