Search Results for author:
Found 44 papers, 10 papers with code
Scalable Bilevel Loss Balancing for Multi-Task Learning
Multi-task learning (MTL) has been widely adopted for its ability to simultaneously learn multiple tasks.
ReGNet: Reciprocal Space-Aware Long-Range Modeling and Multi-Property Prediction for Crystals
We introduce Reciprocal Geometry Network (ReGNet), a novel architecture that integrates geometric GNNs and reciprocal blocks to model short-range and long-range interactions, respectively.
Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements
Diffusion models have emerged as a powerful foundation model for visual generation.
Meta-Learning with Heterogeneous Tasks
Meta-learning is a general approach to equip machine learning models with the ability to handle few-shot scenarios when dealing with many tasks.
Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis
To the best of our knowledge, our methods are the first to completely eliminate the need for stepsize tuning, while achieving theoretical guarantees.
Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? Theoretical Insights and a Remedial Approach
This analysis reveals that naive FT methods struggle with forgetting because the pretrained model retains information about the forgetting data, and the fine-tuning process has no impact on this retained information.
Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy
The framework of IL consists of three primary components: a neural module, a reasoning engine, and a memory system.
MGDA Converges under Generalized Smoothness, Provably
We provide a comprehensive convergence analysis of these algorithms and show that they converge to an $\epsilon$-accurate Pareto stationary point with a guaranteed $\epsilon$-level average CA distance (i. e., the gap between the updating direction and the CA direction) over all iterations, where totally $\mathcal{O}(\epsilon^{-2})$ and $\mathcal{O}(\epsilon^{-4})$ samples are needed for deterministic and stochastic settings, respectively.
Understanding Forgetting in Continual Learning with Linear Regression
In this paper, we provide a general theoretical analysis of forgetting in the linear regression model via Stochastic Gradient Descent (SGD) applicable to both underparameterized and overparameterized regimes.
Theoretical Study of Conflict-Avoidant Multi-Objective Reinforcement Learning
However, these methods often suffer from the issue of \textit{gradient conflict} such that the tasks with larger gradients dominate the update direction, resulting in a performance degeneration on other tasks.
Multi-Objective Reinforcement Learning
reinforcement-learning
+1
Fair Resource Allocation in Multi-Task Learning
Inspired by fair resource allocation in communication networks, we formulate the optimization of MTL as a utility maximization problem, where the loss decreases across tasks are maximized under different fairness measurements.
Discriminative Adversarial Unlearning
We consider the scenario of two networks, the attacker $\mathbf{A}$ and the trained defender $\mathbf{D}$ pitted against each other in an adversarial objective, wherein the attacker aims at teasing out the information of the data to be unlearned in order to infer membership, and the defender unlearns to defend the network against the attack, whilst preserving its general performance.
Achieving ${O}(ε^{-1.5})$ Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization
In this paper, we revisit the bilevel optimization problem, in which the upper-level objective function is generally nonconvex and the lower-level objective function is strongly convex.
SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning
Federated bilevel optimization (FBO) has shown great potential recently in machine learning and edge computing due to the emerging nested optimization structure in meta-learning, fine-tuning, hyperparameter tuning, etc.
Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithms
In this paper, we propose a new direction-oriented multi-objective problem by regularizing the common descent direction within a neighborhood of a direction that optimizes a linear combination of objectives such as the average loss in MTL.
Achieving Linear Speedup in Non-IID Federated Bilevel Learning
However, several important properties in federated learning such as the partial client participation and the linear speedup for convergence (i. e., the convergence rate and complexity are improved linearly with respect to the number of sampled clients) in the presence of non-i. i. d.~datasets, still remain open.
Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation
Federated bilevel optimization has attracted increasing attention due to emerging machine learning and communication applications.
Network Utility Maximization with Unknown Utility Functions: A Distributed, Data-Driven Bilevel Optimization Approach
In this paper, we provide a new solution using a distributed and data-driven bilevel optimization approach, where the lower level is a distributed network utility maximization (NUM) algorithm with concave surrogate utility functions, and the upper level is a data-driven learning algorithm to find the best surrogate utility functions that maximize the sum of true network utility.
Will Bilevel Optimizers Benefit from Loops
Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough to enable rigorous comparison among different implementations.
Data Sampling Affects the Complexity of Online SGD over Dependent Data
Moreover, we show that online SGD with mini-batch sampling can further substantially improve the sample complexity over online SGD with periodic data-subsampling over highly dependent data.
A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima
Existing algorithms designed for such a problem were applicable to restricted situations and do not come with a full guarantee of convergence.
Efficiently Escaping Saddle Points in Bilevel Optimization
Moreover, we propose an inexact NEgative-curvature-Originated-from-Noise Algorithm (iNEON), a pure first-order algorithm that can escape saddle point and find local minimum of stochastic bilevel optimization.
On the Convergence Theory for Hessian-Free Bilevel Algorithms
Bilevel optimization has arisen as a powerful tool in modern machine learning.
ES-Based Jacobian Enables Faster Bilevel Optimization
Bilevel optimization (BO) has arisen as a powerful tool for solving many modern machine learning problems.
How to Improve Sample Complexity of SGD over Highly Dependent Data?
Specifically, with a $\phi$-mixing model that captures both exponential and polynomial decay of the data dependence over time, we show that SGD with periodic data-subsampling achieves an improved sample complexity over the standard SGD in the full spectrum of the $\phi$-mixing data dependence.
Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
For the problem-based formulation, we provide a convergence rate analysis for AID- and ITD-based bilevel algorithms.
Provably Faster Algorithms for Bilevel Optimization
Bilevel optimization has been widely applied in many important machine learning applications such as hyperparameter optimization and meta-learning.
Lower Bounds and Accelerated Algorithms for Bilevel Optimization
Bilevel optimization has recently attracted growing interests due to its wide applications in modern machine learning problems.
Bilevel Optimization: Convergence Analysis and Enhanced Design
For the AID-based method, we orderwisely improve the previous convergence rate analysis due to a more practical parameter selection as well as a warm start strategy, and for the ITD-based method we establish the first theoretical convergence rate.
Boosting One-Point Derivative-Free Online Optimization via Residual Feedback
As a result, our regret bounds are much tighter compared to existing regret bounds for ZO with conventional one-point feedback, which suggests that ZO with residual feedback can better track the optimizer of online optimization problems.
Provably Faster Algorithms for Bilevel Optimization and Applications to Meta-Learning
For the AID-based method, we orderwisely improve the previous finite-time convergence analysis due to a more practical parameter selection as well as a warm start strategy, and for the ITD-based method we establish the first theoretical convergence rate.
A New One-Point Residual-Feedback Oracle For Black-Box Learning and Control
When optimizing a deterministic Lipschitz function, we show that the query complexity of ZO with the proposed one-point residual feedback matches that of ZO with the existing two-point schemes.
Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
Although model-agnostic meta-learning (MAML) is a very successful algorithm in meta-learning practice, it can have high computational cost because it updates all model parameters over both the inner loop of task-specific adaptation and the outer-loop of meta initialization training.
Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization
Our APG-restart is designed to 1) allow for adopting flexible parameter restart schemes that cover many existing ones; 2) have a global sub-linear convergence rate in nonconvex and nonsmooth optimization; and 3) have guaranteed convergence to a critical point and have various types of asymptotic convergence rates depending on the parameterization of local geometry in nonconvex and nonsmooth optimization.
Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
As a popular meta-learning approach, the model-agnostic meta-learning (MAML) algorithm has been widely used due to its simplicity and effectiveness.
Robust Stochastic Bandit Algorithms under Probabilistic Unbounded Adversarial Attack
This paper investigates the attack model where an adversary attacks with a certain probability at each round, and its attack value can be arbitrary and unbounded if it attacks.
SpiderBoost and Momentum: Faster Variance Reduction Algorithms
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization.
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
Two types of zeroth-order stochastic algorithms have recently been designed for nonconvex optimization respectively based on the first-order techniques SVRG and SARAH/SPIDER.
History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms
In this paper, we propose a novel scheme, which eliminates backtracking line search but still exploits the information along optimization path by adapting the batch size via history stochastic gradients.
Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization
In this paper, we develop novel momentum schemes with flexible coefficient settings to accelerate SPIDER for nonconvex and nonsmooth composite optimization, and show that the resulting algorithms achieve the near-optimal gradient oracle complexity for achieving a generalized first-order stationary condition.
Minimax Estimation of Neural Net Distance
An important class of distance metrics proposed for training generative adversarial networks (GANs) is the integral probability metric (IPM), in which the neural net distance captures the practical GAN training via two neural networks.
SpiderBoost and Momentum: Faster Stochastic Variance Reduction Algorithms
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization.
When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset.
Learning Latent Features with Pairwise Penalties in Low-Rank Matrix Completion
Low-rank matrix completion has achieved great success in many real-world data applications.
