Search Results for author:
Found 139 papers, 24 papers with code
An Efficient Primal-Dual Prox Method for Non-Smooth Optimization
We study the non-smooth optimization problems in machine learning, where both the loss function and the regularizer are non-smooth functions.
A Simple Algorithm for Semi-supervised Learning with Improved Generalization Error Bound
In this work, we develop a simple algorithm for semi-supervised regression.
Recovering the Optimal Solution by Dual Random Projection
Random projection has been widely used in data classification.
Online Stochastic Optimization with Multiple Objectives
We first propose a projection based algorithm which attains an $O(T^{-1/3})$ convergence rate.
Semi-Crowdsourced Clustering: Generalizing Crowd Labeling by Robust Distance Metric Learning
One difficulty in learning the pairwise similarity measure is that there is a significant amount of noise and inter-worker variations in the manual annotations obtained via crowdsourcing.
Nyström Method vs Random Fourier Features: A Theoretical and Empirical Comparison
Both random Fourier features and the Nyström method have been successfully applied to efficient kernel learning.
Stochastic Gradient Descent with Only One Projection
Although many variants of stochastic gradient descent have been proposed for large-scale convex optimization, most of them require projecting the solution at {\it each} iteration to ensure that the obtained solution stays within the feasible domain.
O(logT) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions
Traditional algorithms for stochastic optimization require projecting the solution at each iteration into a given domain to ensure its feasibility.
Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections
We consider stochastic strongly convex optimization with a complex inequality constraint.
Stochastic Convex Optimization with Multiple Objectives
It leverages on the theory of Lagrangian method in constrained optimization and attains the optimal convergence rate of $[O(1/ \sqrt{T})]$ in high probability for general Lipschitz continuous objectives.
Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent
We make a progress along the line by presenting a distributed stochastic dual coordinate ascent algorithm in a star network, with an analysis of the tradeoff between computation and communication.
Analysis of Distributed Stochastic Dual Coordinate Ascent
Extraordinary performances have been observed and reported for the well-motivated updates, as referred to the practical updates, compared to the naive updates.
On Data Preconditioning for Regularized Loss Minimization
In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization.
Extracting Certainty from Uncertainty: Transductive Pairwise Classification from Pairwise Similarities
In this work, we study the problem of transductive pairwise classification from pairwise similarities~\footnote{The pairwise similarities are usually derived from some side information instead of the underlying class labels.}.
Object-centric Sampling for Fine-grained Image Classification
In contrast, this paper proposes an \emph{object-centric sampling} (OCS) scheme that samples image windows based on the object location information.
Theory of Dual-sparse Regularized Randomized Reduction
In this paper, we study randomized reduction methods, which reduce high-dimensional features into low-dimensional space by randomized methods (e. g., random projection, random hashing), for large-scale high-dimensional classification.
Analysis of Nuclear Norm Regularization for Full-rank Matrix Completion
To the best of our knowledge, this is first time such a relative bound is proved for the regularized formulation of matrix completion.
An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection
In this paper, we consider the problem of column subset selection.
Hyper-Class Augmented and Regularized Deep Learning for Fine-Grained Image Classification
We demonstrate the success of the proposed framework on two small-scale fine-grained datasets (Stanford Dogs and Stanford Cars) and on a large-scale car dataset that we collected.
Fast Sparse Least-Squares Regression with Non-Asymptotic Guarantees
In this paper, we study a fast approximation method for {\it large-scale high-dimensional} sparse least-squares regression problem by exploiting the Johnson-Lindenstrauss (JL) transforms, which embed a set of high-dimensional vectors into a low-dimensional space.
Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity
We also prove a lower bound for the number of rounds of communication for a broad class of distributed first-order methods including the proposed algorithms in this paper.
Doubly Stochastic Primal-Dual Coordinate Method for Bilinear Saddle-Point Problem
We propose a doubly stochastic primal-dual coordinate optimization algorithm for empirical risk minimization, which can be formulated as a bilinear saddle-point problem.
Online Stochastic Linear Optimization under One-bit Feedback
In this paper, we study a special bandit setting of online stochastic linear optimization, where only one-bit of information is revealed to the learner at each round.
Stochastic subGradient Methods with Linear Convergence for Polyhedral Convex Optimization
In this paper, we show that simple {Stochastic} subGradient Decent methods with multiple Restarting, named {\bf RSGD}, can achieve a \textit{linear convergence rate} for a class of non-smooth and non-strongly convex optimization problems where the epigraph of the objective function is a polyhedron, to which we refer as {\bf polyhedral convex optimization}.
Stochastic Proximal Gradient Descent for Nuclear Norm Regularization
In this paper, we utilize stochastic optimization to reduce the space complexity of convex composite optimization with a nuclear norm regularizer, where the variable is a matrix of size $m \times n$.
Sparse Learning for Large-scale and High-dimensional Data: A Randomized Convex-concave Optimization Approach
In this paper, we develop a randomized algorithm and theory for learning a sparse model from large-scale and high-dimensional data, which is usually formulated as an empirical risk minimization problem with a sparsity-inducing regularizer.
RSG: Beating Subgradient Method without Smoothness and Strong Convexity
We show that, when applied to a broad class of convex optimization problems, RSG method can find an $\epsilon$-optimal solution with a low complexity than SG method.
Improved Dropout for Shallow and Deep Learning
To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization.
Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method.
Learning Attributes Equals Multi-Source Domain Generalization
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval.
Tracking Slowly Moving Clairvoyant: Optimal Dynamic Regret of Online Learning with True and Noisy Gradient
Secondly, we present a lower bound with noisy gradient feedback and then show that we can achieve optimal dynamic regrets under a stochastic gradient feedback and two-point bandit feedback.
Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition
In particular, if the objective function $F(\mathbf w)$ in the $\epsilon$-sublevel set grows as fast as $\|\mathbf w - \mathbf w_*\|_2^{1/\theta}$, where $\mathbf w_*$ represents the closest optimal solution to $\mathbf w$ and $\theta\in(0, 1]$ quantifies the local growth rate, the iteration complexity of first-order stochastic optimization for achieving an $\epsilon$-optimal solution can be $\widetilde O(1/\epsilon^{2(1-\theta)})$, which is optimal at most up to a logarithmic factor.
Homotopy Smoothing for Non-Smooth Problems with Lower Complexity than $O(1/ε)$
In this work, we will show that the proposed HOPS achieved a lower iteration complexity of $\widetilde O(1/\epsilon^{1-\theta})$\footnote{$\widetilde O()$ suppresses a logarithmic factor.}
A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates
In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function.
Improved Dynamic Regret for Non-degenerate Functions
When multiple gradients are accessible to the learner, we first demonstrate that the dynamic regret of strongly convex functions can be upper bounded by the minimum of the path-length and the squared path-length.
Adaptive Accelerated Gradient Converging Methods under Holderian Error Bound Condition
Recent studies have shown that proximal gradient (PG) method and accelerated gradient method (APG) with restarting can enjoy a linear convergence under a weaker condition than strong convexity, namely a quadratic growth condition (QGC).
Homotopy Smoothing for Non-Smooth Problems with Lower Complexity than O(1/\epsilon)
To the best of our knowledge, this is the lowest iteration complexity achieved so far for the considered non-smooth optimization problems without strong convexity assumption.
Efficient Non-oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee
Previously, oblivious random projection based approaches that project high dimensional features onto a random subspace have been used in practice for tackling high-dimensionality challenge in machine learning.
Dynamic Regret of Strongly Adaptive Methods
To cope with changing environments, recent developments in online learning have introduced the concepts of adaptive regret and dynamic regret independently.
Empirical Risk Minimization for Stochastic Convex Optimization: $O(1/n)$- and $O(1/n^2)$-type of Risk Bounds
First, we establish an $\widetilde{O}(d/n + \sqrt{F_*/n})$ risk bound when the random function is nonnegative, convex and smooth, and the expected function is Lipschitz continuous, where $d$ is the dimensionality of the problem, $n$ is the number of samples, and $F_*$ is the minimal risk.
Hybrid safe-strong rules for efficient optimization in lasso-type problems
However, with the ultrahigh-dimensional, large-scale data sets now collected in many real-world applications, it is important to develop algorithms to solve the lasso that efficiently scale up to problems of this size.
SEP-Nets: Small and Effective Pattern Networks
By doing this, we show that previous deep CNNs such as GoogLeNet and Inception-type Nets can be compressed dramatically with marginal drop in performance.
Stochastic Convex Optimization: Faster Local Growth Implies Faster Global Convergence
In this paper, a new theory is developed for first-order stochastic convex optimization, showing that the global convergence rate is sufficiently quantified by a local growth rate of the objective function in a neighborhood of the optimal solutions.
A Simple Analysis for Exp-concave Empirical Minimization with Arbitrary Convex Regularizer
In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization.
On Noisy Negative Curvature Descent: Competing with Gradient Descent for Faster Non-convex Optimization
To the best of our knowledge, the proposed stochastic algorithm is the first one that converges to a second-order stationary point in {\it high probability} with a time complexity independent of the sample size and almost linear in dimensionality.
Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence
In this paper, we study stochastic non-convex optimization with non-convex random functions.
First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time
Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information.
Adaptive SVRG Methods under Error Bound Conditions with Unknown Growth Parameter
The most studied error bound is the quadratic error bound, which generalizes strong convexity and is satisfied by a large family of machine learning problems.
Adaptive Accelerated Gradient Converging Method under H\"{o}lderian Error Bound Condition
Recent studies have shown that proximal gradient (PG) method and accelerated gradient method (APG) with restarting can enjoy a linear convergence under a weaker condition than strong convexity, namely a quadratic growth condition (QGC).
ADMM without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization
The novelty of the proposed scheme lies at that it is adaptive to a local sharpness property of the objective function, which marks the key difference from previous adaptive scheme that adjusts the penalty parameter per-iteration based on certain conditions on iterates.
NEON+: Accelerated Gradient Methods for Extracting Negative Curvature for Non-Convex Optimization
Accelerated gradient (AG) methods are breakthroughs in convex optimization, improving the convergence rate of the gradient descent method for optimization with smooth functions.
Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set.
Learning with Non-Convex Truncated Losses by SGD
Learning with a {\it convex loss} function has been a dominating paradigm for many years.
An Aggressive Genetic Programming Approach for Searching Neural Network Structure Under Computational Constraints
In this paper, we study how to design a genetic programming approach for optimizing the structure of a CNN for a given task under limited computational resources yet without imposing strong restrictions on the search space.
EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
In this paper, we propose an Ecologically-Inspired GENetic (EIGEN) approach that uses the concept of succession, extinction, mimicry, and gene duplication to search neural network structure from scratch with poorly initialized simple network and few constraints forced during the evolution, as we assume no prior knowledge about the task domain.
Level-Set Methods for Finite-Sum Constrained Convex Optimization
To update the level parameter towards the optimality, both methods require an oracle that generates upper and lower bounds as well as an affine-minorant of the level function.
Fast Stochastic AUC Maximization with $O(1/n)$-Convergence Rate
In this paper, we consider statistical learning with AUC (area under ROC curve) maximization in the classical stochastic setting where one random data drawn from an unknown distribution is revealed at each iteration for updating the model.
SADAGRAD: Strongly Adaptive Stochastic Gradient Methods
Although the convergence rates of existing variants of ADAGRAD have a better dependence on the number of iterations under the strong convexity condition, their iteration complexities have a explicitly linear dependence on the dimensionality of the problem.
How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization
The large volume of video content and high viewing frequency demand automatic video summarization algorithms, of which a key property is the capability of modeling diversity.
Improving Sequential Determinantal Point Processes for Supervised Video Summarization
In terms of modeling, we design a new probabilistic distribution such that, when it is integrated into SeqDPP, the resulting model accepts user input about the expected length of the summary.
Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions
For example, there is still a lack of theories of convergence for SGD and its variants that use stagewise step size and return an averaged solution in practice.
A Unified Analysis of Stochastic Momentum Methods for Deep Learning
However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored.
Learning Discriminators as Energy Networks in Adversarial Learning
In this work, we propose to refine the predictions of structured prediction models by effectively integrating discriminative models into the prediction.
Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning
Min-max problems have broad applications in machine learning, including learning with non-decomposable loss and learning with robustness to data distribution.
First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
In this paper, we consider first-order convergence theory and algorithms for solving a class of non-convex non-concave min-max saddle-point problems, whose objective function is weakly convex in the variables of minimization and weakly concave in the variables of maximization.
Stochastic Optimization for DC Functions and Non-smooth Non-convex Regularizers with Non-asymptotic Convergence
In this paper, we propose new stochastic optimization algorithms and study their first-order convergence theories for solving a broad family of DC functions.
Faster Online Learning of Optimal Threshold for Consistent F-measure Optimization
To advance OFO, we propose an efficient online algorithm based on simultaneously learning a posterior probability of class and learning an optimal threshold by minimizing a stochastic strongly convex function with unknown strong convexity parameter.
Adaptive Negative Curvature Descent with Applications in Non-convex Optimization
Negative curvature descent (NCD) method has been utilized to design deterministic or stochastic algorithms for non-convex optimization aiming at finding second-order stationary points or local minima.
Stagewise Training Accelerates Convergence of Testing Error Over SGD
For convex loss functions and two classes of "nice-behaviored" non-convex objectives that are close to a convex function, we establish faster convergence of stagewise training than the vanilla SGD under the PL condition on both training error and testing error.
Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure
The main contribution of this paper is the design and analysis of new stochastic primal-dual algorithms that use a mixture of stochastic gradient updates and a logarithmic number of deterministic dual updates for solving a family of convex-concave problems with no bilinear structure assumed.
A Data Efficient and Feasible Level Set Method for Stochastic Convex Optimization with Expectation Constraints
We design a stochastic feasible level set method (SFLS) for SOECs that has low data complexity and emphasizes feasibility before convergence.
Stochastic Optimization for Non-convex Inf-Projection Problems
In this paper, we study a family of non-convex and possibly non-smooth inf-projection minimization problems, where the target objective function is equal to minimization of a joint function over another variable.
Stochastic AUC Maximization with Deep Neural Networks
In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model.
Improved Schemes for Episodic Memory-based Lifelong Learning
This view leads to two improved schemes for episodic memory based lifelong learning, called MEGA-I and MEGA-II.
Learning with Long-term Remembering: Following the Lead of Mixed Stochastic Gradient
In this paper, we introduce a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks.
A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner.
A Simple and Effective Framework for Pairwise Deep Metric Learning
To tackle this issue, we propose a simple and effective framework to sample pairs in a batch of data for updating the model.
Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $O\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.
Attacking Lifelong Learning Models with Gradient Reversion
We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects.
Minimizing Dynamic Regret and Adaptive Regret Simultaneously
To address this limitation, new performance measures, including dynamic regret and adaptive regret have been proposed to guide the design of online algorithms.
Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimization
In this paper, we bridge this gap by providing a sharp analysis of epoch-wise stochastic gradient descent ascent method (referred to as Epoch-GDA) for solving strongly convex strongly concave (SCSC) min-max problems, without imposing any additional assumption about smoothness or the function's structure.
Revisiting SGD with Increasingly Weighted Averaging: Optimization and Generalization Perspectives
It remains unclear how these averaging schemes affect the convergence of {\it both optimization error and generalization error} (two equally important components of testing error) for {\bf non-strongly convex objectives, including non-convex problems}.
Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks
In this paper, we study distributed algorithms for large-scale AUC maximization with a deep neural network as a predictive model.
Fast Objective & Duality Gap Convergence for Non-Convex Strongly-Concave Min-Max Problems with PL Condition
However, most of the existing algorithms are slow in practice, and their analysis revolves around the convergence to a nearly stationary point. We consider leveraging the Polyak-Lojasiewicz (PL) condition to design faster stochastic algorithms with stronger convergence guarantee.
Nearly Optimal Robust Method for Convex Compositional Problems with Heavy-Tailed Noise
In this paper, we propose robust stochastic algorithms for solving convex compositional problems of the form $f(\E_\xi g(\cdot; \xi)) + r(\cdot)$ by establishing {\bf sub-Gaussian confidence bounds} under weak assumptions about the tails of noise distribution, i. e., {\bf heavy-tailed noise} with bounded second-order moments.
An Online Method for A Class of Distributionally Robust Optimization with Non-Convex Objectives
In this paper, we propose a practical online method for solving a class of distributionally robust optimization (DRO) with non-convex objectives, which has important applications in machine learning for improving the robustness of neural networks.
Variance-Reduced Off-Policy Memory-Efficient Policy Search
To achieve variance-reduced off-policy-stable policy optimization, we propose an algorithm family that is memory-efficient, stochastically variance-reduced, and capable of learning from off-policy samples.
Adam$^+$: A Stochastic Method with Adaptive Variance Reduction
As a result, Adam$^+$ requires few parameter tuning, as Adam, but it enjoys a provable convergence guarantee.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Advanced Graph and Sequence Neural Networks for Molecular Property Prediction and Drug Discovery
Here we develop a suite of comprehensive machine learning methods and tools spanning different computational models, molecular representations, and loss functions for molecular property prediction and drug discovery.
Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification
Our studies demonstrate that the proposed DAM method improves the performance of optimizing cross-entropy loss by a large margin, and also achieves better performance than optimizing the existing AUC square loss on these medical image classification tasks.
Ranked #2 on
Multi-Label Classification
on CheXpert
Attentional-Biased Stochastic Gradient Descent
In this paper, we present a simple yet effective provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning.
Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity
Deep AUC (area under the ROC curve) Maximization (DAM) has attracted much attention recently due to its great potential for imbalanced data classification.
Revisiting Smoothed Online Learning
Moreover, when the hitting cost is both convex and $\lambda$-quadratic growth, we reduce the competitive ratio to $1 + \frac{2}{\sqrt{\lambda}}$ by minimizing the weighted sum of the hitting cost and the switching cost.
Online Convex Optimization with Continuous Switching Constraint
To control the switching cost, we introduce the problem of online convex optimization with continuous switching constraint, where the goal is to achieve a small regret given a budget on the \emph{overall} switching cost.
Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence
Compared with AUROC, AUPRC is a more appropriate metric for highly imbalanced datasets.
A Novel Convergence Analysis for Algorithms of the Adam Family and Beyond
Our analysis exhibits that an increasing or large enough "momentum" parameter for the first-order moment used in practice is sufficient to ensure Adam and its many variants converge under a mild boundness condition on the adaptive scaling factor of the step size.
Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimization
Although numerous studies have proposed stochastic algorithms for solving these problems, they are limited in two perspectives: (i) their sample complexities are high, which do not match the state-of-the-art result for non-convex stochastic optimization; (ii) their algorithms are tailored to problems with only one lower-level problem.
Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
In this paper, we provide a comprehensive generalization analysis of stochastic gradient methods for minimax problems under both convex-concave and nonconvex-nonconcave cases through the lens of algorithmic stability.
A Simple yet Universal Strategy for Online Convex Optimization
In this paper, we propose a simple strategy for universal online convex optimization, which avoids these limitations.
Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
The proposed algorithms require sampling a constant number of tasks and data samples per iteration, making them suitable for the continual learning scenario.
Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
In this paper, we further improve the stochastic optimization of AURPC by (i) developing novel stochastic momentum methods with a better iteration complexity of $O(1/\epsilon^4)$ for finding an $\epsilon$-stationary solution; and (ii) designing a novel family of stochastic adaptive methods with the same iteration complexity, which enjoy faster convergence in practice.
Compositional Training for End-to-End Deep AUC Maximization
The key idea of compositional training is to minimize a compositional objective function, where the outer function corresponds to an AUC loss and the inner function represents a gradient descent step for minimizing a traditional loss, e. g., the cross-entropy (CE) loss.
Deep AUC Maximization for Medical Image Classification: Challenges and Opportunities
In this extended abstract, we will present and discuss opportunities and challenges brought about by a new deep learning method by AUC maximization (aka \underline{\bf D}eep \underline{\bf A}UC \underline{\bf M}aximization or {\bf DAM}) for medical image classification.
Simple Stochastic and Online Gradient DescentAlgorithms for Pairwise Learning
A popular approach to handle streaming data in pairwise learning is an online gradient descent (OGD) algorithm, where one needs to pair the current instance with a buffering set of previous instances with a sufficiently large size and therefore suffers from a scalability issue.
Simple Stochastic and Online Gradient Descent Algorithms for Pairwise Learning
A popular approach to handle streaming data in pairwise learning is an online gradient descent (OGD) algorithm, where one needs to pair the current instance with a buffering set of previous instances with a sufficiently large size and therefore suffers from a scalability issue.
A Novel Convergence Analysis for Algorithms of the Adam Family
Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam.
Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity
We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification that are formulated from distributionally robust optimization (DRO) perspective, where the uncertainty in the given label information are modeled and captured by taking the worse case of distributional weights.
Optimal Algorithms for Stochastic Multi-Level Compositional Optimization
To address these limitations, we propose a Stochastic Multi-level Variance Reduction method (SMVR), which achieves the optimal sample complexity of $\mathcal{O}\left(1 / \epsilon^{3}\right)$ to find an $\epsilon$-stationary point for non-convex objectives.
Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applications
This paper studies stochastic optimization for a sum of compositional functions, where the inner-level function of each summand is coupled with the corresponding summation index.
Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence
To the best of our knowledge, this is the first time that stochastic algorithms are proposed to optimize NDCG with a provable convergence guarantee.
Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
In this paper, we study contrastive learning from an optimization perspective, aiming to analyze and address a fundamental issue of existing contrastive learning methods that either rely on a large batch size or a large dictionary of feature vectors.
When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee
In this paper, we propose systematic and efficient gradient-based methods for both one-way and two-way partial AUC (pAUC) maximization that are applicable to deep learning.
Large-scale Optimization of Partial AUC in a Range of False Positive Rates
The partial AUC, as a generalization of the AUC, summarizes only the TPRs over a specific range of the FPRs and is thus a more suitable performance measure in many real-world situations.
Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices
(i) We benchmark a variety of loss functions with different algorithmic choices for deep AUROC optimization problem.
AUC Maximization in the Era of Big Data and AI: A Survey
We also identify and discuss remaining and emerging issues for deep AUC maximization, and provide suggestions on topics for future work.
Smoothed Online Convex Optimization Based on Discounted-Normal-Predictor
In this paper, we investigate an online prediction strategy named as Discounted-Normal-Predictor (Kapralov and Panigrahy, 2010) for smoothed online convex optimization (SOCO), in which the learner needs to minimize not only the hitting cost but also the switching cost.
Algorithmic Foundations of Empirical X-risk Minimization
This manuscript introduces a new optimization framework for machine learning and AI, named {\bf empirical X-risk minimization (EXM)}.
Multi-block Min-max Bilevel Optimization with Applications in Multi-task Deep AUC Maximization
To tackle this challenge, we present a single-loop randomized stochastic algorithm, which requires updates for only a constant number of blocks at each iteration.
GraphFM: Improving Large-Scale GNN Training via Feature Momentum
GraphFM-IB applies FM to in-batch sampled data, while GraphFM-OB applies FM to out-of-batch data that are 1-hop neighborhood of in-batch data.
Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
The key issue is to track and estimate a sequence of $\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$ across iterations, where $\mathbf g(\mathbf{w})$ has $m$ blocks and it is only allowed to probe $\mathcal{O}(1)$ blocks to attain their stochastic values and Jacobians.
Stochastic Constrained DRO with a Complexity Independent of Sample Size
Distributionally Robust Optimization (DRO), as a popular method to train robust models against distribution shift between training and test sets, has received tremendous attention in recent years.
Fairness via Adversarial Attribute Neighbourhood Robust Learning
Improving fairness between privileged and less-privileged sensitive attribute groups (e. g, {race, gender}) has attracted lots of attention.
FeDXL: Provable Federated Learning for Deep X-Risk Optimization
To this end, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples.
Stochastic Methods for AUC Optimization subject to AUC-based Fairness Constraints
In this work, we formulate the training problem of a fairness-aware machine learning model as an AUC optimization problem subject to a class of AUC-based fairness constraints.
Stochastic Approximation Approaches to Group Distributionally Robust Optimization
First, we formulate GDRO as a stochastic convex-concave saddle-point problem, and demonstrate that stochastic mirror descent (SMD), using $m$ samples in each iteration, achieves an $O(m (\log m)/\epsilon^2)$ sample complexity for finding an $\epsilon$-optimal solution, which matches the $\Omega(m/\epsilon^2)$ lower bound up to a logarithmic factor.
Generalization Analysis for Contrastive Representation Learning
For self-bounding Lipschitz loss functions, we further improve our results by developing optimistic bounds which imply fast rates in a low noise condition.
Provable Multi-instance Deep AUC Maximization with Stochastic Pooling
This paper considers a novel application of deep AUC maximization (DAM) for multi-instance learning (MIL), in which a single class label is assigned to a bag of instances (e. g., multiple 2D slices of a CT scan for a patient).
Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization
In this paper, we aim to optimize a contrastive loss with individualized temperatures in a principled and systematic manner for self-supervised learning.
Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization
In this paper, we consider non-convex multi-block bilevel optimization (MBBO) problems, which involve $m\gg 1$ lower level problems and have important applications in machine learning.
LibAUC: A Deep Learning Library for X-Risk Optimization
This paper introduces the award-winning deep learning (DL) library called LibAUC for implementing state-of-the-art algorithms towards optimizing a family of risk functions named X-risks.
Learning Unnormalized Statistical Models via Compositional Optimization
Learning unnormalized statistical models (e. g., energy-based models) is computationally challenging due to the complexity of handling the partition function.
Stability and Generalization of Stochastic Compositional Gradient Descent Algorithms
Then, we establish the compositional uniform stability results for two popular stochastic compositional gradient descent algorithms, namely SCGD and SCSC.
Everything Perturbed All at Once: Enabling Differentiable Graph Attacks
As powerful tools for representation learning on graphs, graph neural networks (GNNs) have played an important role in applications including social networks, recommendation systems, and online web services.
Non-Smooth Weakly-Convex Finite-sum Coupled Compositional Optimization
This paper investigates new families of compositional optimization problems, called $\underline{\bf n}$on-$\underline{\bf s}$mooth $\underline{\bf w}$eakly-$\underline{\bf c}$onvex $\underline{\bf f}$inite-sum $\underline{\bf c}$oupled $\underline{\bf c}$ompositional $\underline{\bf o}$ptimization (NSWC FCCO).
AUC-mixup: Deep AUC Maximization with Mixup
While deep AUC maximization (DAM) has shown remarkable success on imbalanced medical tasks, e. g., chest X-rays classification and skin lesions classification, it could suffer from severe overfitting when applied to small datasets due to its aggressive nature of pushing prediction scores of positive data away from that of negative data.
ALEXR: An Optimal Single-Loop Algorithm for Convex Finite-Sum Coupled Compositional Stochastic Optimization
This paper revisits a class of convex Finite-Sum Coupled Compositional Stochastic Optimization (cFCCO) problems with many applications, including group distributionally robust optimization (GDRO), learning with imbalanced data, reinforcement learning, and learning to rank.
Multimodal Pretraining of Medical Time Series and Notes
In downstream tasks, including in-hospital mortality prediction and phenotyping, our pretrained model outperforms baselines in settings where only a fraction of the data is labeled, emphasizing its ability to enhance ICU data analysis.
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs.
Accelerating Deep Learning with Millions of Classes
In our experiment, we demonstrate that the proposed frame-work is able to train deep learning models with millions of classes and achieve above 10×speedup compared to existing approaches.
