Search Results for author:
Found 15 papers, 8 papers with code
Transformers as Support Vector Machines
In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal tokens using linear constraints on the outer-products of token pairs.
Max-Margin Token Selection in Attention Mechanism
Interestingly, the SVM formulation of $\boldsymbol{p}$ is influenced by the support vector geometry of $\boldsymbol{v}$.
Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation
Stochastic approximation with multiple coupled sequences (MSA) has found broad applications in machine learning as it encompasses a rich class of problems including bilevel optimization (BLO), multi-level compositional optimization (MCO), and reinforcement learning (specifically, actor-critic methods).
A Penalty-Based Method for Communication-Efficient Decentralized Bilevel Programming
A key feature of the proposed algorithm is to estimate the hyper-gradient of the penalty function via decentralized computation of matrix-vector products and few vector communications, which is then integrated within an alternating algorithm to obtain finite-time convergence analysis under different convexity assumptions.
Online Bilevel Optimization: Regret Analysis of Online Alternating Gradient Methods
This paper introduces \textit{online bilevel optimization} in which a sequence of time-varying bilevel problems is revealed one after the other.
FedNest: Federated Bilevel, Minimax, and Compositional Optimization
Standard federated optimization methods successfully apply to stochastic problems with single-level structure.
Fair Community Detection and Structure Learning in Heterogeneous Graphical Models
In particular, we assume there is some community or clustering structure in the true underlying graph, and we seek to learn a sparse undirected graph and its communities from the data such that demographic groups are fairly represented within the communities.
Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds
Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves $\mathcal{O}(\sqrt{T})$ regret, which can be improved to $\mathcal{O}(polylog(T))$ with partial knowledge of the system.
Certainty Equivalent Quadratic Control for Markov Jump Systems
Real-world control applications often involve complex dynamics subject to abrupt changes or variations.
Solving a class of non-convex min-max games using adaptive momentum methods
Adaptive momentum methods have recently attracted a lot of attention for training of deep neural networks.
Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problems
In this paper, we design and analyze a new family of adaptive subgradient methods for solving an important class of weakly convex (possibly nonsmooth) stochastic optimization problems.
Grassmannian Optimization for Online Tensor Completion and Tracking with the t-SVD
We propose a new fast streaming algorithm for the tensor completion problem of imputing missing entries of a low-tubal-rank tensor using the tensor singular value decomposition (t-SVD) algebraic framework.
Regularized and Smooth Double Core Tensor Factorization for Heterogeneous Data
We introduce a general tensor model suitable for data analytic tasks for {\em heterogeneous} datasets, wherein there are joint low-rank structures within groups of observations, but also discriminative structures across different groups.
Online Distributed Estimation of Principal Eigenspaces
Principal components analysis (PCA) is a widely used dimension reduction technique with an extensive range of applications.
DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization
Adaptive gradient-based optimization methods such as \textsc{Adagrad}, \textsc{Rmsprop}, and \textsc{Adam} are widely used in solving large-scale machine learning problems including deep learning.
