Search Results for author:
Found 89 papers, 18 papers with code
LASER: Attention with Exponential Transformation
To this end, we introduce a new attention mechanism called LASER, which we analytically show to admit a larger gradient signal.
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
In this paper, we theoretically analyze retraining in a linearly separable setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels.
Towards Quantifying the Preconditioning Effect of Adam
Specifically, for a $d$-dimensional quadratic with a diagonal Hessian having condition number $\kappa$, we show that the effective condition number-like quantity controlling the iteration complexity of Adam without momentum is $\mathcal{O}(\min(d, \kappa))$.
Automatic Engineering of Long Prompts
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts.
FINGER: Fast Inference for Graph-based Approximate Nearest Neighbor Search
Approximate K-Nearest Neighbor Search (AKNNS) has now become ubiquitous in modern applications, for example, as a fast search procedure with two tower deep learning models.
Counterfactual Learning To Rank for Utility-Maximizing Query Autocompletion
A shortcoming of this approach is that users often do not know which query will provide the best retrieval performance on the current information retrieval system, meaning that any query autocompletion methods trained to mimic user behavior can lead to suboptimal query suggestions.
Sample Efficiency of Data Augmentation Consistency Regularization
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.
Cluster-and-Conquer: A Framework For Time-Series Forecasting
Our framework -- which we refer to as "cluster-and-conquer" -- is highly general, allowing for any time-series forecasting and clustering method to be used in each step.
Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs.
Multi Label Text Classification
Multi-Label Text Classification
+2
Label Disentanglement in Partition-based Extreme Multilabel Classification
Partition-based methods are increasingly-used in extreme multi-label classification (XMC) problems due to their scalability to large output spaces (e. g., millions or more).
Extreme Multi-label Learning for Semantic Matching in Product Search
In this paper, we aim to improve semantic product search by using tree-based XMC models where inference time complexity is logarithmic in the number of products.
Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
Geometric median (\textsc{Gm}) is a classical method in statistics for achieving a robust estimation of the uncorrupted data; under gross corruption, it achieves the optimal breakdown point of 0. 5.
Ranked #22 on
Image Classification
on MNIST
(Accuracy metric)
On the Convergence of Differentially Private Federated Learning on Non-Lipschitz Objectives, and with Normalized Client Updates
The primary reason for this is that the clipping operation (i. e., projection onto an $\ell_2$ ball of a fixed radius called the clipping threshold) for bounding the sensitivity of the average update to each client's update introduces bias depending on the clipping threshold and the number of local steps in FL, and analyzing this is not easy.
Enabling Efficiency-Precision Trade-offs for Label Trees in Extreme Classification
For such applications, a common approach is to organize these labels into a tree, enabling training and inference times that are logarithmic in the number of labels.
Combinatorial Bandits without Total Order for Arms
Specifically, we focus on a challenging setting where 1) the reward distribution of an arm depends on the set $s$ it is part of, and crucially 2) there is \textit{no total order} for the arms in $\mathcal{A}$.
Linear Bandit Algorithms with Sublinear Time Complexity
For sufficiently large $K$, our algorithms have sublinear per-step complexity and $\tilde O(\sqrt{T})$ regret.
Session-Aware Query Auto-completion using Extreme Multi-label Ranking
Previous queries in the user session can provide useful context for the user's intent and can be leveraged to suggest auto-completions that are more relevant while adhering to the user's prefix.
Faster Non-Convex Federated Learning via Global and Local Momentum
We propose \texttt{FedGLOMO}, a novel federated learning (FL) algorithm with an iteration complexity of $\mathcal{O}(\epsilon^{-1. 5})$ to converge to an $\epsilon$-stationary point (i. e., $\mathbb{E}[\|\nabla f(\bm{x})\|^2] \leq \epsilon$) for smooth non-convex functions -- under arbitrary client heterogeneity and compressed communication -- compared to the $\mathcal{O}(\epsilon^{-2})$ complexity of most prior works.
PECOS: Prediction for Enormous and Correlated Output Spaces
In this paper, we propose the Prediction for Enormous and Correlated Output Spaces (PECOS) framework, a versatile and modular machine learning framework for solving prediction problems for very large output spaces, and apply it to the eXtreme Multilabel Ranking (XMR) problem: given an input instance, find and rank the most relevant items from an enormous but fixed and finite output space.
Learning from eXtreme Bandit Feedback
In POXM, the selected actions for the sIS estimator are the top-p actions of the logging policy, where p is adjusted from the data and is significantly smaller than the size of the action space.
Non-Exhaustive, Overlapping Co-Clustering: An Extended Analysis
To solve this problem, we propose intuitive objective functions, and develop an an efficient iterative algorithm which we call the NEO-CC algorithm.
Primal-Dual Block Generalized Frank-Wolfe
We propose a generalized variant of Frank-Wolfe algorithm for solving a class of sparse/low-rank optimization problems.
Provable Non-linear Inductive Matrix Completion
Inductive matrix completion (IMC) method is a standard approach for this problem where the given query as well as the items are embedded in a common low-dimensional space.
Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure
The architecture of Transformer is based entirely on self-attention, and has been shown to outperform models that employ recurrence on sequence transduction tasks such as machine translation.
Inverting Deep Generative models, One layer at a time
For generative models of arbitrary depth, we show that exact recovery is possible in polynomial time with high probability, if the layers are expanding and the weights are randomly selected.
Primal-Dual Block Frank-Wolfe
We propose a variant of the Frank-Wolfe algorithm for solving a class of sparse/low-rank optimization problems.
AutoAssist: A Framework to Accelerate Training of Deep Neural Networks
In this paper, we propose AutoAssist, a simple framework to accelerate training of a deep neural network.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
The Limitations of Adversarial Training and the Blind-Spot Attack
In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network.
Discrete Adversarial Attacks and Submodular Optimization with Applications to Text Classification
In this paper we formulate the attacks with discrete input on a set function as an optimization task.
SeCSeq: Semantic Coding for Sequence-to-Sequence based Extreme Multi-label Classification
To circumvent the softmax bottleneck, SeCSeq compresses labels into sequences of semantic-aware compact codes, on which Seq2Seq models are trained.
Nonlinear Inductive Matrix Completion based on One-layer Neural Networks
A standard approach to modeling this problem is Inductive Matrix Completion where the predicted rating is modeled as an inner product of the user and the item features projected onto a latent space.
Towards Fast Computation of Certified Robustness for ReLU Networks
Verifying the robustness property of a general Rectified Linear Unit (ReLU) network is an NP-complete problem [Katz, Barrett, Dill, Julian and Kochenderfer CAV17].
Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it controls generalization of RNN for the classification task.
Learning Long Term Dependencies via Fourier Recurrent Units
In this paper we propose a simple recurrent architecture, the Fourier Recurrent Unit (FRU), that stabilizes the gradients that arise in its training while giving us stronger expressive power.
Realtime query completion via deep language models
Search engine users nowadays heavily depend on query completion and correction to shape their queries.
Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels
In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels.
Gradient Boosted Decision Trees for High Dimensional Sparse Output
In this paper, we study the gradient boosted decision trees (GBDT) when the output space is high dimensional and sparse.
Doubly Greedy Primal-Dual Coordinate Descent for Sparse Empirical Risk Minimization
We consider the popular problem of sparse empirical risk minimization with linear predictors and a large number of both features and observations.
Recovery Guarantees for One-hidden-layer Neural Networks
For activation functions that are also smooth, we show $\mathit{local~linear~convergence}$ guarantees of gradient descent under a resampling rule.
Similarity Preserving Representation Learning for Time Series Clustering
A considerable amount of clustering algorithms take instance-feature matrices as their inputs.
Structured Sparse Regression via Greedy Hard Thresholding
Several learning applications require solving high-dimensional regression problems where the relevant features belong to a small number of (overlapping) groups.
Coordinate-wise Power Method
The vanilla power method simultaneously updates all the coordinates of the iterate, which is essential for its convergence analysis.
Dual Decomposed Learning with Factorwise Oracle for Structural SVM of Large Output Domain
In this work, we show that, by decomposing training of Structural Support Vector Machine (SVM) into a series of multiclass SVM problems connected through messages, one can replace expensive structured oracle with Factorwise Maximization Oracle (FMO) that allows efficient implementation of complexity sublinear to the factor domain.
Asynchronous Parallel Greedy Coordinate Descent
n this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints.
Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction
We develop novel regularization schemes and use scalable matrix factorization methods that are eminently suited for high-dimensional time series data that has many missing values.
Mixed Linear Regression with Multiple Components
Furthermore, our empirical results indicate that even with random initialization, our approach converges to the global optima in linear time, providing speed-up of up to two orders of magnitude.
A Greedy Approach for Budgeted Maximum Inner Product Search
Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of a low-rank matrix factorization model for a recommender system.
Communication-Efficient Parallel Block Minimization for Kernel Machines
Kernel machines often yield superior predictive performance on various tasks; however, they suffer from severe computational challenges.
Generalized Root Models: Beyond Pairwise Graphical Models for Univariate Exponential Families
As in the recent work with square root graphical (SQR) models [Inouye et al. 2016]---which was restricted to pairwise dependencies---we give the conditions of the parameters that are needed for normalization using the radial conditionals similar to the pairwise case [Inouye et al. 2016].
PD-Sparse : A Primal and Dual Sparse Approach to Extreme Multiclass and Multilabel Classification
In this work, we show that a margin-maximizing loss with l1 penalty, in case of Extreme Classification, yields extremely sparse solution both in primal and in dual without sacrificing the expressive power of predictor.
Extreme Stochastic Variational Inference: Distributed and Asynchronous
Moreover, it requires the parameters to fit in the memory of a single processor; this is problematic when the number of parameters is in billions.
Square Root Graphical Models: Multivariate Generalizations of Univariate Exponential Families that Permit Positive Dependencies
With this motivation, we give an example of our model class derived from the univariate exponential distribution that allows for almost arbitrary positive and negative dependencies with only a mild condition on the parameter matrix---a condition akin to the positive definiteness of the Gaussian covariance matrix.
Fast Multiplier Methods to Optimize Non-exhaustive, Overlapping Clustering
In this paper, we consider two fast multiplier methods to accelerate the convergence of an augmented Lagrangian scheme: a proximal method of multipliers and an alternating direction method of multipliers (ADMM).
Collaborative Filtering with Graph Information: Consistency and Scalable Methods
Low rank matrix completion plays a fundamental role in collaborative filtering applications, the key idea being that the variables lie in a smaller subspace than the ambient space.
Ranked #1 on
Recommendation Systems
on Flixster
(using extra training data)
Consistent Multilabel Classification
In particular, we show that for multilabel metrics constructed as instance-, micro- and macro-averages, the population optimal classifier can be decomposed into binary classifiers based on the marginal instance-conditional distribution of each label, with a weak association between labels via the threshold.
Fixed-Length Poisson MRF: Adding Dependencies to the Multinomial
We show the effectiveness of our LPMRF distribution over Multinomial models by evaluating the test set perplexity on a dataset of abstracts and Wikipedia.
Sparse Linear Programming via Primal and Dual Augmented Coordinate Descent
Over the past decades, Linear Programming (LP) has been widely used in different areas and considered as one of the mature technologies in numerical optimization.
Matrix Completion with Noisy Side Information
Moreover, we study the effectof general features in theory, and show that by using our model, the sample complexity can still be lower than matrix completion as long as features are sufficiently informative.
High-dimensional Time Series Prediction with Missing Values
High-dimensional time series prediction is needed in applications as diverse as demand forecasting and climatology.
Preference Completion: Large-scale Collaborative Ranking from Pairwise Comparisons
In this paper we consider the collaborative ranking setting: a pool of users each provides a small number of pairwise preferences between $d$ possible items; from these we need to predict preferences of the users for items they have not yet seen.
Optimal Decision-Theoretic Classification Using Non-Decomposable Performance Metrics
We provide a general theoretical analysis of expected out-of-sample utility, also referred to as decision-theoretic classification, for non-decomposable binary classification metrics such as F-measure and Jaccard coefficient.
PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent
In this paper, we parallelize the SDCD algorithms in LIBLINEAR.
A Scalable Asynchronous Distributed Algorithm for Topic Modeling
Learning meaningful topic models with massive document collections which contain millions of documents and billions of tokens is challenging because of two reasons: First, one needs to deal with a large number of topics (typically in the order of thousands).
Multi-Scale Spectral Decomposition of Massive Graphs
Thus, eigenvectors of the clusters serve as good initializations to a block Lanczos algorithm that is used to compute spectral decomposition of the original graph.
Capturing Semantically Meaningful Word Dependencies with an Admixture of Poisson MRFs
We develop a fast algorithm for the Admixture of Poisson MRFs (APM) topic model and propose a novel metric to directly evaluate this model.
Consistent Binary Classification with Generalized Performance Metrics
We consider a fairly large family of performance metrics given by ratios of linear combinations of the four fundamental population quantities.
Sparse Random Feature Algorithm as Coordinate Descent in Hilbert Space
In this paper, we propose a Sparse Random Feature algorithm, which learns a sparse non-linear predictor by minimizing an $\ell_1$-regularized objective function over the Hilbert Space induced from kernel function.
Fast Prediction for Large-Scale Kernel Machines
Second, we provide a new theoretical analysis on bounding the error of the solution computed by using Nystr¨om kernel approximation method, and show that the error is related to the weighted kmeans objective function where the weights are given by the model computed from the original kernel.
QUIC & DIRTY: A Quadratic Approximation Approach for Dirty Statistical Models
In this paper, we develop a family of algorithms for optimizing superposition-structured” or “dirty” statistical estimators for high-dimensional problems involving the minimization of the sum of a smooth loss function with a hybrid regularization.
Constant Nullspace Strong Convexity and Fast Convergence of Proximal Methods under High-Dimensional Settings
State of the art statistical estimators for high-dimensional problems take the form of regularized, and hence non-smooth, convex programs.
PU Learning for Matrix Completion
For the first case, we propose a "shifted matrix completion" method that recovers M using only a subset of indices corresponding to ones, while for the second case, we propose a "biased matrix completion" method that recovers the (thresholded) binary matrix.
Proximal Quasi-Newton for Computationally Intensive L1-regularized M-estimators
We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute.
Learning with Noisy Labels
In this paper, we theoretically study the problem of binary classification in the presence of random classification noise --- the learner, instead of seeing the true labels, sees labels that have independently been flipped with some small probability.
Large Scale Distributed Sparse Precision Estimation
We consider the problem of sparse precision matrix estimation in high dimensions using the CLIME estimator, which has several desirable theoretical properties.
BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables
The l1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix even under high-dimensional settings.
A Divide-and-Conquer Solver for Kernel Support Vector Machines
We show theoretically that the support vectors identified by the subproblem solution are likely to be support vectors of the entire kernel SVM problem, provided that the problem is partitioned appropriately by kernel clustering.
Large-scale Multi-label Learning with Missing Labels
The multi-label classification problem has generated significant interest in recent years.
Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples.
Provable Inductive Matrix Completion
In addition to inductive matrix completion, we show that two other low-rank estimation problems can be studied in our framework: a) general low-rank matrix sensing using rank-1 measurements, and b) multi-label regression with missing labels.
A Divide-and-Conquer Method for Sparse Inverse Covariance Estimation
We derive a bound on the distance of the approximate solution to the true solution.
Nearest Neighbor based Greedy Coordinate Descent
In particular, we investigate the greedy coordinate descent algorithm, and note that performing the greedy step efficiently weakens the costly dependence on the problem size provided the solution is sparse.
Greedy Algorithms for Structurally Constrained High Dimensional Problems
A hallmark of modern machine learning is its ability to deal with high dimensional problems by exploiting structural assumptions that limit the degrees of freedom in the underlying model.
Orthogonal Matching Pursuit with Replacement
Our proof techniques are novel and flexible enough to also permit the tightest known analysis of popular iterative algorithms such as CoSaMP and Subspace Pursuit.
Inductive Regularized Learning of Kernel Functions
Our result shows that the learned kernel matrices parameterize a linear transformation kernel function and can be applied inductively to new data points.
Matrix Completion from Power-Law Distributed Samples
In this paper, we propose a graph theoretic approach to matrix completion that solves the problem for more realistic sampling models.
Guaranteed Rank Minimization via Singular Value Projection
Minimizing the rank of a matrix subject to affine constraints is a fundamental problem with many important applications in machine learning and statistics.
Online Metric Learning and Fast Similarity Search
Metric learning algorithms can provide useful distance functions for a variety of domains, and recent work has shown good accuracy for problems where the learner can access all distance constraints at once.
