Search Results for author:
Found 65 papers, 15 papers with code
TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection
Recent successes in natural language processing have led to the proliferation of large language models (LLMs) by multiple providers.
Mechanics of Next Token Prediction with Self-Attention
}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token.
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.
FLASH: Federated Learning Across Simultaneous Heterogeneities
The key premise of federated learning (FL) is to train ML models across a diverse set of data-owners (clients), without exchanging local data.
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks
State-space models (SSMs), such as Mamba (Gu & Dao, 2023), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention.
Class-attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective
Confirming this, under a gaussian mixture setting, we show that the optimal SVM classifier for balanced accuracy needs to be adaptive to the class attributes.
Plug-and-Play Transformer Modules for Test-Time Adaptation
Parameter-efficient tuning (PET) methods such as LoRA, Adapter, and Visual Prompt Tuning (VPT) have found success in enabling adaptation to new domains by tuning small modules within a transformer model.
MeTA: Multi-source Test Time Adaptation
Test time adaptation is the process of adapting, in an unsupervised manner, a pre-trained source model to each incoming batch of the test data (i. e., without requiring a substantial portion of the test data to be available, as in traditional domain adaptation) and without access to the source data.
Noise in the reverse process improves the approximation capabilities of diffusion models
We use a control theoretic perspective by posing the approximation of the reverse process as a trajectory tracking problem.
Effective Restoration of Source Knowledge in Continual Test Time Adaptation
By restoring the knowledge from the source, it effectively corrects the negative consequences arising from the gradual deterioration of model parameters caused by ongoing shifts in the domain.
Transformers as Support Vector Machines
In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal tokens using linear constraints on the outer-products of token pairs.
Can Transformers Learn Optimal Filtering for Unknown Systems?
Particularly, we train the transformer using various distinct systems and then evaluate the performance on unseen systems with unknown dynamics.
FedYolo: Augmenting Federated Learning with Pretrained Transformers
These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.
Max-Margin Token Selection in Attention Mechanism
Interestingly, the SVM formulation of $\boldsymbol{p}$ is influenced by the support vector geometry of $\boldsymbol{v}$.
On the Role of Attention in Prompt-tuning
Despite its success in LLMs, there is limited theoretical understanding of the power of prompt-tuning and the role of the attention mechanism in prompting.
Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation
Stochastic approximation with multiple coupled sequences (MSA) has found broad applications in machine learning as it encompasses a rich class of problems including bilevel optimization (BLO), multi-level compositional optimization (MCO), and reinforcement learning (specifically, actor-critic methods).
Learning on Manifolds: Universal Approximations Properties using Geometric Controllability Conditions for Neural ODEs
To address this shortcoming, in this paper we study a class of neural ordinary differential equations that, by design, leave a given manifold invariant, and characterize their properties by leveraging the controllability properties of control affine systems.
Provable Pathways: Learning Multiple Tasks over Multiple Paths
A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers.
Stochastic Contextual Bandits with Long Horizon Rewards
The growing interest in complex decision-making and language modeling problems highlights the importance of sample-efficient learning over very long horizons.
Transformers as Algorithms: Generalization and Stability in In-context Learning
We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i. i. d.
Finite Sample Identification of Bilinear Dynamical Systems
This motivates the problem of learning bilinear systems from a single trajectory of the system's states and inputs.
Representation Learning for Context-Dependent Decision-Making
Humans are capable of adjusting to changing environments flexibly and quickly.
FedNest: Federated Bilevel, Minimax, and Compositional Optimization
Standard federated optimization methods successfully apply to stochastic problems with single-level structure.
System Identification via Nuclear Norm Regularization
Hankel regularization encourages the low-rankness of the Hankel matrix, which maps to the low-orderness of the system.
Provable and Efficient Continual Representation Learning
In continual learning (CL), the goal is to design models that can learn a sequence of tasks without catastrophic forgetting.
Towards Sample-efficient Overparameterized Meta-learning
Specifically, for (1), we first show that learning the optimal representation coincides with the problem of designing a task-aware regularization to promote inductive bias.
Non-Stationary Representation Learning in Sequential Linear Bandits
In this paper, we study representation learning for multi-task decision-making in non-stationary environments.
AutoBalance: Optimized Loss Functions for Imbalanced Data
Our experimental findings are complemented with theoretical insights on loss function design and the benefits of train-validation split.
Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds
Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves $\mathcal{O}(\sqrt{T})$ regret, which can be improved to $\mathcal{O}(polylog(T))$ with partial knowledge of the system.
Post-hoc Models for Performance Estimation of Machine Learning Inference
Estimating how well a machine learning model performs during inference is critical in a variety of scenarios (for example, to quantify uncertainty, or to choose from a library of available models).
Certainty Equivalent Quadratic Control for Markov Jump Systems
Real-world control applications often involve complex dynamics subject to abrupt changes or variations.
Generalization Guarantees for Neural Architecture Search with Train-Validation Split
In this approach, it is common to use bilevel optimization where one optimizes the model weights over the training data (inner problem) and various hyperparameters such as the configuration of the architecture over the validation data (outer problem).
Unsupervised Multi-source Domain Adaptation Without Access to Source Data
A recent line of work addressed this problem and proposed an algorithm that transfers knowledge to the unlabeled target domain from a single source model without requiring access to the source data.
Label-Imbalanced and Group-Sensitive Classification under Overparameterization
The goal in label-imbalanced and group-sensitive classification is to optimize relevant metrics such as balanced error and equal opportunity.
Provable Super-Convergence with a Large Cyclical Learning Rate
Conventional wisdom dictates that learning rate should be in the stable regime so that gradient-based algorithms don't blow up.
Sample Efficient Subspace-based Representations for Nonlinear Meta-Learning
We prove that subspace-based representations can be learned in a sample-efficient manner and provably benefit future tasks in terms of sample complexity.
Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks
Deep networks are typically trained with many more parameters than the size of the training dataset.
On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks.
Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View
Our theoretical analysis allows us to precisely characterize how the test error varies over different training algorithms, data distributions, problem dimensions as well as number of classes, inter/intra class correlations and class priors.
Unsupervised Paraphrasing via Deep Reinforcement Learning
Our evaluation also shows that PUP achieves a great trade-off between semantic similarity and diversity of expression.
Exploring Weight Importance and Hessian Bias in Model Pruning
Model pruning is an essential procedure for building compact and computationally-efficient machine learning models.
Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training
We then establish a connection between self-training based semi-supervision and the more general problem of learning with heterogenous data and weak supervision.
Finite Sample System Identification: Improved Rates and the Role of Regularization
This paper studies low-order linear system identification via regularized regression.
On the Role of Dataset Quality and Heterogeneity in Model Confidence
We demonstrate that this leads to heterogenous confidence/accuracy behavior in the test data and is poorly handled by the standard calibration algorithms.
Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systems
If the system is run by a stabilizing input policy, we show that temporally-dependent samples can be approximated by i. i. d.
GENERALIZATION GUARANTEES FOR NEURAL NETS VIA HARNESSING THE LOW-RANKNESS OF JACOBIAN
We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well.
Quickly Finding the Best Linear Model in High Dimensions
We propose projected gradient descent (PGD) algorithm to estimate the population minimizer in the finite sample regime.
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well.
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
In particular, we prove that: (i) In the first few iterations where the updates are still in the vicinity of the initialization gradient descent only fits to the correct labels essentially ignoring the noisy labels.
Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
However, in practice much more moderate levels of overparameterization seems to be sufficient and in many cases overparameterized models seem to perfectly interpolate the training data as soon as the number of parameters exceed the size of the training data by a constant factor.
Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
In this paper we demonstrate that when the loss has certain properties over a minimally small neighborhood of the initial point, first order methods such as (stochastic) gradient descent have a few intriguing properties: (1) the iterates converge at a geometric rate to a global optima even when the loss is nonconvex, (2) among all global optima of the loss the iterates converge to one with a near minimal distance to the initial point, (3) the iterates take a near direct route from the initial point to this global optima.
Stochastic Gradient Descent Learns State Equations with Nonlinear Activations
We study discrete time dynamical systems governed by the state equation $h_{t+1}=\phi(Ah_t+Bu_t)$.
Non-asymptotic Identification of LTI Systems from a Single Trajectory
We consider the problem of learning a realization for a linear time-invariant (LTI) dynamical system from input/output data.
High Dimensional Data Enrichment: Interpretable, Fast, and Data-Efficient
We consider the problem of multi-task learning in the high dimensional setting.
End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition
In this paper we study the problem of learning the weights of a deep convolutional neural network.
Learning Compact Neural Networks with Regularization
Proper regularization is critical for speeding up training, improving generalization performance, and learning compact models that are cost efficient.
Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression
Evaluations on synthetic and real datasets demonstrate that algorithm is competitive with current state-of-the-art and accurately learns feature nonlinearities.
Fast and Reliable Parameter Estimation from Nonlinear Observations
In this paper we study the problem of recovering a structured but unknown parameter ${\bf{\theta}}^*$ from $n$ nonlinear observations of the form $y_i=f(\langle {\bf{x}}_i,{\bf{\theta}}^*\rangle)$ for $i=1, 2,\ldots, n$.
Near-Optimal Bounds for Binary Embeddings of Arbitrary Sets
We characterize the tradeoff between distortion and sample complexity $m$ in terms of the Gaussian width $\omega(K)$ of the set.
Universality laws for randomized dimension reduction, with applications
In the Euclidean setting, one fundamental technique for dimension reduction is to apply a random linear map to the data.
Parallel Correlation Clustering on Big Graphs
We present C4 and ClusterWild!, two algorithms for parallel correlation clustering that run in a polylogarithmic number of rounds and achieve nearly linear speedups, provably.
Sharp Time--Data Tradeoffs for Linear Inverse Problems
We sharply characterize the convergence rate associated with a wide variety of random measurement ensembles in terms of the number of measurements and structural complexity of the signal with respect to the chosen penalty function.
Isometric sketching of any set via the Restricted Isometry Property
In this paper we show that for the purposes of dimensionality reduction certain class of structured random matrices behave similarly to random Gaussian matrices.
Graph Clustering With Missing Data: Convex Algorithms and Analysis
We consider the problem of finding clusters in an unweighted graph, when the graph is partially observed.
The Squared-Error of Generalized LASSO: A Precise Analysis
The first LASSO estimator assumes a-priori knowledge of $f(x_0)$ and is given by $\arg\min_{x}\{{\|y-Ax\|_2}~\text{subject to}~f(x)\leq f(x_0)\}$.
