Search Results for author:
Found 10 papers, 4 papers with code
Mechanics of Next Token Prediction with Self-Attention
}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token.
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.
Transformers as Support Vector Machines
In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal tokens using linear constraints on the outer-products of token pairs.
Max-Margin Token Selection in Attention Mechanism
Interestingly, the SVM formulation of $\boldsymbol{p}$ is influenced by the support vector geometry of $\boldsymbol{v}$.
Provable Pathways: Learning Multiple Tasks over Multiple Paths
A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers.
Stochastic Contextual Bandits with Long Horizon Rewards
The growing interest in complex decision-making and language modeling problems highlights the importance of sample-efficient learning over very long horizons.
Transformers as Algorithms: Generalization and Stability in In-context Learning
We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i. i. d.
Confident Clustering via PCA Compression Ratio and Its Application to Single-cell RNA-seq Analysis
We validate our algorithm on single-cell RNA-seq data, which is a powerful and widely used tool in biology area.
Provable and Efficient Continual Representation Learning
In continual learning (CL), the goal is to design models that can learn a sequence of tasks without catastrophic forgetting.
Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks
Deep networks are typically trained with many more parameters than the size of the training dataset.
