Search Results for author:
Found 3 papers, 1 papers with code
Mechanics of Next Token Prediction with Self-Attention
}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token.
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.
Transformers as Algorithms: Generalization and Stability in In-context Learning
We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i. i. d.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
