Search Results for author:
Found 2 papers, 0 papers with code
Mechanics of Next Token Prediction with Self-Attention
}$ We show that training self-attention with gradient descent learns an automaton which generates the next token in two distinct steps: $\textbf{(1)}$ $\textbf{Hard}$ $\textbf{retrieval:}$ Given input sequence, self-attention precisely selects the $\textit{high-priority}$ $\textit{input}$ $\textit{tokens}$ associated with the last input token.
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
