Scaled dot-product attention is an attention mechanism where the dot products are scaled down by $\sqrt{d_k}$. Formally we have a query $Q$, a key $K$ and a value $V$ and calculate the attention as:
$$ {\text{Attention}}(Q, K, V) = \text{softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V $$
If we assume that $q$ and $k$ are $d_k$-dimensional vectors whose components are independent random variables with mean $0$ and variance $1$, then their dot product, $q \cdot k = \sum_{i=1}^{d_k} u_iv_i$, has mean $0$ and variance $d_k$. Since we would prefer these values to have variance $1$, we divide by $\sqrt{d_k}$.
Source: Attention Is All You NeedPaper | Code | Results | Date | Stars |
---|
Task | Papers | Share |
---|---|---|
RAG | 116 | 11.35% |
Retrieval | 87 | 8.51% |
Question Answering | 31 | 3.03% |
Language Modeling | 27 | 2.64% |
Language Modelling | 27 | 2.64% |
Large Language Model | 21 | 2.05% |
Information Retrieval | 20 | 1.96% |
Decoder | 18 | 1.76% |
Semantic Segmentation | 16 | 1.57% |