Attention Mechanisms
Locality Sensitive Hashing Attention
Introduced by Kitaev et al. in Reformer: The Efficient TransformerLSH Attention, or Locality Sensitive Hashing Attention is a replacement for dot-product attention with one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. LSH refers to a family of functions (known as LSH families) to hash data points into buckets so that data points near each other are located in the same buckets with high probability, while data points far from each other are likely to be in different buckets. It was proposed as part of the Reformer architecture.
Source: Reformer: The Efficient Transformer
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Language Modelling | 3 | 9.68% |
| Time Series Analysis | 2 | 6.45% |
| Time Series Forecasting | 2 | 6.45% |
| Sentence | 2 | 6.45% |
| Image Generation | 2 | 6.45% |
| Reinforcement Learning (RL) | 2 | 6.45% |
| Management | 1 | 3.23% |
| Reading Comprehension | 1 | 3.23% |
| BIG-bench Machine Learning | 1 | 3.23% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
