Data-efficient Image Transformer
Introduced by Touvron et al. in Training data-efficient image transformers & distillation through attentionA Data-Efficient Image Transformer is a type of Vision Transformer for image classification tasks. The model is trained using a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention.
Source: Training data-efficient image transformers & distillation through attention
Latest Papers
| PAPER | DATE |
|---|---|
|
Going deeper with Image Transformers
• • • • |
2021-03-31 |
|
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• • |
2021-03-27 |
|
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• • • • • |
2021-03-19 |
|
Transformer in Transformer
• • • • • |
2021-02-27 |
|
Conditional Positional Encodings for Vision Transformers
• • • • • • |
2021-02-22 |
|
Training data-efficient image transformers & distillation through attention
• • • • • |
2020-12-23 |
Tasks
| TASK | PAPERS | SHARE |
|---|---|---|
| Image Classification | 6 | 75.00% |
| Fine-Grained Image Classification | 2 | 25.00% |
Attention Dropout
Dropout
Feedforward Network
Multi-Head Attention
Scaled Dot-Product Attention