Search Results for author:
Found 3 papers, 1 papers with code
On the special role of class-selective neurons in early training
Finally, in causal experiments where we regularize against class selectivity at different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect on final accuracy.
Knowledge Distillation for Efficient Sequences of Training Runs
We find that augmenting future runs with KD from previous runs dramatically reduces the time necessary to train these models, even taking into account the overhead of KD.
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information.
Ranked #523 on
Image Classification
on ImageNet
