Search Results for author:
Found 17 papers, 7 papers with code
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Linear-Probe Classification
on ImageNet
(using extra training data)
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
In this paper, we propose new differentiable and sparse top-k operators.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint.
On the Adversarial Robustness of Mixture of Experts
We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost.
Sparsity-Constrained Optimal Transport
The smoothness of the objectives increases as $k$ increases, giving rise to a trade-off between convergence speed and sparsity of the optimal plan.
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Ranked #1 on
Image Captioning
on nocaps out-of-domain
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
Learning to Merge Tokens in Vision Transformers
Transformers are widely applied to solve natural language understanding and computer vision tasks.
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, lead to strong performance.
Scaling Vision with Sparse Mixture of Experts
We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 5-shot
Deep Ensembles for Low-Data Transfer Learning
In the low-data regime, it is difficult to train good supervised models from scratch.
Ranked #5 on
Image Classification
on VTAB-1k
(using extra training data)
Which Model to Transfer? Finding the Needle in the Growing Haystack
Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline.
Scalable Transfer Learning with Expert Models
We explore the use of expert representations for transfer with a simple, yet effective, strategy.
Ranked #10 on
Image Classification
on VTAB-1k
(using extra training data)
On Robustness and Transferability of Convolutional Neural Networks
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts.
Big Transfer (BiT): General Visual Representation Learning
We conduct detailed analysis of the main components that lead to high transfer performance.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
(using extra training data)
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
And, how close are we to general visual representations?
Ranked #9 on
Image Classification
on VTAB-1k
(using extra training data)
The Visual Task Adaptation Benchmark
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets.
