Search Results for author:
Found 20 papers, 11 papers with code
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged.
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
And, how close are we to general visual representations?
Ranked #10 on
Image Classification
on VTAB-1k
(using extra training data)
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Big Transfer (BiT): General Visual Representation Learning
We conduct detailed analysis of the main components that lead to high transfer performance.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
(using extra training data)
Scaling Vision with Sparse Mixture of Experts
We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 5-shot
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint.
From Sparse to Soft Mixtures of Experts
Sparse mixture of expert architectures (MoEs) scale model capacity without large increases in training or inference costs.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
On Robustness and Transferability of Convolutional Neural Networks
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts.
Learning to Merge Tokens in Vision Transformers
Transformers are widely applied to solve natural language understanding and computer vision tasks.
Scalable Transfer Learning with Expert Models
We explore the use of expert representations for transfer with a simple, yet effective, strategy.
Ranked #11 on
Image Classification
on VTAB-1k
(using extra training data)
Which Model to Transfer? Finding the Needle in the Growing Haystack
Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline.
Deep Ensembles for Low-Data Transfer Learning
In the low-data regime, it is difficult to train good supervised models from scratch.
Ranked #6 on
Image Classification
on VTAB-1k
(using extra training data)
The Visual Task Adaptation Benchmark
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets.
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
Sparsity-Constrained Optimal Transport
The smoothness of the objectives increases as $k$ increases, giving rise to a trade-off between convergence speed and sparsity of the optimal plan.
On the Adversarial Robustness of Mixture of Experts
We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost.
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
In this paper, we propose new differentiable and sparse top-k operators.
Routers in Vision Mixture of Experts: An Empirical Study
Routers for sparse MoEs can be further grouped into two variants: Token Choice, which matches experts to each token, and Expert Choice, which matches tokens to each expert.
