Search Results for author:
Found 21 papers, 11 papers with code
Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i. e., "foundation models"), in both vision and language domains.
From Sparse to Soft Mixtures of Experts
Sparse mixture of expert architectures (MoEs) scale model capacity without large increases in training or inference costs.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Linear-Probe Classification
on ImageNet
(using extra training data)
On the Adversarial Robustness of Mixture of Experts
We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost.
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Ranked #1 on
Image Captioning
on nocaps out-of-domain
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
Learning to Merge Tokens in Vision Transformers
Transformers are widely applied to solve natural language understanding and computer vision tasks.
Scaling Vision with Sparse Mixture of Experts
We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 5-shot
Deep Ensembles for Low-Data Transfer Learning
In the low-data regime, it is difficult to train good supervised models from scratch.
Ranked #6 on
Image Classification
on VTAB-1k
(using extra training data)
Which Model to Transfer? Finding the Needle in the Growing Haystack
Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline.
Scalable Transfer Learning with Expert Models
We explore the use of expert representations for transfer with a simple, yet effective, strategy.
Ranked #11 on
Image Classification
on VTAB-1k
(using extra training data)
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
Uncertainty quantification for deep learning is a challenging open problem.
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
And, how close are we to general visual representations?
Ranked #10 on
Image Classification
on VTAB-1k
(using extra training data)
The Visual Task Adaptation Benchmark
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets.
Google Research Football: A Novel Reinforcement Learning Environment
Recent progress in the field of reinforcement learning has been accelerated by virtual learning environments such as video games, where novel algorithms and ideas can be quickly tested in a safe and reproducible manner.
Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation.
Practical and Consistent Estimation of f-Divergences
The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning.
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical.
Ranked #1 on
Multi-Armed Bandits
on Mushroom
Active Learning for Accurate Estimation of Linear Models
We explore the sequential decision making problem where the goal is to estimate uniformly well a number of linear models, given a shared budget of random contexts independently sampled from a known distribution.
Human Interaction with Recommendation Systems
Based on this model, we prove that naive estimators, i. e. those which ignore this feedback loop, are not consistent.
Online Active Linear Regression via Thresholding
We consider the problem of online active learning to collect data for regression modeling.
