Search Results for author:
Found 55 papers, 31 papers with code
Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i. e., "foundation models"), in both vision and language domains.
From Sparse to Soft Mixtures of Experts
Sparse mixture of expert architectures (MoEs) scale model capacity without large increases in training or inference costs.
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged.
Scaling Open-Vocabulary Object Detection
However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31. 2% to 44. 6% (43% relative improvement).
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
Dual PatchNorm
We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers.
Massively Scaling Heteroscedastic Classifiers
Heteroscedastic classifiers, which learn a multivariate Gaussian distribution over prediction logits, have been shown to perform well on image classification problems with hundreds to thousands of classes.
Adaptive Computation with Elastic Input Sequence
However, most standard neural networks have a fixed function type and computation budget regardless of the sample's nature or difficulty.
CLIPPO: Image-and-Language Understanding from Pixels Only
Multimodal models are becoming increasingly effective, in part due to unified components, such as the Transformer architecture.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint.
Location-Aware Self-Supervised Transformers for Semantic Segmentation
Pixel-level labels are particularly expensive to acquire.
Transcending Scaling Laws with 0.1% Extra Compute
This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute.
Ranked #2 on
Cross-Lingual Question Answering
on TyDiQA-GoldP
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Ranked #1 on
Image Captioning
on nocaps out-of-domain
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes
We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks.
Robust and Efficient Medical Imaging with Self-Supervision
These results suggest that REMEDIS can significantly accelerate the life-cycle of medical imaging AI development thereby presenting an important step forward for medical imaging AI to deliver broad impact.
Simple Open-Vocabulary Object Detection with Vision Transformers
Combining simple architectures with large-scale pre-training has led to massive improvements in image classification.
Ranked #1 on
One-Shot Object Detection
on COCO
UL2: Unifying Language Learning Paradigms
Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
Ranked #7 on
Long-range modeling
on SCROLLS
Do better ImageNet classifiers assess perceptual similarity better?
Perceptual distances between images, as measured in the space of pre-trained deep features, have outperformed prior low-level, pixel-based metrics on assessing perceptual similarity.
Learning to Merge Tokens in Vision Transformers
Transformers are widely applied to solve natural language understanding and computer vision tasks.
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models.
The Benchmark Lottery
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods.
Revisiting the Calibration of Modern Neural Networks
Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks.
Scaling Vision with Sparse Mixture of Experts
We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 5-shot
Scaling Vision Transformers
As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90. 45% top-1 accuracy.
Ranked #3 on
Image Classification
on VTAB-1k
(using extra training data)
MLP-Mixer: An all-MLP Architecture for Vision
Convolutional Neural Networks (CNNs) are the go-to model for computer vision.
Ranked #18 on
Image Classification
on OmniBenchmark
SI-Score: An image dataset for fine-grained analysis of robustness to object location, rotation and size
Before deploying machine learning models it is critical to assess their robustness.
Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
To bridge this gap, we perform a cross-family study of the best transfer and meta learners on both a large-scale meta-learning benchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task Adaptation Benchmark, VTAB).
Supervised Transfer Learning at Scale for Medical Imaging
However, for medical imaging, the value of transfer learning is less clear.
Underspecification Presents Challenges for Credibility in Modern Machine Learning
Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited.
Ranked #1 on
Image Classification
on CIFAR-10
(using extra training data)
Deep Ensembles for Low-Data Transfer Learning
In the low-data regime, it is difficult to train good supervised models from scratch.
Ranked #6 on
Image Classification
on VTAB-1k
(using extra training data)
Representation learning from videos in-the-wild: An object-centric approach
We propose a method to learn image representations from uncurated videos.
Training general representations for remote sensing using in-domain knowledge
Automatically finding good and general remote sensing representations allows to perform transfer learning on a wide range of applications - improving the accuracy and reducing the required number of training samples.
Scalable Transfer Learning with Expert Models
We explore the use of expert representations for transfer with a simple, yet effective, strategy.
Ranked #11 on
Image Classification
on VTAB-1k
(using extra training data)
On Robustness and Transferability of Convolutional Neural Networks
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts.
Automatic Shortcut Removal for Self-Supervised Representation Learning
In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation.
Big Transfer (BiT): General Visual Representation Learning
We conduct detailed analysis of the main components that lead to high transfer performance.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
(using extra training data)
Self-Supervised Learning of Video-Induced Visual Invariances
We propose a general framework for self-supervised learning of transferable visual representations based on Video-Induced Visual Invariances (VIVI).
Ranked #15 on
Image Classification
on VTAB-1k
(using extra training data)
In-domain representation learning for remote sensing
Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community.
Ranked #1 on
Multi-Label Image Classification
on BigEarthNet
(mAP (macro) metric)
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
And, how close are we to general visual representations?
Ranked #10 on
Image Classification
on VTAB-1k
(using extra training data)
The Visual Task Adaptation Benchmark
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets.
Parameter-Efficient Transfer Learning for NLP
On GLUE, we attain within 0. 4% of the performance of full fine-tuning, adding only 3. 6% parameters per task.
Ranked #5 on
Image Classification
on OmniBenchmark
(using extra training data)
Neural Architecture Search Over a Graph Search Space
However, the success of NAS depends on the definition of the search space.
Self-Supervised GANs via Auxiliary Rotation Loss
In this work we exploit two popular unsupervised learning techniques, adversarial training and self-supervision, and take a step towards bridging the gap between conditional and unconditional GANs.
Ranked #6 on
Image Generation
on CelebA-HQ 128x128
Self-Supervised GAN to Counter Forgetting
To counter forgetting, we encourage the discriminator to maintain useful representations by adding a self-supervision.
On Self Modulation for Generative Adversarial Networks
Training Generative Adversarial Networks (GANs) is notoriously challenging.
Transfer Learning with Neural AutoML
We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks.
Analyzing Language Learned by an Active Question Answering Agent
We analyze the language learned by an agent trained with reinforcement learning as a component of the ActiveQA system [Buck et al., 2017].
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer.
A Filtering Approach to Stochastic Variational Inference
Stochastic variational inference (SVI) uses stochastic optimization to scale up Bayesian computation to massive data.
Collaborative Gaussian Processes for Preference Learning
We present a new model based on Gaussian processes (GPs) for learning pairwise preferences expressed by multiple users.
Bayesian Active Learning for Classification and Preference Learning
Information theoretic active learning has been widely studied for probabilistic models.
