Search Results for author:
Found 39 papers, 10 papers with code
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training.
When does Privileged Information Explain Away Label Noise?
Leveraging privileged information (PI), or features available during training but not at test time, has recently been shown to be an effective method for addressing label noise.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Linear-Probe Classification
on ImageNet
(using extra training data)
Massively Scaling Heteroscedastic Classifiers
Heteroscedastic classifiers, which learn a multivariate Gaussian distribution over prediction logits, have been shown to perform well on image classification problems with hundreds to thousands of classes.
On the Adversarial Robustness of Mixture of Experts
We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost.
Plex: Towards Reliability using Pretrained Large Model Extensions
A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures.
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
Transfer and Marginalize: Explaining Away Label Noise with Privileged Information
Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e. g. the ID of the annotator that provided the label.
Predicting the utility of search spaces for black-box optimization:a simple, budget-aware approach
For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable.
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, lead to strong performance.
Deep Classifiers with Label Noise Modeling and Distance Awareness
Uncertainty estimation in deep learning has recently emerged as a crucial area of interest to advance reliability and robustness in safety-critical applications.
Scaling Vision with Sparse Mixture of Experts
We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 5-shot
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning
In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks.
Correlated Input-Dependent Label Noise in Large-Scale Image Classification
We place a multivariate Normal distributed latent variable on the final hidden layer of a neural network classifier.
Ranked #5 on
Image Classification
on WebVision-1000
Amazon SageMaker Autopilot: a white box AutoML solution at scale
AutoML systems provide a black-box solution to machine learning problems by selecting the right way of processing features, choosing an algorithm and tuning the hyperparameters of the entire pipeline.
Amazon SageMaker Automatic Model Tuning: Scalable Gradient-Free Optimization
To democratize access to machine learning systems, it is essential to automate the tuning.
Distilling Ensembles Improves Uncertainty Estimates
We seek to bridge the performance gap between batch ensembles (ensembles of deep networks with shared parameters) and deep ensembles on tasks which require not only predictions, but also uncertainty estimates for these predictions.
Training independent subnetworks for robust prediction
Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network.
Hyperparameter Ensembles for Robustness and Uncertainty Quantification
Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration.
On Mixup Regularization
We show that Mixup can be interpreted as standard empirical risk minimization estimator subject to a combination of data transformation and random perturbation of the transformed data.
Ranked #74 on
Image Classification
on ObjectNet
(using extra training data)
A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
By tuning the softmax temperature, we improve accuracy, log-likelihood and calibration on both image classification benchmarks with controlled label noise as well as Imagenet-21k which has naturally occurring label noise.
The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights.
How Good is the Bayes Posterior in Deep Neural Networks Really?
In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD.
Hydra: Preserving Ensemble Diversity for Model Distillation
As a result, the diversity of the ensemble predictions, stemming from each member, is lost.
Constrained Bayesian Optimization with Max-Value Entropy Search
We propose constrained Max-value Entropy Search (cMES), a novel information theoretic-based acquisition function implementing this formulation.
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning
Despite its simplicity, we show that our approach considerably boosts BO by reducing the size of the search space, thus accelerating the optimization of a variety of black-box optimization problems.
On the Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks.
Scalable Hyperparameter Transfer Learning
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization, such as hyperparameter optimization.
Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization.
Bayesian Optimization with Tree-structured Dependencies
The benefit of leveraging this structure is twofold: we explore the search space more efficiently and posterior inference scales more favorably with the number of observations than Gaussian Process-based approaches published in the literature.
Online optimization and regret guarantees for non-additive long-term constraints
We consider online optimization in the 1-lookahead setting, where the objective does not decompose additively over the rounds of the online game.
Adaptive Algorithms for Online Convex Optimization with Long-term Constraints
We present an adaptive online gradient descent algorithm to solve online convex optimization problems with long-term constraints , which are constraints that need to be satisfied when accumulated over a finite number of rounds T , but can be violated in intermediate rounds.
Sparse and spurious: dictionary learning with noise and outliers
A popular approach within the signal processing and machine learning communities consists in modelling signals as sparse linear combinations of atoms selected from a learned dictionary.
On The Sample Complexity of Sparse Dictionary Learning
The main goal of this paper is to provide a sample complexity estimate that controls to what extent the empirical average deviates from the cost function.
Sample Complexity of Dictionary Learning and other Matrix Factorizations
Many modern tools in machine learning and signal processing, such as sparse dictionary learning, principal component analysis (PCA), non-negative matrix factorization (NMF), $K$-means clustering, etc., rely on the factorization of a matrix obtained by concatenating high-dimensional vectors from a training collection.
Convex Relaxations for Permutation Problems
Seriation seeks to reconstruct a linear order between variables using unsorted similarity information.
A latent factor model for highly multi-relational data
While there is a large body of work focused on modeling these data, few considered modeling these multiple types of relationships jointly.
Network Flow Algorithms for Structured Sparsity
Our algorithm scales up to millions of groups and variables, and opens up a whole new range of applications for structured sparse models.
Structured Sparse Principal Component Analysis
We present an extension of sparse PCA, or sparse dictionary learning, where the sparsity patterns of all dictionary elements are structured and constrained to belong to a prespecified set of shapes.
