Search Results for author:
Found 29 papers, 13 papers with code
PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models.
PaliGemma: A versatile 3B VLM for transfer
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs.
LocCa: Visual Pretraining with Location-aware Captioners
In this paper, we propose a simple visual pretraining method with location-aware captioners (LocCa).
CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?
Interestingly, data and architectural improvements seem to mitigate the negative impact of data balancing on performance; e. g. applying M4 to SigLIP-B/16 with data quality filters improves COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and ImageNet 0-shot classification from 77% to 77. 5%!
Fractal Patterns May Illuminate the Success of Next-Token Prediction
We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not formally shown.
On Scaling Up a Multilingual Vision and Language Model
We explore the boundaries of scaling up a multilingual vision and language model both in terms of size of the components and the breadth of its training task mixture.
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger.
Ranked #2 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
Adapting to Latent Subgroup Shifts via Concepts and Proxies
We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain, and unlabeled data from the target.
FlexiViT: One Model for All Patch Sizes
Vision Transformers convert images to sequences by slicing them into patches.
Layer-Stack Temperature Scaling
Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy.
Revisiting Neural Scaling Laws in Language and Vision
The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules.
A Reduction to Binary Approach for Debiasing Multiclass Datasets
We propose a novel reduction-to-binary (R2B) approach that enforces demographic parity for multiclass classification with non-binary sensitive attributes via a reduction to a sequence of binary debiasing tasks.
Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
Diagnosing and mitigating changes in model fairness under distribution shift is an important component of the safe deployment of machine learning in healthcare settings.
Fair Wrapping for Black-box Predictions
We introduce a new family of techniques to post-process ("wrap") a black-box classifier in order to reduce its bias.
Improving the Post-hoc Calibration of Modern Neural Networks with Probe Scaling
We present "probe scaling": a post-hoc recipe for calibrating the predictions of modern neural networks.
The Impact of Reinitialization on Generalization in Convolutional Neural Networks
Recent results suggest that reinitializing a subset of the parameters of a neural network during training can improve generalization, particularly for small training sets.
A Generalized Lottery Ticket Hypothesis
We introduce a generalization to the lottery ticket hypothesis in which the notion of "sparsity" is relaxed by choosing an arbitrary basis in the space of parameters.
A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models
We present a scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNs), which we prove to be near-optimal by bounding its excess Bayes risk.
A Near-Optimal Recipe for Debiasing Trained Machine Learning Models
We present an efficient and scalable algorithm for debiasing trained models, including deep neural networks (DNNs), which we prove to be near-optimal by bounding its excess Bayes risk.
What Do Neural Networks Learn When Trained With Random Labels?
We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling.
Fair Classification via Unconstrained Optimization
In addition, it can accommodate many fairness criteria that have been previously proposed in the literature, such as equalized odds and statistical parity.
Information Theoretic Guarantees for Empirical Risk Minimization with Applications to Model Selection and Large-Scale Optimization
In this paper, we derive bounds on the mutual information of the empirical risk minimization (ERM) procedure for both 0-1 and strongly-convex loss classes.
Uniform Generalization, Concentration, and Adaptive Learning
Mathematically, this requires that the learning algorithm enjoys a small generalization risk, which is defined either in expectation or in probability.
A Mathematical Theory of Learning
Depending on the hypothesis space and how the final hypothesis is selected, we show that a learning process can be assigned a numeric score, called learning capacity, which is analogous to Shannon's channel capacity and satisfies similar interesting properties as well such as the data-processing inequality and the information-cannot-hurt inequality.
