Search Results for author:
Found 77 papers, 45 papers with code
Evaluating Machine Accuracy on ImageNet
We perform an in-depth evaluation of human accuracy on the ImageNet dataset.
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation.
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs.
Better Alignment with Instruction Back-and-Forth Translation
We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs).
Resolving Discrepancies in Compute-Optimal Scaling of Language Models
As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $\beta_2$ parameter is essential at lower batch sizes.
Large Scale Transfer Learning for Tabular Data via Language Modeling
In this work, we seek to narrow this gap and present TabuLa-8B, a language model for tabular prediction.
DataComp-LM: In search of the next generation of training sets for language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models.
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs).
Why are Visually-Grounded Language Models Bad at Image Classification?
Image classification is one of the most fundamental capabilities of machine vision intelligence.
Multilingual Diversity Improves Vision-Language Representations
By translating all multilingual image-text pairs from a raw web crawl to English and re-filtering them, we increase the prevalence of (translated) multilingual data in the resulting training set.
Getting it Right: Improving Spatial Consistency in Text-to-Image Models
One of the key shortcomings in current text-to-image (T2I) models is their inability to consistently generate images which faithfully follow the spatial relationships specified in the text prompt.
A Sober Look at the Robustness of CLIPs to Spurious Features
Large vision language models, such as CLIP, demonstrate impressive robustness to spurious features than single-modal models trained on ImageNet.
Language models scale reliably with over-training and on downstream tasks
Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law.
Benchmarking Distribution Shift in Tabular Data with TableShift
However, high-quality benchmarks for distribution shift in tabular machine learning tasks are still lacking despite the widespread real-world use of tabular data and differences in the models used for tabular data in comparison to text and images.
Data Filtering Networks
Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on ImageNet can yield worse training sets than a model with low ImageNet accuracy that is trained on a small amount of high-quality data.
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment.
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters.
Ranked #14 on
Visual Question Answering (VQA)
on InfiMM-Eval
Are aligned neural networks adversarially aligned?
We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force.
Neural Priming for Sample-Efficient Adaptation
Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks.
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
Stable and low-precision training for large-scale vision-language models
We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved.
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
We introduce WHOOPS!, a new dataset and benchmark for visual commonsense.
Ranked #1 on
Image-to-Text Retrieval
on WHOOPS!
(using extra training data)
The Role of Pre-training Data in Transfer Learning
We investigate the impact of pre-training data distribution on the few-shot and full fine-tuning performance using 3 pre-training methods (supervised, contrastive language-image and image-image), 7 pre-training datasets, and 9 downstream datasets.
Objaverse: A Universe of Annotated 3D Objects
Massive data corpora like WebText, Wikipedia, Conceptual Captions, WebImageText, and LAION have propelled recent dramatic progress in AI.
Reproducible scaling laws for contrastive language-image learning
To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository.
Ranked #1 on
Zero-Shot Image Classification
on Country211
(using extra training data)
Editing Models with Task Arithmetic
Changing how pre-trained models behave -- e. g., improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems.
Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation
Researchers have proposed many methods for fair and robust machine learning, but comprehensive empirical evaluation of their subgroup robustness is lacking.
Exploring The Landscape of Distributional Robustness for Question Answering Models
We conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering.
lo-fi: distributed fine-tuning without communication
When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step.
LAION-5B: An open large-scale dataset for training next generation image-text models
We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale.
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.
Ranked #4 on
Question Answering
on Bamboogle
Patching open-vocabulary models by interpolating weights
We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes.
Adversarial Scrutiny of Evidentiary Statistical Software
Responding to this need to adversarially scrutinize output from such software, we propose robust adversarial testing as an audit framework to examine the validity of evidentiary statistical software.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts.
Ranked #94 on
Image Classification
on ObjectNet
(using extra training data)
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
Ranked #1 on
Image Classification
on ImageNet V2
(using extra training data)
Robust fine-tuning of zero-shot models
Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.
Ranked #12 on
Image Classification
on ObjectNet
(using extra training data)
Retiring Adult: New Datasets for Fair Machine Learning
Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning.
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML.
Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments.
Predicting with Confidence on Unseen Distributions
Our work connects techniques from domain adaptation and predictive uncertainty literature, and allows us to predict model accuracy on challenging unseen distributions without access to labeled data.
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning.
Measuring Robustness to Natural Distribution Shifts in Image Classification
We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets.
Ranked #41 on
Domain Generalization
on VizWiz-Classification
The Effect of Natural Distribution Shift on Question Answering Models
We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data.
Neural Kernels Without Tangents
We investigate the connections between neural networks and simple building blocks in kernel space.
A Meta-Analysis of Overfitting in Machine Learning
By systematically comparing the public ranking with the final ranking, we assess how much participants adapted to the holdout set over the course of a competition.
When Robustness Doesn’t Promote Robustness: Synthetic vs. Natural Distribution Shifts on ImageNet
We conduct a large experimental comparison of various robustness metrics for image classification.
Do Image Classifiers Generalize Across Time?
Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points.
Unlabeled Data Improves Adversarial Robustness
We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning.
Model Similarity Mitigates Test Set Overuse
Excessive reuse of test data has become commonplace in today's machine learning workflows.
A Systematic Framework for Natural Perturbations from Videos
We introduce a systematic framework for quantifying the robustness of classifiers to naturally occurring perturbations of images found in videos.
Do ImageNet Classifiers Generalize to ImageNet?
We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.
Model Reconstruction from Model Explanations
We show through theory and experiment that gradient-based explanations of a model quickly reveal the model itself.
Do CIFAR-10 Classifiers Generalize to CIFAR-10?
Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models.
Adversarially Robust Generalization Requires More Data
We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.
Fast and Sample Near-Optimal Algorithms for Learning Multidimensional Histograms
We give an algorithm for this learning problem that uses $n = \tilde{O}_d(k/\epsilon^2)$ samples and runs in time $\tilde{O}_d(n)$.
On the limitations of first order approximation in GAN dynamics
This suggests that such usage of the first order approximation of the discriminator, which is a de-facto standard in all the existing GAN dynamics, might be one of the factors that makes GAN training so challenging in practice.
A Classification-Based Perspective on GAN Distributions
A fundamental, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether GANs are actually able to capture the key characteristics of the datasets they are trained on.
Learning Representations for Faster Similarity Search
Based on our experiments, we propose a number of training modifications that lead to significantly better datasets for nearest neighbor algorithms.
Graph-Sparse Logistic Regression
We introduce Graph-Sparse Logistic Regression, a new algorithm for classification for the case in which the support should be sparse but connected on a graph.
Exploring the Landscape of Spatial Robustness
The study of adversarial robustness has so far largely focused on perturbations bound in p-norms.
Communication-Efficient Distributed Learning of Discrete Distributions
For the case of structured distributions, such as k-histograms and monotone distributions, we design distributed learning algorithms that achieve significantly better communication guarantees than the naive ones, and obtain tight upper and lower bounds in several regimes.
A Classification-Based Study of Covariate Shift in GAN Distributions
A basic, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether they are truly able to capture all the fundamental characteristics of the distributions they are trained on.
On the Limitations of First-Order Approximation in GAN Dynamics
While Generative Adversarial Networks (GANs) have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, both in theory and in practice.
Towards Deep Learning Models Resistant to Adversarial Attacks
Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.
On the Fine-Grained Complexity of Empirical Risk Minimization: Kernel Methods and Neural Networks
We also give similar hardness results for computing the gradient of the empirical loss, which is the main computational burden in many non-convex learning tasks.
Fast recovery from a union of subspaces
We address the problem of recovering a high-dimensional but structured vector from linear observations in a general setting where the vector can come from an arbitrary union of subspaces.
Fast Algorithms for Segmented Regression
We study the fixed design segmented regression problem: Given noisy samples from a piecewise linear function $f$, we want to recover $f$ up to a desired accuracy in mean-squared error.
Differentially Private Learning of Structured Discrete Distributions
We investigate the problem of learning an unknown probability distribution over a discrete population from random samples.
Practical and Optimal LSH for Angular Distance
Our lower bound implies that the above LSH family exhibits a trade-off between evaluation time and quality that is close to optimal for a natural class of LSH functions.
A Nearly Optimal and Agnostic Algorithm for Properly Learning a Mixture of k Gaussians, for any Constant k
One notion of learning a GMM is proper learning: here, the goal is to find a mixture of $k$ Gaussians $\mathcal{M}$ that is close to the density $f$ of the unknown distribution from which we draw samples.
Sample-Optimal Density Estimation in Nearly-Linear Time
Let $f$ be the density function of an arbitrary univariate distribution, and suppose that $f$ is $\mathrm{OPT}$-close in $L_1$-distance to an unknown piecewise polynomial function with $t$ interval pieces and degree $d$.
