Search Results for author:
Found 26 papers, 15 papers with code
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters.
Improving Multimodal Datasets with Image Captioning
Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo.
TaskWeb: Selecting Better Source Tasks for Multi-task NLP
TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target, and to choose a subset of helpful training tasks for multi-task learning.
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
Reproducible scaling laws for contrastive language-image learning
To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository.
Ranked #4 on
Zero-Shot Cross-Modal Retrieval
on Flickr30k
(Image-to-text R@5 metric)
Editing Models with Task Arithmetic
Changing how pre-trained models behave -- e. g., improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems.
Adaptive Testing of Computer Vision Models
Vision models often fail systematically on groups of data that share common semantic characteristics (e. g., rare objects or unusual scenes), but identifying these failure modes is a challenge.
Exploring The Landscape of Distributional Robustness for Question Answering Models
We conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering.
Patching open-vocabulary models by interpolating weights
We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes.
Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts.
Ranked #93 on
Image Classification
on ObjectNet
(using extra training data)
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
(using extra training data)
Robust fine-tuning of zero-shot models
Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.
Ranked #10 on
Image Classification
on ObjectNet
(using extra training data)
Recognizing Multimodal Entailment
How information is created, shared and consumed has changed rapidly in recent decades, in part thanks to new social platforms and technologies on the web.
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.
MultiModalQA: Complex Question Answering over Text, Tables and Images
When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources.
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning.
Finetuning Pretrained Transformers into RNNs
Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune.
Ranked #2 on
Machine Translation
on WMT2017 Chinese-English
High Performance Natural Language Processing
Scale has played a central role in the rapid progress natural language processing has enjoyed in recent years.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Probing Contextual Language Models for Common Ground with Visual Representations
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
We publicly release all of our experimental data, including training and validation scores for 2, 100 trials, to encourage further analysis of training dynamics during fine-tuning.
Large-scale representation learning from visually grounded untranscribed speech
Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning.
General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
We address fundamental flaws in previously used metrics and show how Dynamic Time Warping (DTW), a long known method of measuring similarity between two time series, can be used for evaluation of navigation agents.
