Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

1 code implementation • 3 May 2022 • Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt

Contrastively trained image-text models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts.

CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration

no code implementations • 20 Mar 2022 • Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, Shuran Song

Employing this philosophy, we design CLIP on Wheels (CoW) baselines for the task and evaluate each zero-shot model in both Habitat and RoboTHOR simulators.

Image Classification Object Localization

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

2 code implementations • 10 Mar 2022 • Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt

In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin.

Domain Generalization Image Classification +1

Robust fine-tuning of zero-shot models

1 code implementation • 4 Sep 2021 • Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt

Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.

Transfer Learning

Learning Neural Network Subspaces

1 code implementation • 20 Feb 2021 • Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, Mohammad Rastegari

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance.

Deconstructing the Structure of Sparse Neural Networks

no code implementations • 30 Nov 2020 • Maxwell Van Gelder, Mitchell Wortsman, Kiana Ehsani

Although sparse neural networks have been studied extensively, the focus has been primarily on accuracy.

Supermasks in Superposition

1 code implementation • NeurIPS 2020 • Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi

We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting.

Soft Threshold Weight Reparameterization for Learnable Sparsity

1 code implementation • ICML 2020 • Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, Ali Farhadi

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget.

Network Pruning

What's Hidden in a Randomly Weighted Neural Network?

3 code implementations • CVPR 2020 • Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari

Training a neural network is synonymous with learning the values of the weights.

Image Classification

Discovering Neural Wirings

4 code implementations • NeurIPS 2019 • Mitchell Wortsman, Ali Farhadi, Mohammad Rastegari

In this work we propose a method for discovering neural wirings.

Feature Engineering Network Pruning +1

Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning

2 code implementations • CVPR 2019 • Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi

In this paper we study the problem of learning to learn at both training and test time in the context of visual navigation.

Meta-Learning Meta Reinforcement Learning +1