Search Results for author: Simon Kornblith

Found 33 papers, 16 papers with code

Interpretability of artificial neural network models in artificial Intelligence vs. neuroscience

no code implementations7 Jun 2022 Kohitij Kar, Simon Kornblith, Evelina Fedorenko

Given the widespread calls to improve the interpretability of AI systems, we here highlight these different notions of interpretability and argue that the neuroscientific interpretability of ANNs can be pursued in parallel with, but independently from, the ongoing efforts in AI.

Decision Making

Decoder Denoising Pretraining for Semantic Segmentation

no code implementations23 May 2022 Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, Mohammad Norouzi

We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder.

Denoising Semantic Segmentation

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

3 code implementations10 Mar 2022 Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.

 Ranked #1 on Domain Generalization on ImageNet-Sketch (using extra training data)

Domain Generalization Image Classification +2

On the Origins of the Block Structure Phenomenon in Neural Network Representations

1 code implementation15 Feb 2022 Thao Nguyen, Maithra Raghu, Simon Kornblith

Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.

Meta-Learning to Improve Pre-Training

no code implementations NeurIPS 2021 Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, David Duvenaud

Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains.

Data Augmentation Hyperparameter Optimization +1

Generalized Shape Metrics on Neural Representations

1 code implementation NeurIPS 2021 Alex H. Williams, Erin Kunz, Simon Kornblith, Scott W. Linderman

In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations

no code implementations29 Sep 2021 Thao Nguyen, Maithra Raghu, Simon Kornblith

Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.

Robust fine-tuning of zero-shot models

2 code implementations CVPR 2022 Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt

Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.

Transfer Learning

Do Vision Transformers See Like Convolutional Neural Networks?

4 code implementations NeurIPS 2021 Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy

Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Classification Image Classification +1

MIST: Multiple Instance Spatial Transformer

1 code implementation CVPR 2021 Baptiste Angles, Yuhe Jin, Simon Kornblith, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Image Reconstruction

Big Self-Supervised Models Advance Medical Image Classification

no code implementations ICCV 2021 Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, Mohammad Norouzi

Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis.

Contrastive Learning General Classification +3

Demystifying Loss Functions for Classification

no code implementations1 Jan 2021 Simon Kornblith, Honglak Lee, Ting Chen, Mohammad Norouzi

It is common to use the softmax cross-entropy loss to train neural networks on classification datasets where a single class label is assigned to each example.

Classification General Classification +1

Boosting Contrastive Self-Supervised Learning with False Negative Cancellation

1 code implementation23 Nov 2020 Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, Maryam Khademi

While positive pairs can be generated reliably (e. g., as different views of the same image), it is difficult to accurately establish negative pairs, defined as samples from different images regardless of their semantic content or visual features.

Contrastive Learning Representation Learning +3

Teaching with Commentaries

1 code implementation ICLR 2021 Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, Geoffrey Hinton

We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.

Data Augmentation

Why Do Better Loss Functions Lead to Less Transferable Features?

no code implementations NeurIPS 2021 Simon Kornblith, Ting Chen, Honglak Lee, Mohammad Norouzi

We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks.

General Classification Image Classification

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

3 code implementations ICLR 2021 Thao Nguyen, Maithra Raghu, Simon Kornblith

We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models.

Big Self-Supervised Models are Strong Semi-Supervised Learners

8 code implementations NeurIPS 2020 Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.

Self-Supervised Image Classification Semi-Supervised Image Classification

Generalised Lipschitz Regularisation Equals Distributional Robustness

no code implementations11 Feb 2020 Zac Cranko, Zhan Shi, Xinhua Zhang, Richard Nock, Simon Kornblith

The problem of adversarial examples has highlighted the need for a theory of regularisation that is general enough to apply to exotic function classes, such as universal approximators.

Subclass Distillation

no code implementations10 Feb 2020 Rafael Müller, Simon Kornblith, Geoffrey Hinton

By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data.

Revisiting Spatial Invariance with Low-Rank Local Connectivity

no code implementations ICML 2020 Gamaleldin F. Elsayed, Prajit Ramachandran, Jonathon Shlens, Simon Kornblith

Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias.

Inductive Bias

The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

no code implementations NeurIPS 2020 Katherine L. Hermann, Ting Chen, Simon Kornblith

By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets.

Data Augmentation

MIST: Multiple Instance Spatial Transformer Networks

no code implementations25 Sep 2019 Baptiste Angles, Simon Kornblith, Shahram Izadi, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Image Reconstruction

Certifying Distributional Robustness using Lipschitz Regularisation

no code implementations25 Sep 2019 Zac Cranko, Zhan Shi, Xinhua Zhang, Simon Kornblith, Richard Nock

Distributional robust risk (DRR) minimisation has arisen as a flexible and effective framework for machine learning.

Saccader: Improving Accuracy of Hard Attention Models for Vision

2 code implementations NeurIPS 2019 Gamaleldin F. Elsayed, Simon Kornblith, Quoc V. Le

Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret.

Hard Attention Image Classification

When Does Label Smoothing Help?

2 code implementations NeurIPS 2019 Rafael Müller, Simon Kornblith, Geoffrey Hinton

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels.

Knowledge Distillation Speech Recognition +1

Cerberus: A Multi-headed Derenderer

no code implementations28 May 2019 Boyang Deng, Simon Kornblith, Geoffrey Hinton

To generalize to novel visual scenes with new viewpoints and new object poses, a visual system needs representations of the shapes of the parts of an object that are invariant to changes in viewpoint or pose.

Similarity of Neural Network Representations Revisited

8 code implementations ICML 2019 2019 Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation.

MIST: Multiple Instance Spatial Transformer Network

1 code implementation26 Nov 2018 Baptiste Angles, Yuhe Jin, Simon Kornblith, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Anomaly Detection In Surveillance Videos Image Reconstruction

Domain Adaptive Transfer Learning with Specialist Models

no code implementations16 Nov 2018 Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V. Le, Ruoming Pang

Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning.

Ranked #2 on Fine-Grained Image Classification on Stanford Cars (using extra training data)

Domain Adaptation Fine-Grained Image Classification +2

Lipschitz Networks and Distributional Robustness

no code implementations4 Sep 2018 Zac Cranko, Simon Kornblith, Zhan Shi, Richard Nock

Robust risk minimisation has several advantages: it has been studied with regards to improving the generalisation properties of models and robustness to adversarial perturbation.

Do Better ImageNet Models Transfer Better?

no code implementations CVPR 2019 Simon Kornblith, Jonathon Shlens, Quoc V. Le

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer.

Fine-Grained Image Classification General Classification +1

Cannot find the paper you are looking for? You can Submit a new open access paper.