Search Results for author: Simon Kornblith

Found 52 papers, 25 papers with code

Neither hype nor gloom do DNNs justice

no code implementations8 Dec 2023 Felix A. Wichmann, Simon Kornblith, Robert Geirhos

Neither the hype exemplified in some exaggerated claims about deep neural networks (DNNs), nor the gloom expressed by Bowers et al. do DNNs as models in vision science justice: DNNs rapidly evolve, and today's limitations are often tomorrow's successes.

Probing clustering in neural network representations

no code implementations14 Nov 2023 Thao Nguyen, Simon Kornblith

Neural network representations contain structure beyond what was present in the training labels.


Replacing softmax with ReLU in Vision Transformers

no code implementations15 Sep 2023 Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, Simon Kornblith

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU.

Guiding Image Captioning Models Toward More Specific Captions

no code implementations ICCV 2023 Simon Kornblith, Lala Li, ZiRui Wang, Thao Nguyen

We further explore the use of language models to guide the decoding process, obtaining small improvements over the Pareto frontier of reference-free vs. reference-based captioning metrics that arises from classifier-free guidance, and substantially improving the quality of captions generated from a model trained only on minimally curated web data.

Image Captioning Image Retrieval

Synthetic Data from Diffusion Models Improves ImageNet Classification

no code implementations17 Apr 2023 Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, David J. Fleet

Deep generative models are becoming increasingly powerful, now generating diverse high fidelity photo-realistic samples given text prompts.

Classification Data Augmentation

Hyperbolic Contrastive Learning for Visual Representations beyond Objects

1 code implementation CVPR 2023 Songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, David Jacobs

To exploit such a structure, we propose a contrastive learning framework where a Euclidean loss is used to learn object representations and a hyperbolic loss is used to encourage representations of scenes to lie close to representations of their constituent objects in a hyperbolic space.

Contrastive Learning Image Classification +5

Human alignment of neural network representations

1 code implementation2 Nov 2022 Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, Simon Kornblith

Linear transformations of neural network representations learned from behavioral responses from one dataset substantially improve alignment with human similarity judgments on the other two datasets.

Odd One Out

Gaussian-Bernoulli RBMs Without Tears

1 code implementation19 Oct 2022 Renjie Liao, Simon Kornblith, Mengye Ren, David J. Fleet, Geoffrey Hinton

We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations.

Scaling Forward Gradient With Local Losses

1 code implementation7 Oct 2022 Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton

Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks.

Patching open-vocabulary models by interpolating weights

1 code implementation10 Aug 2022 Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt

We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.

Image Classification

A Study on Self-Supervised Object Detection Pretraining

no code implementations9 Jul 2022 Trung Dang, Simon Kornblith, Huy Thong Nguyen, Peter Chin, Maryam Khademi

In this work, we study different approaches to self-supervised pretraining of object detection models.

Object object-detection +2

Interpretability of artificial neural network models in artificial Intelligence vs. neuroscience

no code implementations7 Jun 2022 Kohitij Kar, Simon Kornblith, Evelina Fedorenko

Given the widespread calls to improve the interpretability of AI systems, we here highlight these different notions of interpretability and argue that the neuroscientific interpretability of ANNs can be pursued in parallel with, but independently from, the ongoing efforts in AI.

Decision Making

Decoder Denoising Pretraining for Semantic Segmentation

1 code implementation23 May 2022 Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, Mohammad Norouzi

We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder.

Denoising Segmentation +1

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

7 code implementations10 Mar 2022 Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.

 Ranked #1 on Image Classification on ImageNet V2 (using extra training data)

Domain Generalization Image Classification +2

On the Origins of the Block Structure Phenomenon in Neural Network Representations

1 code implementation15 Feb 2022 Thao Nguyen, Maithra Raghu, Simon Kornblith

Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.

Meta-Learning to Improve Pre-Training

no code implementations NeurIPS 2021 Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, David Duvenaud

Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains.

Data Augmentation Hyperparameter Optimization +1

Generalized Shape Metrics on Neural Representations

2 code implementations NeurIPS 2021 Alex H. Williams, Erin Kunz, Simon Kornblith, Scott W. Linderman

In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations

no code implementations29 Sep 2021 Thao Nguyen, Maithra Raghu, Simon Kornblith

Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.

Robust fine-tuning of zero-shot models

3 code implementations CVPR 2022 Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt

Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.

Ranked #12 on Image Classification on ObjectNet (using extra training data)

Image Classification Transfer Learning

Do Vision Transformers See Like Convolutional Neural Networks?

4 code implementations NeurIPS 2021 Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy

Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Classification Image Classification +1

MIST: Multiple Instance Spatial Transformer

1 code implementation CVPR 2021 Baptiste Angles, Yuhe Jin, Simon Kornblith, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Image Reconstruction

Big Self-Supervised Models Advance Medical Image Classification

1 code implementation ICCV 2021 Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, Mohammad Norouzi

Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis.

Contrastive Learning General Classification +3

Demystifying Loss Functions for Classification

no code implementations1 Jan 2021 Simon Kornblith, Honglak Lee, Ting Chen, Mohammad Norouzi

It is common to use the softmax cross-entropy loss to train neural networks on classification datasets where a single class label is assigned to each example.

Classification General Classification +1

Boosting Contrastive Self-Supervised Learning with False Negative Cancellation

1 code implementation23 Nov 2020 Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, Maryam Khademi

While positive pairs can be generated reliably (e. g., as different views of the same image), it is difficult to accurately establish negative pairs, defined as samples from different images regardless of their semantic content or visual features.

Contrastive Learning Representation Learning +3

Teaching with Commentaries

1 code implementation ICLR 2021 Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, Geoffrey Hinton

We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.

Data Augmentation

Why Do Better Loss Functions Lead to Less Transferable Features?

no code implementations NeurIPS 2021 Simon Kornblith, Ting Chen, Honglak Lee, Mohammad Norouzi

We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks.

General Classification Image Classification

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

4 code implementations ICLR 2021 Thao Nguyen, Maithra Raghu, Simon Kornblith

We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models.

Big Self-Supervised Models are Strong Semi-Supervised Learners

8 code implementations NeurIPS 2020 Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.

Self-Supervised Image Classification Semi-Supervised Image Classification

Generalised Lipschitz Regularisation Equals Distributional Robustness

no code implementations11 Feb 2020 Zac Cranko, Zhan Shi, Xinhua Zhang, Richard Nock, Simon Kornblith

The problem of adversarial examples has highlighted the need for a theory of regularisation that is general enough to apply to exotic function classes, such as universal approximators.

Subclass Distillation

no code implementations10 Feb 2020 Rafael Müller, Simon Kornblith, Geoffrey Hinton

By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data.

Revisiting Spatial Invariance with Low-Rank Local Connectivity

no code implementations ICML 2020 Gamaleldin F. Elsayed, Prajit Ramachandran, Jonathon Shlens, Simon Kornblith

Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias.

Inductive Bias Test

The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

no code implementations NeurIPS 2020 Katherine L. Hermann, Ting Chen, Simon Kornblith

By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets.

Data Augmentation

MIST: Multiple Instance Spatial Transformer Networks

no code implementations25 Sep 2019 Baptiste Angles, Simon Kornblith, Shahram Izadi, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Image Reconstruction

Certifying Distributional Robustness using Lipschitz Regularisation

no code implementations25 Sep 2019 Zac Cranko, Zhan Shi, Xinhua Zhang, Simon Kornblith, Richard Nock

Distributional robust risk (DRR) minimisation has arisen as a flexible and effective framework for machine learning.

Saccader: Improving Accuracy of Hard Attention Models for Vision

2 code implementations NeurIPS 2019 Gamaleldin F. Elsayed, Simon Kornblith, Quoc V. Le

Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret.

Hard Attention Image Classification

When Does Label Smoothing Help?

3 code implementations NeurIPS 2019 Rafael Müller, Simon Kornblith, Geoffrey Hinton

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels.

Image Classification Knowledge Distillation +3

Cerberus: A Multi-headed Derenderer

no code implementations28 May 2019 Boyang Deng, Simon Kornblith, Geoffrey Hinton

To generalize to novel visual scenes with new viewpoints and new object poses, a visual system needs representations of the shapes of the parts of an object that are invariant to changes in viewpoint or pose.

Similarity of Neural Network Representations Revisited

8 code implementations ICML 2019 2019 Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation.

MIST: Multiple Instance Spatial Transformer Network

1 code implementation26 Nov 2018 Baptiste Angles, Yuhe Jin, Simon Kornblith, Andrea Tagliasacchi, Kwang Moo Yi

We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.

Anomaly Detection In Surveillance Videos Image Reconstruction

Domain Adaptive Transfer Learning with Specialist Models

no code implementations16 Nov 2018 Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V. Le, Ruoming Pang

Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning.

Ranked #3 on Fine-Grained Image Classification on Stanford Cars (using extra training data)

Domain Adaptation Fine-Grained Image Classification +2

Lipschitz Networks and Distributional Robustness

no code implementations4 Sep 2018 Zac Cranko, Simon Kornblith, Zhan Shi, Richard Nock

Robust risk minimisation has several advantages: it has been studied with regards to improving the generalisation properties of models and robustness to adversarial perturbation.

Do Better ImageNet Models Transfer Better?

no code implementations CVPR 2019 Simon Kornblith, Jonathon Shlens, Quoc V. Le

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer.

Fine-Grained Image Classification General Classification +1

Cannot find the paper you are looking for? You can Submit a new open access paper.