Search Results for author:
Found 52 papers, 25 papers with code
Neither hype nor gloom do DNNs justice
Neither the hype exemplified in some exaggerated claims about deep neural networks (DNNs), nor the gloom expressed by Bowers et al. do DNNs as models in vision science justice: DNNs rapidly evolve, and today's limitations are often tomorrow's successes.
Probing clustering in neural network representations
Neural network representations contain structure beyond what was present in the training labels.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
Getting aligned on representational alignment
Finally, we lay out open problems in representational alignment where progress can benefit all three of these fields.
Small-scale proxies for large-scale Transformer training instabilities
In this work, we seek ways to reproduce and study training stability and instability at smaller scales.
Replacing softmax with ReLU in Vision Transformers
Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU.
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters.
Ranked #14 on
Visual Question Answering (VQA)
on InfiMM-Eval
Guiding Image Captioning Models Toward More Specific Captions
We further explore the use of language models to guide the decoding process, obtaining small improvements over the Pareto frontier of reference-free vs. reference-based captioning metrics that arises from classifier-free guidance, and substantially improving the quality of captions generated from a model trained only on minimally curated web data.
Towards Generalist Biomedical AI
While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.
Synthetic Data from Diffusion Models Improves ImageNet Classification
Deep generative models are becoming increasingly powerful, now generating diverse high fidelity photo-realistic samples given text prompts.
FlexiViT: One Model for All Patch Sizes
Vision Transformers convert images to sequences by slicing them into patches.
On the Relationship Between Explanation and Prediction: A Causal View
Our work borrows tools from causal inference to systematically assay this relationship.
Hyperbolic Contrastive Learning for Visual Representations beyond Objects
To exploit such a structure, we propose a contrastive learning framework where a Euclidean loss is used to learn object representations and a hyperbolic loss is used to encourage representations of scenes to lie close to representations of their constituent objects in a hyperbolic space.
Human alignment of neural network representations
Linear transformations of neural network representations learned from behavioral responses from one dataset substantially improve alignment with human similarity judgments on the other two datasets.
Gaussian-Bernoulli RBMs Without Tears
We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations.
Improving Dense Contrastive Learning with Dense Negative Pairs
Many contrastive representation learning methods learn a single global representation of an entire image.
Scaling Forward Gradient With Local Losses
Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks.
Patching open-vocabulary models by interpolating weights
We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
A Study on Self-Supervised Object Detection Pretraining
In this work, we study different approaches to self-supervised pretraining of object detection models.
Interpretability of artificial neural network models in artificial Intelligence vs. neuroscience
Given the widespread calls to improve the interpretability of AI systems, we here highlight these different notions of interpretability and argue that the neuroscientific interpretability of ANNs can be pursued in parallel with, but independently from, the ongoing efforts in AI.
Decoder Denoising Pretraining for Semantic Segmentation
We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder.
Robust and Efficient Medical Imaging with Self-Supervision
These results suggest that REMEDIS can significantly accelerate the life-cycle of medical imaging AI development thereby presenting an important step forward for medical imaging AI to deliver broad impact.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
Ranked #1 on
Image Classification
on ImageNet V2
(using extra training data)
On the Origins of the Block Structure Phenomenon in Neural Network Representations
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.
Meta-Learning to Improve Pre-Training
Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains.
Generalized Shape Metrics on Neural Representations
In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.
Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations
Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.
Robust fine-tuning of zero-shot models
Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.
Ranked #12 on
Image Classification
on ObjectNet
(using extra training data)
Do Vision Transformers See Like Convolutional Neural Networks?
Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.
MIST: Multiple Instance Spatial Transformer
We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.
Big Self-Supervised Models Advance Medical Image Classification
Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis.
Demystifying Loss Functions for Classification
It is common to use the softmax cross-entropy loss to train neural networks on classification datasets where a single class label is assigned to each example.
Boosting Contrastive Self-Supervised Learning with False Negative Cancellation
While positive pairs can be generated reliably (e. g., as different views of the same image), it is difficult to accurately establish negative pairs, defined as samples from different images regardless of their semantic content or visual features.
Teaching with Commentaries
We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process.
Why Do Better Loss Functions Lead to Less Transferable Features?
We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks.
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models.
Big Self-Supervised Models are Strong Semi-Supervised Learners
The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.
Self-Supervised Image Classification
Semi-Supervised Image Classification
A Simple Framework for Contrastive Learning of Visual Representations
This paper presents SimCLR: a simple framework for contrastive learning of visual representations.
Ranked #4 on
Contrastive Learning
on imagenet-1k
Contrastive Learning
Self-Supervised Image Classification
+3
Generalised Lipschitz Regularisation Equals Distributional Robustness
The problem of adversarial examples has highlighted the need for a theory of regularisation that is general enough to apply to exotic function classes, such as universal approximators.
Subclass Distillation
By training a small "student" model to match these probabilities, it is possible to transfer most of the generalization ability of the teacher to the student, often producing a much better small model than directly training the student on the training data.
Revisiting Spatial Invariance with Low-Rank Local Connectivity
Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias.
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets.
MIST: Multiple Instance Spatial Transformer Networks
We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.
Certifying Distributional Robustness using Lipschitz Regularisation
Distributional robust risk (DRR) minimisation has arisen as a flexible and effective framework for machine learning.
Saccader: Improving Accuracy of Hard Attention Models for Vision
Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret.
When Does Label Smoothing Help?
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels.
Cerberus: A Multi-headed Derenderer
To generalize to novel visual scenes with new viewpoints and new object poses, a visual system needs representations of the shapes of the parts of an object that are invariant to changes in viewpoint or pose.
Similarity of Neural Network Representations Revisited
We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation.
MIST: Multiple Instance Spatial Transformer Network
We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts.
Anomaly Detection In Surveillance Videos
Image Reconstruction
Domain Adaptive Transfer Learning with Specialist Models
Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning.
Ranked #3 on
Fine-Grained Image Classification
on Stanford Cars
(using extra training data)
Lipschitz Networks and Distributional Robustness
Robust risk minimisation has several advantages: it has been studied with regards to improving the generalisation properties of models and robustness to adversarial perturbation.
Do Better ImageNet Models Transfer Better?
Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer.
