Search Results for author:
Found 33 papers, 16 papers with code
Effective pruning of web-scale datasets based on complexity of concept clusters
Using a simple and intuitive complexity measure, we are able to reduce the training cost to a quarter of regular training.
Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Armed with this knowledge, we devise novel pruning metrics that operate in embedding space to identify and remove low-quality entries in the Stack dataset.
PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation.
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time.
Emergence of Maps in the Memories of Blind Navigation Agents
A positive answer to this question would (a) explain the surprising phenomenon in recent literature of ostensibly map-free neural-networks achieving strong performance, and (b) strengthen the evidence of mapping as a fundamental mechanism for navigation by intelligent embodied agents, whether they be biological or artificial.
lo-fi: distributed fine-tuning without communication
When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step.
Beyond neural scaling laws: beating power law scaling via data pruning
Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder.
Ranked #1 on
Image Classification
on ImageNet V2
(using extra training data)
Learning Background Invariance Improves Generalization and Robustness in Self-Supervised Learning on ImageNet and Beyond
Through a systematic, comprehensive investigation, we show that background augmentations lead to improved generalization with substantial improvements ($\sim$1-2% on ImageNet) in performance across a spectrum of state-of-the-art self-supervised methods (MoCo-v2, BYOL, SwAV) on a variety of tasks, even enabling performance on par with the supervised baseline.
Grounding inductive biases in natural images:invariance stems from variations in data
While prior work has focused on synthetic data, we attempt here to characterize the factors of variation in a real dataset, ImageNet, and study the invariance of both standard residual networks and the recently proposed vision transformer with respect to changes in these factors.
Grounding inductive biases in natural images: invariance stems from variations in data
While prior work has focused on synthetic data, we attempt here to characterize the factors of variation in a real dataset, ImageNet, and study the invariance of both standard residual networks and the recently proposed vision transformer with respect to changes in these factors.
Width Transfer: On the (In)variance of Width Optimization
In this work, we propose width transfer, a technique that harnesses the assumptions that the optimized widths (or channel counts) are regular across sizes and depths.
Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks.
Ranked #83 on
Image Classification
on ObjectNet
(using extra training data)
Uncovering the impact of learning rate for global magnitude pruning
However, prior work has implicitly assumed that the best training configuration for model performance was also the best configuration for mask discovery.
Reservoir Transformers
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.
Are all negatives created equal in contrastive instance discrimination?
Using methodology from MoCo v2 (Chen et al., 2020), we divided negatives by their difficulty for a given query and studied which difficulty ranges were most important for learning useful representations.
PareCO: Pareto-aware Channel Optimization for Slimmable Neural Networks
Slimmable neural networks provide a flexible trade-off front between prediction error and computational cost (such as the number of floating-point operations or FLOPs) with the same storage cost as a single model.
Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
We also found that the input-unit gradient was more variable across samples and units in high-selectivity networks compared to low-selectivity networks.
Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks
In this work, we propose a general framework to enable joint optimization for both width configurations and weights of slimmable networks.
On the relationship between class selectivity, dimensionality, and robustness
While the relative trade-offs between sparse and distributed representations in deep neural networks (DNNs) are well-studied, less is known about how these trade-offs apply to representations of semantically-meaningful information.
Plan2Vec: Unsupervised Representation Learning by Latent Plans
In this paper we introduce plan2vec, an unsupervised representation learning approach that is inspired by reinforcement learning.
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features.
The Early Phase of Neural Network Training
We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset.
The Generalization-Stability Tradeoff In Neural Network Pruning
Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting.
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Perhaps surprisingly, we found that, within the natural images domain, winning ticket initializations generalized across a variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and Places365, often achieving performance close to that of winning tickets generated on the same dataset.
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019).
Learning to Make Analogies by Contrasting Abstract Relational Structure
Here, we study how analogical reasoning can be induced in neural networks that learn to perceive and reason about raw visual data.
Analyzing biological and artificial neural networks: challenges with opportunities for synergy?
We explore opportunities for synergy between the two fields, such as the use of DNNs as in-silico model systems for neuroscience, and how this synergy can lead to new hypotheses about the operating principles of biological neural networks.
Measuring abstract reasoning in neural networks
To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways.
Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
Recent progress in artificial intelligence through reinforcement learning (RL) has shown great success on increasingly complex single-agent environments and two-player turn-based games.
Insights on representational similarity in neural networks with canonical correlation
Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training.
Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNs
In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers.
On the importance of single directions for generalization
Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.
