Search Results for author:
Found 14 papers, 4 papers with code
Trade-offs of Local SGD at Scale: An Empirical Study
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time.
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Finally, we experiment initializing the T-CNN from a partially trained CNN, and find that it reaches better performance than the corresponding hybrid model trained from scratch, while reducing training time.
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information.
Ranked #261 on
Image Classification
on ImageNet
Towards falsifiable interpretability research
Methods for understanding the decisions of and mechanisms underlying deep neural networks (DNNs) typically rely on building intuition by emphasizing sensory or semantic features of individual examples.
Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
Furthermore, the input-unit gradient is more variable across samples and units in high-selectivity networks compared to low-selectivity networks.
CURI: A Benchmark for Productive Concept Learning Under Uncertainty
Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts ("a scene with objects that have the same color") and ad-hoc categories defined through goals ("objects that could fall on one's head").
Analyzing Visual Representations in Embodied Navigation Tasks
Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task.
Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
For ResNet20 trained on CIFAR10 we could reduce class selectivity by a factor of 2. 5 with no impact on test accuracy, and reduce it nearly to zero with only a small ($\sim$2%) drop in test accuracy.
Representation Learning Through Latent Canonicalizations
We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision.
Pruning Convolutional Neural Networks with Self-Supervision
In this work, we investigate the use of standard pruning methods, developed primarily for supervised learning, for networks trained without labels (i. e. on self-supervised tasks).
Insights on Visual Representations for Embodied Navigation Tasks
Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We leverage this scaling to train an agent for 2. 5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
Ranked #1 on
PointGoal Navigation
on Gibson PointGoal Navigation
Finding Winning Tickets with Limited (or No) Supervision
The lottery ticket hypothesis argues that neural networks contain sparse subnetworks, which, if appropriately initialized (the winning tickets), are capable of matching the accuracy of the full network when trained in isolation.
Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
We analyze the dynamics of training deep ReLU networks and their implications on generalization capability.
