Search Results for author:
Found 20 papers, 6 papers with code
Continual Few-Shot Learning Using HyperTransformers
We focus on the problem of learning without forgetting from multiple tasks arriving sequentially, where each task is defined using a few-shot episode of novel or already seen classes.
Training trajectories, mini-batch losses and the curious role of the learning rate
In particular, for Exponential Moving Average (EMA) and Stochastic Weight Averaging we show that our proposed model matches the observed training trajectories on ImageNet.
Transformers learn in-context by gradient descent
We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss.
Decentralized Learning with Multi-Headed Distillation
We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.
Fine-tuning Image Transformers using Learnable Memory
In this paper we propose augmenting Vision Transformer models with learnable memory tokens.
HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning
In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples.
HyperTransformer: Attention-Based CNN Model Generation from Few Samples
In this work we propose a HyperTransformer, a transformer based model that generates all weights of a CNN model directly from the support samples.
Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
From practical perspective, our approach allows to: (a) reuse existing modules for learning new task by adjusting the computation order, (b) use it for unsupervised multi-source domain adaptation to illustrate that adaptation to unseen data can be achieved by only manipulating the order of pretrained modules, (c) show how our approach can be used to increase accuracy of existing architectures for image classification tasks such as ImageNet, without any parameter increase, by reusing the same block multiple times.
BasisNet: Two-stage Model Synthesis for Efficient Inference
With early termination, the average cost can be further reduced to 198M MAdds while maintaining accuracy of 80. 0% on ImageNet.
Ranked #661 on
Image Classification
on ImageNet
Meta-Learning Bidirectional Update Rules
We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule.
Large-Scale Generative Data-Free Distillation
Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning.
Image segmentation via Cellular Automata
The update rule is applied repeatedly in parallel to a large random subset of cells and after convergence is used to produce segmentation masks that are then back-propagated to learn the optimal update rules using standard gradient descent methods.
Non-discriminative data or weak model? On the relative importance of data and model resolution
We explore the question of how the resolution of the input image ("input resolution") affects the performance of a neural network when compared to the resolution of the hidden layers ("internal resolution").
Information-Bottleneck Approach to Salient Region Discovery
We propose a new method for learning image attention masks in a semi-supervised setting based on the Information Bottleneck principle.
K For The Price Of 1: Parameter Efficient Multi-task And Transfer Learning
We introduce a novel method that enables parameter-efficient transfer and multitask learning.
K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks.
MobileNetV2: Inverted Residuals and Linear Bottlenecks
In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes.
Ranked #7 on
Retinal OCT Disease Classification
on OCT2017
CycleGAN, a Master of Steganography
CycleGAN (Zhu et al. 2017) is one recent successful approach to learn a transformation between two image distributions.
The Power of Sparsity in Convolutional Neural Networks
Deep convolutional networks are well-known for their high computational and memory demands.
Inverting face embeddings with convolutional neural networks
Deep neural networks have dramatically advanced the state of the art for many areas of machine learning.
