A Reversible Residual Network, or RevNet, is a variant of a ResNet where each layer’s activations can be reconstructed exactly from the next layer’s. Therefore, the activations for most layers need not be stored in memory during backpropagation. The result is a network architecture whose activation storage requirements are independent of depth, and typically at least an order of magnitude smaller compared with equally sized ResNets.
RevNets are composed of a series of reversible blocks. Units in each layer are partitioned into two groups, denoted $x_{1}$ and $x_{2}$; the authors find what works best is partitioning the channels. Each reversible block takes inputs $\left(x_{1}, x_{2}\right)$ and produces outputs $\left(y_{1}, y_{2}\right)$ according to the following additive coupling rules – inspired the transformation in NICE (nonlinear independent components estimation) – and residual functions $F$ and $G$ analogous to those in standard ResNets:
$$y_{1} = x_{1} + F\left(x_{2}\right)$$ $$y_{2} = x_{2} + G\left(y_{1}\right)$$
Each layer’s activations can be reconstructed from the next layer’s activations as follows:
$$ x_{2} = y_{2} − G\left(y_{1}\right)$$ $$ x_{1} = y_{1} − F\left(x_{2}\right)$$
Note that unlike residual blocks, reversible blocks must have a stride of 1 because otherwise the layer discards information, and therefore cannot be reversible. Standard ResNet architectures typically have a handful of layers with a larger stride. If we define a RevNet architecture analogously, the activations must be stored explicitly for all non-reversible layers.
Source: The Reversible Residual Network: Backpropagation Without Storing ActivationsPaper | Code | Results | Date | Stars |
---|
Task | Papers | Share |
---|---|---|
Image Classification | 3 | 13.64% |
Image Generation | 2 | 9.09% |
Classification | 1 | 4.55% |
Denoising | 1 | 4.55% |
Fine-Grained Image Classification | 1 | 4.55% |
Super-Resolution | 1 | 4.55% |
Image Restoration | 1 | 4.55% |
Uncertainty Quantification | 1 | 4.55% |
Dimensionality Reduction | 1 | 4.55% |
Component | Type |
|
---|---|---|
Batch Normalization
|
Normalization | (optional) |
Convolution
|
Convolutions | |
Pointwise Convolution
|
Convolutions | |
ReLU
|
Activation Functions | (optional) |
Reversible Residual Block
|
Skip Connection Blocks |