Orthogonal Regularization

Introduced by Brock et al. in Neural Photo Editing with Introspective Adversarial Networks

Orthogonal Regularization is a regularization technique for convolutional neural networks, introduced with generative modelling as the task in mind. Orthogonality is argued to be a desirable quality in ConvNet filters, partially because multiplication by an orthogonal matrix leaves the norm of the original matrix unchanged. This property is valuable in deep or recurrent networks, where repeated matrix multiplication can result in signals vanishing or exploding. To try to maintain orthogonality throughout training, Orthogonal Regularization encourages weights to be orthogonal by pushing them towards the nearest orthogonal manifold. The objective function is augmented with the cost:

$$ \mathcal{L}_{ortho} = \sum\left(|WW^{T} − I|\right) $$

Where $\sum$ indicates a sum across all filter banks, $W$ is a filter bank, and $I$ is the identity matrix

Source: Neural Photo Editing with Introspective Adversarial Networks


Paper Code Results Date Stars


Task Papers Share
Image Generation 3 12.50%
Sentiment Analysis 2 8.33%
Speech Synthesis 2 8.33%
Image Classification 2 8.33%
Video Generation 2 8.33%
Video Prediction 2 8.33%
General Classification 2 8.33%
Fairness 1 4.17%
Model Compression 1 4.17%


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign