Regularization
Off-Diagonal Orthogonal Regularization
Introduced by Brock et al. in Large Scale GAN Training for High Fidelity Natural Image SynthesisOff-Diagonal Orthogonal Regularization is a modified form of orthogonal regularization originally used in BigGAN. The original orthogonal regularization is known to be limiting so the authors explore several variants designed to relax the constraint while still imparting the desired smoothness to the models. They opt for a modification where they remove diagonal terms from the regularization, and aim to minimize the pairwise cosine similarity between filters but does not constrain their norm:
$$ R_{\beta}\left(W\right) = \beta|| W^{T}W \odot \left(\mathbf{1}-I\right) ||^{2}_{F} $$
where $\mathbf{1}$ denotes a matrix with all elements set to 1. The authors sweep $\beta$ values and select $10^{−4}$.
Source: Large Scale GAN Training for High Fidelity Natural Image Synthesis
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Image Generation | 36 | 19.05% |
| Conditional Image Generation | 15 | 7.94% |
| Multi-agent Reinforcement Learning | 7 | 3.70% |
| Super-Resolution | 6 | 3.17% |
| Reinforcement Learning (RL) | 6 | 3.17% |
| Decision Making | 5 | 2.65% |
| Denoising | 4 | 2.12% |
| Clustering | 4 | 2.12% |
| Benchmarking | 3 | 1.59% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
