Off-Diagonal Orthogonal Regularization

Introduced by Brock et al. in Large Scale GAN Training for High Fidelity Natural Image Synthesis

Off-Diagonal Orthogonal Regularization is a modified form of orthogonal regularization originally used in BigGAN. The original orthogonal regularization is known to be limiting so the authors explore several variants designed to relax the constraint while still imparting the desired smoothness to the models. They opt for a modification where they remove diagonal terms from the regularization, and aim to minimize the pairwise cosine similarity between filters but does not constrain their norm:

$$ R_{\beta}\left(W\right) = \beta|| W^{T}W \odot \left(\mathbf{1}-I\right) ||^{2}_{F} $$

where $\mathbf{1}$ denotes a matrix with all elements set to 1. The authors sweep $\beta$ values and select $10^{−4}$.

Source: Large Scale GAN Training for High Fidelity Natural Image Synthesis

Latest Papers

PAPER DATE
Conditioning Trick for Training Stable GANs
Mohammad EsmaeilpourRaymel Alfonso SalloOlivier St-GeorgesPatrick CardinalAlessandro Lameiras Koerich
2020-10-12
SMYRF: Efficient Attention using Asymmetric Clustering
| Giannis DarasNikita KitaevAugustus OdenaAlexandros G. Dimakis
2020-10-11
LOGAN: Local Group Bias Detection by Clustering
Jieyu ZhaoKai-Wei Chang
2020-10-06
TinyGAN: Distilling BigGAN for Conditional Image Generation
| Ting-Yun ChangChi-Jen Lu
2020-09-29
not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget
Seungwook HanAkash SrivastavaCole HurwitzPrasanna SattigeriDavid D. Cox
2020-09-09
A Spectral Energy Distance for Parallel Speech Synthesis
Alexey A. GritsenkoTim SalimansRianne van den BergJasper SnoekNal Kalchbrenner
2020-08-03
Instance Selection for GANs
Terrance DeVriesMichal DrozdzalGraham W. Taylor
2020-07-30
Interpolating GANs to Scaffold Autotelic Creativity
Ziv EpsteinOcéane BoulaisSkylar GordonMatt Groh
2020-07-21
Differentiable Augmentation for Data-Efficient GAN Training
| Shengyu ZhaoZhijian LiuJi LinJun-Yan ZhuSong Han
2020-06-18
Training Generative Adversarial Networks with Limited Data
| Tero KarrasMiika AittalaJanne HellstenSamuli LaineJaakko LehtinenTimo Aila
2020-06-11
Learning disconnected manifolds: a no GANs land
Ugo TanielianThibaut IssenhuthElvis DohmatobJeremie Mary
2020-06-08
Big GANs Are Watching You: Towards Unsupervised Object Segmentation with Off-the-Shelf Generative Models
| Andrey VoynovStanislav MorozovArtem Babenko
2020-06-08
A U-Net Based Discriminator for Generative Adversarial Networks
Edgar Schonfeld Bernt Schiele Anna Khoreva
2020-06-01
Network Fusion for Content Creation with Conditional INNs
Robin RombachPatrick EsserBjörn Ommer
2020-05-27
GANSpace: Discovering Interpretable GAN Controls
| Erik HärkönenAaron HertzmannJaakko LehtinenSylvain Paris
2020-04-06
Evolving Normalization-Activation Layers
| Hanxiao LiuAndrew BrockKaren SimonyanQuoc V. Le
2020-04-06
Feature Quantization Improves GAN Training
| Yang ZhaoChunyuan LiPing YuJianfeng GaoChangyou Chen
2020-04-05
BigGAN-based Bayesian reconstruction of natural images from human brain activity
Kai QiaoJian ChenLinyuan WangChi ZhangLi TongBin Yan
2020-03-13
A U-Net Based Discriminator for Generative Adversarial Networks
| Edgar SchönfeldBernt SchieleAnna Khoreva
2020-02-28
Improved Consistency Regularization for GANs
Zhengli ZhaoSameer SinghHonglak LeeZizhao ZhangAugustus OdenaHan Zhang
2020-02-11
Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Milad MozafariLeila ReddyRufin VanRullen
2020-01-31
HRFA: High-Resolution Feature-based Attack
| Zhixing YeSizhe ChenPeidong ZhangChengjin SunXiaolin Huang
2020-01-21
Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
Mohamed El Amine SeddikCosme LouartMohamed TamaazoustiRomain Couillet
2020-01-21
CNN-generated images are surprisingly easy to spot... for now
| Sheng-Yu WangOliver WangRichard ZhangAndrew OwensAlexei A. Efros
2019-12-23
Detecting GAN generated errors
Xiru ZhuFengdi CheTianzi YangTzuyang YuDavid MegerGregory Dudek
2019-12-02
LOGAN: Latent Optimisation for Generative Adversarial Networks
| Yan WuJeff DonahueDavid BalduzziKaren SimonyanTimothy Lillicrap
2019-12-02
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
| Ceyuan YangYujun ShenBolei Zhou
2019-11-21
Improving sample diversity of a pre-trained, class-conditional GAN by changing its class embeddings
| Qi LiLong MaiMichael A. AlcornAnh Nguyen
2019-10-10
High Fidelity Speech Synthesis with Adversarial Networks
| Mikołaj BińkowskiJeff DonahueSander DielemanAidan ClarkErich ElsenNorman CasagrandeLuis C. CoboKaren Simonyan
2019-09-25
Adversarial Video Generation on Complex Datasets
Aidan ClarkJeff DonahueKaren Simonyan
2019-07-15
Large Scale Adversarial Representation Learning
| Jeff DonahueKaren Simonyan
2019-07-04
Improved Precision and Recall Metric for Assessing Generative Models
| Tuomas KynkäänniemiTero KarrasSamuli LaineJaakko LehtinenTimo Aila
2019-04-15
High-Fidelity Image Generation With Fewer Labels
| Mario LucicMichael TschannenMarvin RitterXiaohua ZhaiOlivier BachemSylvain Gelly
2019-03-06
Metropolis-Hastings view on variational inference and adversarial training
Kirill NeklyudovEvgenii EgorovPavel ShvechikovDmitry Vetrov
2018-10-16
Large Scale GAN Training for High Fidelity Natural Image Synthesis
| Andrew BrockJeff DonahueKaren Simonyan
2018-09-28

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories