Spectral Normalization

Introduced by Miyato et al. in Spectral Normalization for Generative Adversarial Networks

Spectral Normalization is a normalization technique used for generative adversarial networks, used to stabilize training of the discriminator. Spectral normalization has the convenient property that the Lipschitz constant is the only hyper-parameter to be tuned.

It controls the Lipschitz constant of the discriminator $f$ by constraining the spectral norm of each layer $g : \textbf{h}_{in} \rightarrow \textbf{h}_{out}$. The Lipschitz norm $\Vert{g}\Vert_{\text{Lip}}$ is equal to $\sup_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right)$, where $\sigma\left(a\right)$ is the spectral norm of the matrix $A$ ($L_{2}$ matrix norm of $A$):

$$ \sigma\left(a\right) = \max_{\textbf{h}:\textbf{h}\neq{0}}\frac{\Vert{A\textbf{h}}\Vert_{2}}{\Vert\textbf{h}\Vert_{2}} = \max_{\Vert\textbf{h}\Vert_{2}\leq{1}}{\Vert{A\textbf{h}}\Vert_{2}} $$

which is equivalent to the largest singular value of $A$. Therefore for a linear layer $g\left(\textbf{h}\right) = W\textbf{h}$ the norm is given by $\Vert{g}\Vert_{\text{Lip}} = \sup_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right) = \sup_{\textbf{h}}\sigma\left(W\right) = \sigma\left(W\right) $. Spectral normalization normalizes the spectral norm of the weight matrix $W$ so it satisfies the Lipschitz constraint $\sigma\left(W\right) = 1$:

$$ \bar{W}_{\text{SN}}\left(W\right) = W / \sigma\left(W\right) $$

Source: Spectral Normalization for Generative Adversarial Networks

Latest Papers

PAPER DATE
Conditioning Trick for Training Stable GANs
Mohammad EsmaeilpourRaymel Alfonso SalloOlivier St-GeorgesPatrick CardinalAlessandro Lameiras Koerich
2020-10-12
SMYRF: Efficient Attention using Asymmetric Clustering
| Giannis DarasNikita KitaevAugustus OdenaAlexandros G. Dimakis
2020-10-11
LOGAN: Local Group Bias Detection by Clustering
Jieyu ZhaoKai-Wei Chang
2020-10-06
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior
Zi LinJeremiah Zhe LiuZi YangNan HuaDan Roth
2020-10-05
TinyGAN: Distilling BigGAN for Conditional Image Generation
| Ting-Yun ChangChi-Jen Lu
2020-09-29
Anonymization of labeled TOF-MRA images for brain vessel segmentation using generative adversarial networks
Tabea KossenPooja SubramaniamVince I. MadaiAnja HennemuthKristian HildebrandAdam HilbertJan SobeskyMichelle LivneIvana GalinovicAhmed A. KhalilJochen B. FiebachDietmar Frey
2020-09-09
not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget
Seungwook HanAkash SrivastavaCole HurwitzPrasanna SattigeriDavid D. Cox
2020-09-09
Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
Zinan LinVyas SekarGiulia Fanti
2020-09-06
Improving the Speed and Quality of GAN by Adversarial Training
Jiachen ZhongXuanqing LiuCho-Jui Hsieh
2020-08-07
A Spectral Energy Distance for Parallel Speech Synthesis
Alexey A. GritsenkoTim SalimansRianne van den BergJasper SnoekNal Kalchbrenner
2020-08-03
Instance Selection for GANs
Terrance DeVriesMichal DrozdzalGraham W. Taylor
2020-07-30
Interpolating GANs to Scaffold Autotelic Creativity
Ziv EpsteinOcéane BoulaisSkylar GordonMatt Groh
2020-07-21
NVAE: A Deep Hierarchical Variational Autoencoder
Arash VahdatJan Kautz
2020-07-08
PriorGAN: Real Data Prior for Generative Adversarial Nets
Shuyang GuJianmin BaoDong ChenFang Wen
2020-06-30
Differentiable Augmentation for Data-Efficient GAN Training
| Shengyu ZhaoZhijian LiuJi LinJun-Yan ZhuSong Han
2020-06-18
BatVision with GCC-PHAT Features for Better Sound to Vision Predictions
Jesper Haahr ChristensenSascha HornauerStella Yu
2020-06-14
Training Generative Adversarial Networks with Limited Data
| Tero KarrasMiika AittalaJanne HellstenSamuli LaineJaakko LehtinenTimo Aila
2020-06-11
Learning disconnected manifolds: a no GANs land
Ugo TanielianThibaut IssenhuthElvis DohmatobJeremie Mary
2020-06-08
Big GANs Are Watching You: Towards Unsupervised Object Segmentation with Off-the-Shelf Generative Models
| Andrey VoynovStanislav MorozovArtem Babenko
2020-06-08
A U-Net Based Discriminator for Generative Adversarial Networks
Edgar Schonfeld Bernt Schiele Anna Khoreva
2020-06-01
Network Fusion for Content Creation with Conditional INNs
Robin RombachPatrick EsserBjörn Ommer
2020-05-27
Synthesizing Unrestricted False Positive Adversarial Objects Using Generative Models
Martin KotuliakSandro E. SchoenbornAndrei Dan
2020-05-19
An FPGA-Based On-Device Reinforcement Learning Approach using Online Sequential Learning
Hirohisa WatanabeMineto TsukadaHiroki Matsutani
2020-05-10
Mimicry: Towards the Reproducibility of GAN Research
| Kwot Sin LeeChristopher Town
2020-05-05
Text-Guided Neural Image Inpainting
| Lisai ZhangQingcai ChenBaotian HuShuoran Jiang
2020-04-07
GANSpace: Discovering Interpretable GAN Controls
| Erik HärkönenAaron HertzmannJaakko LehtinenSylvain Paris
2020-04-06
Evolving Normalization-Activation Layers
| Hanxiao LiuAndrew BrockKaren SimonyanQuoc V. Le
2020-04-06
Feature Quantization Improves GAN Training
| Yang ZhaoChunyuan LiPing YuJianfeng GaoChangyou Chen
2020-04-05
Controllable Orthogonalization in Training DNNs
| Lei HuangLi LiuFan ZhuDiwen WanZehuan YuanBo LiLing Shao
2020-04-02
BigGAN-based Bayesian reconstruction of natural images from human brain activity
Kai QiaoJian ChenLinyuan WangChi ZhangLi TongBin Yan
2020-03-13
Transformation-based Adversarial Video Prediction on Large-Scale Data
Pauline LucAidan ClarkSander DielemanDiego de Las CasasYotam DoronAlbin CassirerKaren Simonyan
2020-03-09
A U-Net Based Discriminator for Generative Adversarial Networks
| Edgar SchönfeldBernt SchieleAnna Khoreva
2020-02-28
Momentum-Net for Low-Dose CT Image Reconstruction
Siqi YeYong LongIl Yong Chun
2020-02-27
When Relation Networks meet GANs: Relation GANs with Triplet Loss
| Runmin WuKunyao ZhangLijun WangYue WangPingping ZhangHuchuan LuYizhou Yu
2020-02-24
Improved Consistency Regularization for GANs
Zhengli ZhaoSameer SinghHonglak LeeZizhao ZhangAugustus OdenaHan Zhang
2020-02-11
Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Milad MozafariLeila ReddyRufin VanRullen
2020-01-31
HRFA: High-Resolution Feature-based Attack
| Zhixing YeSizhe ChenPeidong ZhangChengjin SunXiaolin Huang
2020-01-21
Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
Mohamed El Amine SeddikCosme LouartMohamed TamaazoustiRomain Couillet
2020-01-21
A Uniform Generalization Error Bound for Generative Adversarial Networks
Anonymous
2020-01-01
CNN-generated images are surprisingly easy to spot... for now
| Sheng-Yu WangOliver WangRichard ZhangAndrew OwensAlexei A. Efros
2019-12-23
Detecting GAN generated errors
Xiru ZhuFengdi CheTianzi YangTzuyang YuDavid MegerGregory Dudek
2019-12-02
LOGAN: Latent Optimisation for Generative Adversarial Networks
| Yan WuJeff DonahueDavid BalduzziKaren SimonyanTimothy Lillicrap
2019-12-02
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models
| Giannis DarasAugustus OdenaHan ZhangAlexandros G. Dimakis
2019-11-27
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
| Ceyuan YangYujun ShenBolei Zhou
2019-11-21
Deep Motion Blur Removal Using Noisy/Blurry Image Pairs
Shuang ZhangAda ZhenRobert L. Stevenson
2019-11-19
Consistency Regularization for Generative Adversarial Networks
Han ZhangZizhao ZhangAugustus OdenaHonglak Lee
2019-10-26
Improving sample diversity of a pre-trained, class-conditional GAN by changing its class embeddings
| Qi LiLong MaiMichael A. AlcornAnh Nguyen
2019-10-10
Attribute Manipulation Generative Adversarial Networks for Fashion Images
Kenan E. Ak Joo Hwee Lim Jo Yew Tham Ashraf A. Kassim
2019-10-01
High Fidelity Speech Synthesis with Adversarial Networks
| Mikołaj BińkowskiJeff DonahueSander DielemanAidan ClarkErich ElsenNorman CasagrandeLuis C. CoboKaren Simonyan
2019-09-25
Adversarial Video Generation on Complex Datasets
Aidan ClarkJeff DonahueKaren Simonyan
2019-07-15
Mean Spectral Normalization of Deep Neural Networks for Embedded Automation
| Anand Krishnamoorthy SubramanianNak Young Chong
2019-07-09
Large Scale Adversarial Representation Learning
| Jeff DonahueKaren Simonyan
2019-07-04
Deep Compressed Sensing
| Yan WuMihaela RoscaTimothy Lillicrap
2019-05-16
DISTRIBUTIONAL CONCAVITY REGULARIZATION FOR GANS
Shoichiro YamaguchiMasanori Koyama
2019-05-01
On Computation and Generalization of Generative Adversarial Networks under Spectrum Control
Haoming JiangZhehui ChenMinshuo ChenFeng LiuDingding WangTuo Zhao
2019-05-01
Improved Precision and Recall Metric for Assessing Generative Models
| Tuomas KynkäänniemiTero KarrasSamuli LaineJaakko LehtinenTimo Aila
2019-04-15
Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
| Zhiming ZhouJian ShenYuxuan SongWeinan ZhangYong Yu
2019-04-02
Conditional GANs For Painting Generation
Adeel MuftiBiagio AntonelliJulius Monello
2019-03-06
High-Fidelity Image Generation With Fewer Labels
| Mario LucicMichael TschannenMarvin RitterXiaohua ZhaiOlivier BachemSylvain Gelly
2019-03-06
Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
Mayank SharmaAayush YadavSumit SomanJayadeva
2019-01-31
Unsupervised Image-to-Image Translation with Self-Attention Networks
| Taewon KangKwang Hee Lee
2019-01-24
On Computation and Generalization of GANs with Spectrum Control
Haoming JiangZhehui ChenMinshuo ChenFeng LiuDingding WangTuo Zhao
2018-12-28
How does Lipschitz Regularization Influence GAN Training?
Yipeng QinNiloy MitraPeter Wonka
2018-11-23
Neural Lander: Stable Drone Landing Control using Learned Dynamics
Guanya ShiXichen ShiMichael O'ConnellRose YuKamyar AzizzadenesheliAnimashree AnandkumarYisong YueSoon-Jo Chung
2018-11-19
Discriminator Rejection Sampling
| Samaneh AzadiCatherine OlssonTrevor DarrellIan GoodfellowAugustus Odena
2018-10-16
Metropolis-Hastings view on variational inference and adversarial training
Kirill NeklyudovEvgenii EgorovPavel ShvechikovDmitry Vetrov
2018-10-16
Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images
| Faisal MahmoodDaniel BordersRichard ChenGregory N. McKayKevan J. SalimianAlexander BarasNicholas J. Durr
2018-09-29
Large Scale GAN Training for High Fidelity Natural Image Synthesis
| Andrew BrockJeff DonahueKaren Simonyan
2018-09-28
Generative Adversarial Network with Spatial Attention for Face Attribute Editing
| Gang ZhangMeina KanShiguang ShanXilin Chen
2018-09-01
Rethinking Monocular Depth Estimation with Adversarial Training
Richard ChenFaisal MahmoodAlan YuilleNicholas J. Durr
2018-08-22
The relativistic discriminator: a key element missing from standard GAN
| Alexia Jolicoeur-Martineau
2018-07-02
CapsGAN: Using Dynamic Routing for Generative Adversarial Networks
| Raeid SaqurSal Vivona
2018-06-07
Self-Attention Generative Adversarial Networks
| Han ZhangIan GoodfellowDimitris MetaxasAugustus Odena
2018-05-21
Spectral Normalization for Generative Adversarial Networks
| Takeru MiyatoToshiki KataokaMasanori KoyamaYuichi Yoshida
2018-02-16
cGANs with Projection Discriminator
| Takeru MiyatoMasanori Koyama
2018-02-15

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories