Sandwich Batch Normalization

22 Feb 2021 Xinyu Gong Wuyang Chen Tianlong Chen Zhangyang Wang

We present Sandwich Batch Normalization (SaBN), an embarrassingly easy improvement of Batch Normalization (BN) with only a few lines of code changes. SaBN is motivated by addressing the inherent feature distribution heterogeneity that one can be identified in many tasks, which can arise from data heterogeneity (multiple input domains) or model heterogeneity (dynamic architectures, model conditioning, etc.)... (read more)

PDF Abstract PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Neural Architecture Search NAS-Bench-201, CIFAR-100 DARTS-SaBN Accuracy (Test) 71.56 # 1
Neural Architecture Search NAS-Bench-201, ImageNet-16-120 DARTS-SaBN Accuracy (Test) 45.85 # 5

Methods used in the Paper


METHOD TYPE
SaBN
Normalization
Batch Normalization
Normalization