KNAS: Green Neural Architecture Search

26 Nov 2021  ·  Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu sun, Hongxia Yang ·

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at \url{https://github.com/Jingjing-NLP/KNAS} .

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Neural Architecture Search NAS-Bench-201, CIFAR-10 KNAS (k=40) Accuracy (Test) 93.43 # 22
Neural Architecture Search NAS-Bench-201, CIFAR-100 KNAS (k=40) Accuracy (Test) 71.05 # 21
Neural Architecture Search NAS-Bench-201, ImageNet-16-120 KNAS (k=40) Accuracy (Test) 45.05 # 27
Neural Architecture Search NATS-Bench Topology, CIFAR-10 KNAS (Xu et al., 2021) Test Accuracy 93.05 # 8

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Neural Architecture Search NATS-Bench Topology, CIFAR-100 KNAS (Xu et al., 2021) Test Accuracy 68.91 # 8
Neural Architecture Search NATS-Bench Topology, ImageNet16-120 KNAS (Xu et al., 2021) Test Accuracy 34.11 # 8

Methods


No methods listed for this paper. Add relevant methods here