HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search

23 Feb 2021  ·  Niv Nayman, Yonathan Aflalo, Asaf Noy, Lihi Zelnik-Manor ·

Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS), however, previous methods enforce the constraint only softly. Therefore, the resulting networks do not exactly adhere to the resource constraint and their accuracy is harmed. In this work we resolve this by introducing Hard Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate formulation of the expected resource requirement and a scalable search method that satisfies the hard constraint throughout the search. Our experiments show that HardCoRe-NAS generates state-of-the-art architectures, surpassing other NAS methods, while strictly satisfying the hard resource constraints without any tuning required.

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Neural Architecture Search ImageNet HardcoreNAS_E_KD Top-1 Error Rate 19.9 # 21
Accuracy 80.1 # 16
MACs 330M # 100
Neural Architecture Search ImageNet HardcoreNAS_D_KD Top-1 Error Rate 20.5 # 30
Accuracy 79.5 # 23
MACs 290M # 88
Neural Architecture Search ImageNet HardcoreNAS_C_KD Top-1 Error Rate 21.1 # 39
Accuracy 78.9 # 30
MACs 270M # 84
Neural Architecture Search ImageNet HardcoreNAS_B_KD Top-1 Error Rate 21.2 # 41
Accuracy 78.8 # 32
MACs 250M # 81
Neural Architecture Search ImageNet HardcoreNAS_A_KD Top-1 Error Rate 21.7 # 49
Accuracy 78.3 # 38
MACs 220M # 73
Neural Architecture Search ImageNet HardcoreNAS_F Top-1 Error Rate 21.9 # 52
Accuracy 78.1 # 41
MACs 340M # 102
Neural Architecture Search ImageNet HardcoreNAS_E Top-1 Error Rate 22.1 # 54
Accuracy 77.9 # 43
MACs 330M # 100
Neural Architecture Search ImageNet HardcoreNAS_D Top-1 Error Rate 22.6 # 64
Accuracy 77.4 # 52
MACs 290M # 88
Neural Architecture Search ImageNet HardcoreNAS_C Top-1 Error Rate 22.9 # 72
Accuracy 77.1 # 58
MACs 270M # 84
Neural Architecture Search ImageNet HardcoreNAS_B Top-1 Error Rate 23.5 # 81
Accuracy 76.5 # 65
MACs 250M # 81
Neural Architecture Search ImageNet HardcoreNAS_A Top-1 Error Rate 24.1 # 98
Accuracy 75.9 # 77
MACs 220M # 73

Methods