DU-DARTS: Decreasing the Uncertainty of Differentiable Architecture Search
Differentiable Neural Architecture Search (DARTS) recently attracts a lot of research attention because of its high efficiency. However, the competition of candidate operations in DARTS introduces high uncertainty for selecting the truly important operation, thus leading to serious performance collapse. In this work, we decrease the uncertainty of differentiable architecture search (DU-DARTS) by enforcing the distribution of architecture parameters to approach the one-hot categorical distribution and by replacing the zero operation with a gate switch. Without any extra search cost, our method achieves state-of-the-art performance with 2.32%, 16.74%, and 24.1% test error on CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. Moreover, DU-DARTS can robustly find an excellent architecture on NAS-Bench-1Shot1 and NAS-Bench-201, which further demonstrates the effectiveness of our method. The source code is available at https://github.com/ShunLu91/DU-DARTS.
PDF AbstractCode
CIFAR-10
ImageNet
CIFAR-100
NAS-Bench-201
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Neural Architecture Search | CIFAR-10 | DU-DARTS | Top-1 Error Rate | 2.32% | # 11 | |
| Search Time (GPU days) | 0.4 | # 13 | ||||
| Parameters | 3.8M | # 30 | ||||
| Neural Architecture Search | CIFAR-100 | DU-DARTS | Percentage Error | 16.74 | # 9 | |
| PARAMS | 3.1M | # 6 | ||||
| Neural Architecture Search | ImageNet | DU-DARTS | Top-1 Error Rate | 24.1 | # 98 | |
| Accuracy | 75.9 | # 77 | ||||
| Params | 5.3M | # 36 | ||||
| Neural Architecture Search | NAS-Bench-201, CIFAR-10 | DU-DARTS | Accuracy (Test) | 93.86 | # 19 | |
| Accuracy (Val) | 91.21 | # 16 | ||||
| Neural Architecture Search | NAS-Bench-201, CIFAR-100 | DU-DARTS | Accuracy (Test) | 71.84 | # 19 | |
| Accuracy (Val) | 71.88 | # 18 | ||||
| Neural Architecture Search | NAS-Bench-201, ImageNet-16-120 | DU-DARTS | Accuracy (Test) | 45.94 | # 20 |
