.png){kind=icon}
.png){kind=icon}
Return
iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2021.3105698.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this article, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25% and 24.7% are reported within 0.2 and 1.9 GPU-day for architecture search, respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.
Keywords:
Computer architecture
Degradation
Training
Microprocessors
Optimization
Computational efficiency
Neural networks
AutoML
deep learning
Differentiable ARchiTecture Search (DARTS)
Neural Architecture Search (NAS)
Journal
IF:
8.9
Papers:
7.5K
Citations:
7.2W