Evolutionary Multiobjective Neural Architecture Search for Binary Neural Networks by Two-Stage Optimization

Binary neural networks (BNNs) have been applied in limited resources and mobile devices because of their extreme model compression ability. However, manually designing suitable architectures is challenging given the specialized structure of binarized operations. Neural architecture search (NAS) provides a promising approach for designing high-performance BNN architectures. In practice, various situations require networks with different parameter sizes and performance levels. Therefore, this article proposes a multiobjective evolutionary NAS algorithm for BNNs based on a two-stage training strategy (MO-TS-BNAS) to solve these problems. First, the ApproxSign function is used to approximate the gradient error in the training of BNNs. To avoid the small model trap problem, two auxiliary objectives are introduced in nondominated sorting to retain larger models with similar errors. Then, a two-stage training strategy with flexible use of auxiliary objectives is proposed, forming the selection mechanism in environmental selection. The path dropout method is used in the second stage to prevent hypernetwork overfitting. In addition, the mini-batch gradient descent strategy is improved to speed up individual architecture evaluation and reduce time cost in the search process. Finally, the full precision baseline search space is binarized for general experimental comparison. Our MO-TS-BNAS algorithm balances the two different objective functions of the model size and error. A large number of experiments are carried out on the CIFAR10 and ImageNet datasets, and the results show the effectiveness of the proposed method.