Enhanced convolutional neural networks by using adaptive conditional dropout with dynamic regularization

Deep convolutional networks have high capacity and thus tend to overfit, especially when trained on restricted or imbalanced data, which compromises their applicability to safety applications. To address this, we propose Conditional Dropout with Dynamic Regularization (CDDR), a simple phase aware regularization method that requires no architectural changes. In CDDR, dropout is completely disabled during the first four epochs to encourage dropout free feature learning. From the fifth epoch onward, the dropout rate is adaptively determined depending on the derivative of the first order of the training loss, and then its value is fixed for the remainder of training. Meanwhile, the L2 weight decay strength is also updated according to training dynamics. Additionally, during training the CDDR method uses the conditional dropout technique, and optionally at the time of inference, Monte Carlo Dropout is applied in order to approximate predictive uncertainty. When applied to VGG-16 and ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet, CDDR surpass DropBlock, standard Dropout, and SpatialDropout, by achieving up to + 3.3% on CIFAR-10, + 9.27% on CIFAR-100, and + 3.94% on Tiny ImageNet while reducing the train test gap by 58.8% on CIFAR-100. These results show the effectiveness of CDDR in enhancing current CNN structures, especially in applications where data scarcity and class imbalance occur.