Advancing skin cancer detection with a hybrid deep learning model integrating CNNs and transformer architectures

It is essential to early detect skin cancer, especially melanoma since it significantly affects patients' survival. However, an experienced dermatologist still has difficulty distinguishing between a healthy lesion and a tumor; there are fine distinctions between benign and malignant tumors. In this research, we have developed a combined deep learning system, which uses Convolutional Neural Networks (CNNs) in combination with Vision Transformers (ViTs), to develop an automated diagnostic tool for detecting skin cancer based upon skin images. In developing this hybrid model, we utilized EfficientNet-B4 as a local feature extractor and a Vision Transformer as a global feature extractor. To combine these two feature extractors, we employed a special fusion module. This module used concatenation to merge the feature sets from each branch into a single layer and then processed them through a multi-layer perception. We were able to train and test the model using the ISIC 2020 data set, which contains 33,126 skin images, with successively improved training methodologies using a technique called 5-fold cross-validation. On the test set, the proposed hybrid model had an accuracy of 95.4%, a sensitivity of 90.7%, a specificity of 95.1%, and an AUC-ROC of 0.982. The above results show that the hybrid CNN-Transformer design performed better than the previous state-of-the-art EfficientNet-B4 + Attention model (accuracy of 92.1%) and had significant increases in both sensitivity (+1.8%) and specificity (+1.0%). These findings indicate that a hybrid CNN-Transformer design can provide a hopeful means of assisting physicians in diagnosing skin cancer, thus potentially improving physician decision making.