Deepfake image detection using different combinations of vision transformers and CNNs

In recent times, deepfakes have posed a significant threat to digital media by blurring the lines between real and AI- generated content, making it easier to spread misinformation. Current detection mechanisms are failing to keep up with the diverse techniques that are being employed to create highly realistic deepfakes. In this work, we explore the benefits of hybrid deep learning architectures for classification tasks by combining the SWIN transformer with a variety of stateof-the-art CNNs such as DenseNet, ConvNeXt, and MobileNet as well as a vision transformer namely CLIP. Our goal is to leverage the hierarchical feature extraction capabilities of convolutional backbones and the global contextual understanding of transformers to improve model performance. The model is trained and evaluated on DFDC, FF++, 140k Real and Fake Faces, and Celeb-DF (v2) datasets. We assessed the models using conventional measures such test loss, accuracy, precision, recall, F1-score, and AUC-ROC. Experimental results reveal that hybrid models consistently outperform their standalone counterparts. The ConvNeXt-SWIN hybrid has the highest accuracy (93.47%) and F1-score (93.01%), while the CLIP-SWIN Hybrid shows a significant improvement in recall (94.99%) and AUC-ROC (0.9741). Our findings indicate that hybrid architecture provides a compelling upgrade path for vision-based classification tasks, particularly in performance-critical and resource-constrained situations. While our hybrid models show promising results in deepfake detection, further work can be done to improve their generalization and real-world applicability.