Hybrid Vision Transformer and Convolutional Network for Automated Road Damage Detection

ABSTRACT Efficient and accurate road damage detection is essential for transportation safety and infrastructure management. Traditional convolutional models often struggle with complex contextual patterns in real‐world scenarios. This paper introduces a novel hybrid architecture integrating convolutional neural networks (CNNs) for local texture extraction with vision transformers (ViTs) enhanced by shifted patch tokenization (SPT) and multi‐head self‐attention (MHSA) for global spatial reasoning. The framework effectively identifies surface irregularities such as cracks, potholes, and asphalt deformation. There are 1600 high‐resolution images in the original data set. Domain‐specific augmentation methods were used to overcome the problem of data scarcity, as well as to increase the generalization of the models by expanding the size of the training set to 6300 images throughout training. Validation and test sets had original unaugmented images so that there is unbiased evaluation. The ViT‐ResNet50 hybrid achieves state‐of‐the‐art classification accuracy of 99.1%, outperforming variants based on MobileNetV2, EfficientNet‐B0, and DenseNet121. explainable AI (XAI) techniques, including gradient‐weighted class activation mapping (Grad‐CAM) and LIME, provide visual insights into model decisions, enhancing transparency and trust. Ablation studies confirm that SPT and MHSA improve performance by an average of 10.4%. This approach represents a significant advancement toward interpretable, deployable systems for intelligent transportation and smart city ecosystems.