Peritoneal Carcinomatosis Detection Based on Improved YOLOv8 With Attention Mechanism and Optimized Backbone Integration

Peritoneal Carcinomatosis (PC) represents a severe form of metastatic cancer, with lesions spreading across the peritoneum and often leading to a poor prognosis. Early detection of PC lesions is essential for timely treatment, yet identifying smaller lesions in medical images remains challenging due to their subtle appearance and varied sizes. Traditional detection techniques often fall short in this area, underscoring the need for more advanced solutions. This study introduces improvements to the YOLOv8 architecture aimed at enhancing the detection of small PC lesions in laparoscopic images. The proposed modifications include adding a Convolutional Block Attention Module (CBAM) to improve feature extraction, incorporating a P2 layer for better detection of small objects, and using a CIoU loss function for more effective training. CIoU surpassed DIoU, WIoU, GIoU, EIoU, and SIoU loss functions in performance. Experimental results across Nano, Small, and Medium model scales demonstrated significant gains in Precision, Recall, mAP50, and F1 Score over the original YOLOv8. The modified model achieved a Precision of 0.887, Recall of 0.769, mAP50 of 0.861, and an F1 Score of 0.824. Ablation studies validated the benefits of each modification, with the enhanced model outperforming other YOLO versions (v5–v10) and transformer-based detectors such as RT-DETR, as well as alternative attention mechanisms including ECA and SA. These findings highlight the effectiveness of the proposed YOLOv8 modifications for detecting small lesions in medical imaging.