Brain tumour identification using improved YOLOv8

Accurately identifying the extent of brain tumours remains a major challenge in brain cancer treatment, primarily due to the difficulty in detecting tumour boundaries from MRI scans. Manual detection is time-consuming and requires expert knowledge. In this study, we propose a modified YOLOv8 model for precise brain tumour detection in MRI images. We replaced the traditional non-maximum suppression (NMS) with a real-time detection transformer (RT-DETR) to eliminate hand-designed filtering. Additionally, we integrated ghost convolution to reduce computational costs while maintaining accuracy, and introduced a vision transformer block in the backbone to enhance context-aware feature extraction. The model was trained and tested on a publicly available brain tumour dataset. Experimental results show that our modified YOLOv8 outperforms the original YOLOv8 and other popular object detectors including faster R-CNN, mask R-CNN, YOLOv3-v5, SSD, RetinaNet, EfficientDet, and DETR, achieving a mAP@0.5 of 0.91.