Shape-guided Instance Segmentation of Cell Nuclei

Due to the dense distribution and blurred boundaries of cell nuclei in pathological images, automatic cell nucleus segmentation remains a challenging task in terms of robustness. While traditional instance segmentation methods based on rectangular bounding boxes perform well in natural images, they require special adjustments when segmenting cell nuclei. Therefore, based on the YOLO framework, we propose a new instance segmentation method guided by Elliptical Shapes. (YOLO-ES). This method utilizes morphological knowledge of cell nuclei to more accurately align bounding boxes with the cell nucleus shape, thereby improving instance segmentation performance. The network model effectively integrates window attention and shifted window attention mechanisms, achieving progressive feature extraction and long-range dependency attention fusion. To handle irregularly shaped elliptical cell nuclei, a probabilistic method for calculating the Intersection over Union (IoU) of ellipses is introduced to replace the commonly used rectangular IoU calculation, thus improving the accuracy of morphological matching. We compared our method with other state-of-the-art algorithms using the 2018 Data Science Bowl dataset and the MoNuSeg dataset. Extensive experimental results demonstrate the effectiveness of our method.