High-resolution fluorescence microscopy dataset for neuron segmentation and morphological analysis

The accurate segmentation and analysis of neurons in 2D fluorescent microscopy images is essential to advance our understanding of intricate neural structures. In this study, we introduce a novel, expertly annotated dataset of dental stem cell created especially for instance segmentation tasks involving three classes: cell, incomplete cell, and other. Using this dataset, we evaluate the performance of three cutting-edge deep learning models: Mask R-CNN, YOLOv8, and YOLOv11. To guarantee excellent training and evaluation, thorough pre-processing and well-selected augmentation techniques were used. The results show that YOLOv8 outperformed the other models, indicating its effectiveness in capturing complex neuronal features. This dataset addresses the problem of segmenting complex and overlapping neuron structures, making it an invaluable resource for the computer vision and neuroscience communities. Future work will expand this dataset to include 3D images and explore advanced training strategies to further improve segmentation accuracy and generalizability.