Cell segmentation in microscopy images using a SAM-based U-Net architecture and a novel dataset

BACKGROUND AND OBJECTIVE: Accurate cell instance segmentation is essential for analysing morphology, phenotypes, and dynamics in biomedical research. Challenges such as dense cell populations, blurred boundaries, and overlapping structures hinder reliable segmentation, and deep learning models remain constrained by the scarcity of fully annotated datasets. METHODS: We present mCellSeg, a dataset of 200 expert-annotated microscopy images containing 16,199 cells from HEK-293T and HUVEC lines, covering diverse morphologies and confluency levels. To utilise this dataset, we propose mSAMUNet, a hybrid neural network that integrates the Segment Anything Model (SAM) with a multiscale U-Net-inspired encoder. This architecture combines the global context modelling of Transformers with the local detail extraction of CNNs, while multiscale branches enhance performance on cells of varying sizes. We benchmarked mSAMUNet against U-Net, StarDist, SAM, and microSAM across four datasets using five-fold cross-validation. RESULTS: mSAMUNet achieved superior performance across all datasets and metrics, including F1 score, SA50, SA75, and mean SA. On the mCellSeg dataset with dense and blurred cell boundaries, mSAMUNet outperformed microSAM (F1 = 0.7071 vs. 0.6994). On the NeurIPS22 dataset, it achieved an F1 score of 0.8675 compared to 0.8483 for microSAM. U-Net and StarDist performed moderately, while SAM was the weakest overall. These results underscore the effectiveness of combining CNN-based multiscale encoders with Transformer-based models for biomedical image segmentation. CONCLUSION: mSAMUNet provides a robust and accurate solution for microscopy cell instance segmentation, surpassing state-of-the-art methods across multiple datasets. Alongside the mCellSeg dataset, this work delivers valuable resources for advancing automated cell analysis with applications in drug discovery, disease modelling, and high-throughput biomedical research.