Hyperspectral Vectors for Machine Learning Segmentation of Stained and Unstained Cells From Serous Cavity Effusions

In addition to information about cell morphology, hyperspectral images (HSIs) provide consistent spectral details across hundreds of wavelengths. Cytological analysis of samples extracted from serous cavity effusions is used to diagnose their benign or malignant origin but faces several challenges due to interference of various cell types. Therefore, cells segmentation from microscopy images is necessary to enable the extraction of specific quantitative biological characteristics. Here, we propose an automated semantic segmentation of cells into nucleus and cytoplasm classes using machine learning classifiers applied to HSIs. Moreover, the proposed procedure can be applied regardless of whether the cells are stained or unstained. We defined as input data multiple hyperspectral vectors derived from spectral profiles collected at the single-pixel scale from HSIs. The ensemble models achieving highest overall accuracy on the test sets were selected as demonstrators and subsequently applied under real conditions to the regions of interest of new, independent HSIs. The limitations associated with poor contrast in unstained samples were mitigated by introducing spectral shape transformations, along with a series of predefined post-processing functions, in order to improve the robustness of external evaluation. We achieved an overall accuracy of 0.98 for stained samples and 0.97 for unstained samples on the test sets. In the external evaluation, the Dice similarity coefficient was 0.94 for stained samples and 0.70 for unstained samples for the nucleus class, respectively, 0.88 and 0.82, for the cytoplasm class. This study advances the automatic segmentation methods for HSIs of stained and unstained samples. Future work will focus on integrating these findings into automated classification frameworks for cells harvested from serous cavity fluids.