NucleoNet and DropNet: Generalist deep learning models for instance segmentation of nuclei and lipid droplets from electron microscopy images

Abstract Automating cellular organelle segmentation is key to increasing the throughput in electron microscopy (EM) and volume EM (vEM) workflows. Deep learning (DL) has accelerated this process, but model development has predominately centered on mitochondria, partly because of a scarcity of suitable training datasets for other features. Here, we crowdsourced the manual step of labeling nuclei and lipid droplets (LDs) from complex cellular EM images and trained Panoptic DeepLab (PDL) models on these large, heterogenous annotated datasets as well as on publicly available vEM datasets. NucleoNet and DropNet , the resulting instance segmentation models for nuclei and LDs, respectively, yield high-quality results on varied benchmarks. We applied these models to quantify differences between 2D and 3D in vitro cancer models versus in vivo tumors, highlighting a path toward robust quantitation in EM. NucleoNet and DropNet are publicly available on our napari plugin, empanada v1.2 , for easy point-and-click segmentation of 2D and 3D cellular EM images.