Segmetron: Sample-efficient model-agnostic semantic segmentation with a trustworthy reject option via PQ learning

Semantic segmentation can help gain a deeper understanding of a depicted scene and deliver a variety of transformative technologies. However, its application is limited by the covariate shift and the lack of reliable detection techniques. In this study, we introduce a trustworthy, sample-efficient, distribution-free and model-agnostic hypothesis test, named Segmetron, to detect image-level covariate shift in semantic segmentation. To assess an unlabelled target domain, Segmetron relies on an existing (but random) pre-trained semantic segmentation model and the labelled samples used to train it. The test statistic is based on the sample disagreement rate of two ensemble models trained to disagree with the baseline segmenter on unseen samples from the training and deployment sets, respectively. To obtain theoretical guarantees on unknown arbitrary test distributions, we build on recent work on the PQ learning setting of selective classification and extend it to a different discriminative model (i.e. segmenters). To train the enforced disagreement segmenters of each ensemble, we innovatively propose loss functions (to agree) which are more apropos to the semantic segmentation task and comply with the training of the baseline segmenter. We demonstrate that Segmetron outperforms other state-of-the-art techniques in terms of statistical power on two challenging real-world tasks from the cardiovascular magnetic resonance imaging field, concerned with two or more semantic classes, given access to only one image. This work aligns with “Responsible AI” principles, supporting reliable deployment of AI by enhancing robustness. It can potentially enable the widespread adoption of deep learning semantic segmentation technologies across various fields. • Segmetron is proposed to detect image-level covariate shift in semantic segmentation. • Segmetron is reliable, sample-efficient, distribution-free and model-agnostic. • PQ learning guarantees strong performance for unknown arbitrary test distributions. • Real-world natural covariate shifts from cardiovascular imaging field are analysed. • Segmetron outperforms state-of-the-art methods, given access to only one image.