Fuzzy masks: boosting radiomic reliability in head and neck tumors amid delineation uncertainty

Background and purpose The clinical utility of radiomics in head-and-neck (H&N) cancer is hindered by poor reliability caused by delineation uncertainties from the use of binary mask (BinMask). This study introduced a fuzzy mask (FuzzMask) approach to enhance the reliability of computed tomography (CT)-based radiomics for precision prognosis. Materials and methods This retrospective study included 2,539H&N cancer patients (855 laryngeal cancer (LC), 1,336 oropharyngeal cancer (OPC), 348 nasopharyngeal carcinoma (NPC)). Delineation uncertainty was simulated via perturbation techniques. Radiomic features (RFs) were extracted using BinMask and FuzzMask, respectively. The evaluation focused on feature reliability and relevance via the Intraclass Correlation Coefficient (ICC) and hierarchical clustering. In addition, the predictive performance and output reliability of Penalized Cox's proportional hazard models were assessed using the Concordance index (C-index) and ICC, respectively. Results The FuzzMask improved feature reliability, yielding 21, 29, and 5 additional reliable features for LC, OPC, and NPC cancers, respectively, compared to BinMask. The FuzzMask also reduced feature redundancy, generating up to 70 more clusters in hierarchical clustering, particularly for smaller tumors in complex peritumoral environments. While FuzzMask showed marginal improvements in predictive performance (C-index: +0.1% for OPC, +0.4% for NPC, p > 0.05). However, model reliability was enhanced by FuzzMask, with ICC values increasing by 0.024, 0.022, and 0.007 for LC, OPC, and NPC, respectively, compared to BinMask ( p ≥ 0.05). Conclusions The proposed FuzzMask technique significantly improved feature reliability and model robustness against delineation uncertainty, offering greater trustworthiness for clinical translation, although predictive accuracy remains unaffected