NextGen DeepTab transformer with autoencoder screening for thyroid disease diagnosis

Thyroid disorders are chronic diseases that are among the most common endocrine diseases, and they need prompt and accurate diagnosis to avoid long-term complications. This paper proposes a modular three-stage deep learning (DL) framework integrating anomaly screening, thyroid subtype classification, and recurrence risk prediction. The framework is designed as a conceptually sequential diagnostic framework, where each stage addresses a distinct diagnostic objective using publicly available datasets. Firstly, patient clinical and biochemical data are pre-processed and subjected to anomaly screening using enhanced autoencoder-based model to detect abnormal thyroid function patterns by the metric of reconstruction error, which is then compared with a statistically derived threshold. Once anomalous samples are detected, they are further processed using the FT-transformer-style model, termed as NextGen DeepTab Transformer (NextGen DTT), which incorporates hierarchical nonlinear feature embedding to perform the classification of thyroid disease subtypes. The model, by means of feature embeddings and multi-head self-attention, captures intricate nonlinear dependencies among thyroid biomarkers to correctly identify various subtypes of thyroid disorders. For patients with differentiated thyroid cancer, a last prediction step is done to compute the risk of recurrence based on clinic pathologic characteristics, which may contribute to a long-term prognosis. The proposed models validated with three public datasets. The dataset used for screening includes 6,666 normal instances and 250 anomalous instances, classification dataset comprising over 9,000 patient records for classification and 383 cases for recurrence prediction. The proposed anomaly detection model achieved 92.2% accuracy, while the classification and recurrence models achieved 89% and 86% accuracy, respectively. These results demonstrate that the proposed framework demonstrate promising predictive performance and provides structured analytical approach for thyroid disease assessment.