Multidimensional Analysis of the Diagnostic Performance of Machine Learning Models in Disease Classification: Implications for Clinical Decision Support Systems

Clinical decision support systems face a fundamental challenge in the form of incomplete medical data. This study presents a methodology for designing an expert system for classifying disease entities according to ICD-10. The system operates under conditions of incomplete clinical data and diagnostic uncertainty. A total of 1,655 clinical cases from internal medicine (n = 616) and pediatrics (n = 1,039) were analyzed. Nineteen classification algorithms were compared. A two-stage diagnostic model (TOP-1 and TOP-3), reflecting clinical practice, was applied. The main hypothesis assumed that machine learning algorithms can effectively compensate for diagnostic uncertainty through mechanisms of differential diagnosis. In internal medicine, Gradient Boosting achieved the highest performance (84.74% accuracy), whereas in pediatrics the Multi-Layer Perceptron performed best (87.20% accuracy). Pediatrics demonstrated higher overall performance (mean TOP-3: 81.99%) than internal medicine (77.43%). The TOP-3 mechanism ensured a probability of identifying the correct diagnosis exceeding 94% in pediatrics and 91% in internal medicine, which substantially enhances the clinical safety of the system. The identification of different optimal algorithms for each specialty confirms the need to tailor modeling strategies to the clinical context. On external validation on 120 cases from a separate facility, treated as the primary benchmark of real-world performance, TOP-1 accuracy was 73.33% (internal medicine) and 78.33% (pediatrics), with TOP-3 accuracy of 86.67% and 90.00% respectively, and SHAP and LIME confirming clinically relevant predictions. The TOP-3 methodology developed constitutes an important contribution to the design of decision support systems under limited data availability.