Predictive Modeling of Childhood Disease Severity at a Telemedicine Service: Evidence from the Improving Nighttime Access to Care and Treatment Studies in Haiti

OBJECTIVE: To develop models within pediatric telemedicine that identify potentially "sick" cases for additional safety checks and integrate those models into electronic clinical decision support tools. STUDY DESIGN: We conducted a secondary analysis of paired virtual and in-person examinations across 3 consecutive implementation studies conducted at a telemedicine and medication delivery service in Haiti. Artificial intelligence/machine learning (XGBoost) was applied to derive models focused on identifying "sick" patients (moderate or severe) and those requiring escalation. Given the limited sample size, we used an ensemble method based on gradient boosted decision trees. The area under the receiver operating characteristic curve (AUC) was the primary outcome measure. RESULTS: A total of 683 paired records were available for this secondary analysis from 2225 participants enrolled. The median age was 15 months and 47% were female. For prediction of a "sick" child, we found an AUC of 0.82 (95% confidence interval [CI]: 0.78-0.86) after 5-fold cross-validation; calibration slope and intercept were 1.09 (95% CI: 0.91-1.26) and 0.16 (95% CI: 0.03-0.35), respectively. For prediction of escalation, we found an AUC of 0.77 (95% CI: 0.73-0.81); calibration slope and intercept were 0.81 (95% CI: 0.66-0.96) and 0.08 (95% CI: 0.11-0.26), respectively. CONCLUSIONS: These methods and findings offer an innovative and important proof-of-concept for how to aid clinical decision-making within pediatric telemedicine environments. The models require external validation prior to electronic clinical decision support integration and deployment. Once validated, the models will provide a critical safety check for experienced providers and digitally convey expertise to new providers.