External Validation, Recalibration, and Extension of a Prediction Model of Early Acute Kidney Injury in Critically Ill Children Using Multicenter Data

BACKGROUND: Acute kidney injury (AKI) is common in critically ill children and is associated with high morbidity and mortality. Risk prediction models designed for clinical decision support implementation can facilitate early identification and proactive mitigation of AKI risk. Existing models have primarily been validated using single-center data, partly because of the lack of appropriately detailed multicenter datasets. OBJECTIVE: To determine the performance of a single-center model to predict new AKI at 72 hours of ICU admission in children across two multicenter datasets and refine this model to improve prediction performance while maintaining acceptable alert burden. DERIVATION AND VALIDATION COHORTS: We analyzed two datasets: the Pediatric Learning Health System Network-Virtual Pediatric Systems (PEDSNET-VPS) dataset, created through the linkage of PEDSnet electronic health record (EHR) extraction with VPS (LLC, http://www.myvps.org), and the PICU Data Collaborative dataset, created through EHR extraction and harmonization from eight participating institutions. We divided each dataset into a derivation and test split. PREDICTION MODEL: We first recalibrated an existing single-center model and measured discrimination (area under the receiver operating characteristic curve [AUROC] and area under the precision-recall curve [AUPRC]) and performance at multiple cutpoints. We next added features available at 12 hours of ICU admission, optimizing by precision and recall. We measured discrimination and performance at multiple cutpoints and identified the features contributing most to the risk score. RESULTS: In total we analyzed 186,540 ICU admissions. We found early AKI by serum creatinine criteria within 72 hours of admission in 2.2-2.7%. Initial recalibration of an existing single-center model demonstrated poor discrimination (AUROC 0.65-0.78; AUPRC 0.10-0.12). Following the addition of new features, the model had higher AUROC (0.80-0.88) and AUPRC (0.13-0.22). CONCLUSIONS: In this first use of two new multicenter datasets, we found improved performance in a model designed using features available at 12 hours of ICU admission, balancing sensitivity and precision to predict patients at risk for AKI development.