Clinician Performance in Training Data Curation for an Arrhythmia Machine Learning Model: Is Anyone Qualified?

Abstract Background Early identification of arrhythmias in the intensive care unit (ICU) is important to prevent ICU morbidity and mortality. Timely arrhythmia detection relies on bedside providers’ telemetry interpretation. Machine learning (ML) models can function as clinical support tools to facilitate diagnoses. ML model development requires well-curated training data. The differential performance between labelers of different roles and experience is currently unknown. Methods This was a prospective observational study with frontline providers. 300 (200 original, 100 duplicate) 10-second telemetry tracings were labeled including sinus rhythm, 2 nd /3 rd degree atrioventricular (AV) block, junctional ectopic tachycardia (JET), ectopic atrial tachycardia (EAT), and reentrant supraventricular tachycardia (SVT). Interrater reliability was calculated against the ground truth label as the primary performance measure (intrarater reliability for consistency utilizing duplicate labels). Results 11 participants completed the study: 1 Cardiology fellow, 4 pediatric ICU fellows, 2 pediatric cardiac ICU fellows, 3 pediatric cardiac ICU NPs, and 1 Pediatrics resident. Highest level of agreement was moderate (κ 0.68, p Junctional (0.47), EAT (0.25), SVT (0.49). There was good intrarater reliability (κ 0.71 [median], p