Selecting, Scaling, and Measuring the Value of Ambient AI in a Nonacademic Health System: Multiphase Pilot Study

Abstract Background Most US health systems operate on a local or regional scale and face substantial financial and staffing pressures, which are intensified by challenges related to physician satisfaction and difficulties in recruitment and retention. Ambient artificial intelligence (AI) documentation solutions have the potential to reduce burdens and improve satisfaction, but vendor selection is often undermined by cognitive biases, unvalidated marketing claims, and limited real-world testing. Objective To address this, McLeod Health developed and implemented an objective, multiphase approach to evaluate and adopt an ambient AI solution across its multihospital system. Methods Our evaluation process began in spring 2024 with 4 leading vendors tested through live clinical simulations using 15 complex outpatient scripts, with organizational leaders serving as standardized patients. Ambient patient encounters were captured in real time, and AI-generated notes were scored by physicians, revenue cycle experts, and nonclinical reviewers for accuracy, billing quality, and readability. The top 2 vendors advanced to demonstrations of Epic workflow integration, with physician usability feedback guiding the final selection. In the third phase, the chosen vendor underwent a 90-day pilot across 5 ambulatory specialties beginning in October 2024, followed by system-wide implementation in January 2025. Key performance indicators included documentation time, coding, and financial trends, as well as patient and provider satisfaction. All statistical comparisons were 2-sided using a 95% CI. Results The 3-phase evaluation process resulted in careful vendor selection. The pilot showed a 35.4% decrease in pajama time ( P =.054, trend toward significance) and a 28.3% decrease in time in notes (n=23; P