Accuracy, Efficiency, and usability of a Semi-Automated SMART on FHIR venous thromboembolism risk assessment App: A randomised crossover simulation study

BACKGROUND: Venous thromboembolism (VTE) is the leading cause of maternal death. Current VTE risk assessment (VTERA) tools often require manual data entry, which can reduce accuracy and affect thromboprophylaxis decisions. Building on 'Thrombocalc', an established postpartum manual VTERA tool, we developed a semi-automated VTERA application ('app') using SMART on FHIR. This app integrates with the electronic health record (EHR) to automatically extract 11 risk factors. OBJECTIVES: The primary objective was to evaluate the accuracy of a semi-automated VTERA app. The secondary objectives were to evaluate efficiency and usability. PATIENTS/METHODS: A randomised crossover study was conducted at a tertiary maternity hospital. Healthcare professionals (HCPs) completed ten simulated patient assessments using both manual and semi-automated VTERA tools. Accuracy was evaluated based on the inclusion of appropriate risk factors, accuracy of risk scores, and appropriateness of thromboprophylaxis recommendations. These recommendations pertained to thromboprophylaxis indication, duration, and dosage. Mouse clicks, keystrokes, and completion times were logged to assess efficiency. Usability of the semi-automated tool was evaluated using the System Usability Scale (SUS). Linear and logistic mixed-effects models were used to analyse accuracy and efficiency outcomes. RESULTS: Thirty-one HCPs participated. The proportion of accurate risk scores increased from 63% with the manual tool, to 87% with the semi-automated tool. Accurate thromboprophylaxis recommendations increased from 79% to 91%. Mixed-effects models demonstrated higher odds of producing accurate risk scores (OR 6.99, 95% CI 3.31-14.67) and recommendations (OR 3.59, 95% CI 1.64-7.88) with the semi-automated tool. Task completion time decreased by 49 s (95% CI 32.8-65). Mouse clicks were reduced by 12 clicks (95% CI 9.01-15.37), and keystrokes by 94% (95% CI 93.4-94.2%). The semi-automated tool achieved a mean SUS score of 89.8, indicating excellent usability. CONCLUSION: Automation can enhance postpartum VTE risk assessment. However, optimal performance requires reliable data and HCP oversight.