Pre‐implementation safety evaluation of an AI decision‐support system for surgical antimicrobial prophylaxis

BACKGROUND: Large language models (LLMs) show potential to support antimicrobial prescribing but require simulation-based, institution-specific safety evaluation prior to any consideration of clinical use. In Australia, antimicrobial prescribing represents a high-risk domain for digital decision-support systems due to patient safety and antimicrobial resistance implications. AIM: To characterise prescribing accuracy, error phenotypes and antimicrobial stewardship risk associated with a LLM that was provided with publicly available surgical prophylaxis guidelines during inference (without fine-tuning or model modification) across 20 simulated surgical scenarios. METHODS: Twenty simulated surgical scenarios were tested using a LLM that was prompt-conditioned with publicly available guideline text during inference, without any fine-tuning or modification of model weights. For each case, the model generated recommendations for agent, dose, timing, re-dosing and guideline citation. Outputs were independently assessed by two local clinicians familiar with the guideline, with accuracy scored across five domains and harm classified using a modified National Coordinating Council for Medical Error Reporting and Prevention (NCC MERP) Index. RESULTS: Clinically significant antimicrobial prescribing risk was identified in 10% of simulated scenarios (2/20), recognising wide confidence intervals due to the small sample size. These included omission of required anaerobic coverage and failure to redose prophylaxis in prolonged procedures. Overall guideline concordance was 4/5, with perfect dose accuracy but lower performance for timing (70%) and guideline citation (45%). CONCLUSIONS: This study demonstrates the feasibility of constructing institutionally governed, guideline-based AI systems while identifying stewardship-relevant safety risks that currently preclude clinical use without further validation.