Effectiveness of Informed AI Use on Clinical Competence of General Practitioners and Internists: Pre-Post Intervention Study

BACKGROUND: Artificial intelligence (AI) shows promise in clinical diagnosis, treatment support, and health care efficiency. However, its adoption in real-world practice remains limited due to insufficient clinical validation and an unclear impact on practitioners' competence. Addressing these gaps is essential for effective, confident, and ethical integration of AI into modern health care settings. OBJECTIVE: This study aimed to evaluate the effectiveness of informed AI use, following a tailored AI training course, on the performance of general practitioners (GPs) and internists in test-based clinical competence assessments and their attitudes toward clinical AI applications. METHODS: A pre-post intervention study was conducted with 326 physicians from 39 countries. Participants completed a baseline test of clinical decision-making skills, covering diagnosis, treatment planning, and patient counseling; attended a 1.5-hour online training on effective AI use; and then took a similar postcourse test with AI assistance permitted (GPT-4.0). Test performance and time per question were compared before and after the training. Participants also rated AI accuracy, efficiency, perceived need for structured AI training, and their willingness to use AI in clinical practice before and after the course. RESULTS: The average test scores improved from 56.9% (SD 15.7%) to 77.6% (SD 12.7%; P