A Guideline-Concordant Chatbot Framework for Structured Colorectal Cancer Screening: Multistage Feasibility Study

BACKGROUND: Colorectal cancer (CRC) screening relies on structured risk assessment and guideline-concordant communication, which remain challenging to implement consistently in real-world practice. Digital tools based on large language models (LLMs) may support such workflows, but their feasibility and safety in structured screening contexts have not been well evaluated. OBJECTIVE: This study aimed to develop and evaluate a guideline-concordant chatbot framework for structured CRC screening. METHODS: A multistage feasibility study was conducted. In phase 1, baseline performance of contemporary LLMs was assessed using 14 standardized CRC screening questions and validated expert-rated instruments. In phase 2, structured prompt versions were iteratively optimized based on screening guidelines and expert feedback and tested using simulated user scenarios. In phase 3, the optimized chatbot was evaluated in 50 screening-eligible adults to assess feasibility, safety, and guideline concordance. RESULTS: In phase 1, both LLMs demonstrated satisfactory performance in responding to standardized CRC screening questions, with DISCERN instrument for AI-generated content scores of 12.02 (SD 0.30) for GPT-4o and 13.36 (SD 0.26) for DeepSeek-V3, Global Quality Score scores of 3.96 (SD 0.10) for GPT-4o and 4.39 (SD 0.08) for DeepSeek-V3, Natural Language Assessment Tool for AI scores of 21.41 (SD 0.34) for GPT-4o and 22.73 (SD 0.27) for DeepSeek-V3, and Patient Education Materials Assessment Tool adapted for AI outputs scores of 0.900 (SD 0.016) for GPT-4o and 0.906 (SD 0.015) for DeepSeek-V3. In phase 2, iterative prompt optimization significantly improved all 6 expert-rated dialogue evaluation dimensions (all P values