Development and Evaluation of a Learning-by-Teaching Framework for CBT Skill-Building using Commonly Accessible AI: A Pilot Study (Preprint)

BACKGROUND Current AI interventions in mental health positions LLMs to act as therapists, raising concerns regarding simulated emotional bond and clinical safety. These systems risk patients becoming dependent on the tool, instead of fostering their own therapeutic skills for long-term recovery. OBJECTIVE This paper explores design considerations for adapting commonly-used LLMs (e.g., Gemini, ChatGPT, Llama) for clinical use, using them as a skill-building tool rather than a replacement for therapists. METHODS Guided by the educational theories, we developed a dual-persona chatbot. The first persona is a distressed character with cognitive distortion; the second persona is a facilitator that provides the user with scaffolding and instructions to navigate the interaction safely and successfully. Users are tasked to “help” the first persona, with the aid of the second persona, by identifying and restructuring their cognitive distortions. Through a process involving initial testing, establishing personas, and ensuring fidelity/safety, we developed three versions of the system. Four raters with varying clinical expertise assessed simulated interactions across four domains: Character Fidelity, Effective Facilitation, Boundary Management, and Overall Utility. RESULTS Inter-rater reliability among the raters was high (ICC = 0.76). The final version of the system was rated as effective in terms of character fidelity, learning facilitation, and clinical boundaries. The largest improvement across versions was in the construction of an effective and safe learning environment (F2,61 = 42.11, P