A Language Model for Pediatric Occupational Therapy Documentation: Model Development and Pilot Study

Background: In occupational therapy, progress notes and other client-related administrative tasks are essential for providing treatment but are time-consuming. Therapists spend at least as much time on these tasks as providing care, which contributes to growing waitlists. Objective: This study aimed to create a custom large language model to make the process of writing progress notes more efficient by converting point-form scratch notes from pediatric occupational therapy treatment sessions into draft documentation in subjective-objective-assessment-plan format. Methods: Using a dataset of redacted historical progress notes, various training methods, including domain-adaptive pretraining and low-rank adaptation fine-tuning, were applied to train Llama 2 and 3 models. Since the historical notes lacked corresponding scratch notes, few-shot prompting with human-in-the-loop evaluations was used to generate synthetic scratch notes. This pairing of historical notes and generated scratch notes enabled effective fine-tuning of the Llama models on the desired task. The final model, a fine-tuned Llama 3 8B Instruct model, was piloted in a pediatric rehabilitation center and compared with Microsoft Copilot. Ten therapists used both models for 3 weeks each. Results: The custom model notes scored higher than manually written notes on clarity, completeness, relevance, and organization (P