Large Language Models for the Classification of Verbal Communication in Dementia Care (Preprint)

Read full text Summarise with VIRAT
Authors: Atsushi Nakazawa, Yuto Hogan, Yuri Nagai, 美緒 伊東
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

BACKGROUND Effective verbal communication is a core component of nursing care, particularly in dementia care such as Humanitude. However, manual evaluation of communication quality is time-consuming, subjective, and difficult to scale in training settings. Large language models (LLMs) may enable automated and scalable analysis of verbal communication in caregiving. OBJECTIVE This study evaluated whether LLMs can reliably classify verbal communication in nursing care training sessions and detect differences in communication patterns across caregiver expertise levels. METHODS Care sessions involving simulated patients were conducted with 18 participants, including Humanitude instructors, intermediate practitioners, and novice nurses. Audio recordings were transcribed, segmented into utterances, and classified into 6 communication categories: positive/affectionate expression, request/suggestion, gratitude, explanation, question/confirmation, and none. Four human annotators independently labeled the utterances, and the same transcripts were analyzed using GPT, Claude, and Gemini. Agreement was evaluated using pairwise agreement rates and Cohen’s kappa coefficients. Model performance was further assessed against consensus labels derived from multiple annotators, and non-inferiority/equivalence was tested using two one-sided tests (TOST). RESULTS Inter-annotator agreement among the human annotators was moderate, with pairwise agreement rates ranging from 64.44% to 74.21% and Cohen’s kappa values ranging from 0.554 to 0.664. Among the evaluated LLMs, Claude showed the highest agreement with human annotations, followed by Gemini and GPT. Against consensus labels, Claude achieved the highest accuracy (0.836 for ≥2-annotator consensus; 0.902 for ≥3-annotator consensus), followed by Gemini (0.779; 0.837) and GPT (0.672; 0.732). TOST analysis showed that Gemini achieved statistical equivalence with human annotation (p=0.040), while Claude demonstrated non-inferiority and exceeded the human baseline (p=0.001). Across caregiver groups, instructors showed a higher proportion of positive/affectionate expressions, whereas novice caregivers showed a higher proportion of task-oriented and uncategorized utterances. Overall, LLM-based classification reproduced the general communication patterns observed in human annotations. CONCLUSIONS LLM-based classification demonstrated reliability comparable to human annotation for caregiving communication analysis. Claude showed the strongest overall performance, and Gemini achieved statistical equivalence with human annotation. These findings suggest that LLM-based analysis may provide a scalable and objective approach to assessing communication behaviors in Humanitude training and support communication assessment in nursing and medical education. CLINICALTRIAL Gunma University Hospital (HS2024-044)

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Artificial Intelligence in Healthcare Speech and dialogue systems

Publication details

Publication date
25 March 2026
Language
EN
Document type
Preprint