Automated multimodal visual analytics of social and epistemic dimensions in team communication for reflective practice in healthcare simulation

Effective team communication is critical in high-stakes sectors such as healthcare. However, learners often struggle to reflect and improve on their communication skills within these fast-paced scenarios. While educators play a vital role in guiding students to reflect on the social and epistemic aspects of their communication, they require evidence to facilitate learners in recalling their experience to reflect effectively. Although video recordings are often suggested as a solution, the labour-intensive nature of reviewing and editing footage makes this approach often impractical in educational settings. To address this gap, this paper proposes an AI-powered, Multimodal Learning Analytics (MMLA) system that automatically generates visualisations of team communication in healthcare simulations. Specifically, the system employs spatial analytics and a transformer-based language model (i.e., BERT) to process learners’ physical positioning and speech data, enabling the visualisation of communication dynamics as Social and Epistemic Networks, serving as evidence to facilitate the reflection process. We investigated the pedagogical potential of this approach to facilitate guided team reflections by conducting qualitative analyses of how educators integrated the analytics into 28 authentic team debriefing sessions. Moreover, we conducted follow-up interviews with 15 learners and 5 educators to understand their perceptions of whether the network visualisations can enrich reflective learning and support the development of team communication skills. The results indicate that educators effectively utilised social networks to guide reflection on engagement and roles, and epistemic networks to examine closed-loop communication strategies. While social networks were comprehensible to learners, epistemic networks presented interpretation challenges for some learners, highlighting a need for simplified representations to illustrate the epistemic dimension of communication. Ultimately, the identified comprehensibility and perceived usefulness suggest the potential feasibility of adapting social and epistemic networks, traditionally utilised as analytical tools for researchers, into pedagogical instruments that support evidence-informed reflection in dynamic physical learning environments.