Explainable Affective Body Expression Recognition with Multi-Scale Spatiotemporal Encoding and LLM-Based Reasoning

Automatic emotion recognition (AER) based on 3D body expressions is a pivotal component of human-centered affective computing. Despite significant progress, existing methods often struggle to jointly capture multi-scale spatiotemporal features and bidirectional long-term dependencies, while lacking explainable reasoning mechanisms. To address these challenges, we propose the Explainable Affective Body Expression Recognition (EABER) framework. Specifically, EABER introduces a Multi-Scale Convolutional Mamba Network (MSCMNet). It employs multi-receptive-field convolutional modeling to learn multi-scale spatiotemporal representation, while a bidirectional state-space architecture is used to capture global bidirectional dependencies. Furthermore, EABER incorporates an LLM-based Emotion-Action Interpreter (EAI-LLM) for final emotion recognition with explainable affective reasoning. The spatiotemporal Serialization with Unified Masking (STSUM) strategy is introduced into the MSCMNet and EAI-LLM models, facilitating cross-dataset joint training while preserving fine-grained spatiotemporal information. Experimental results across four diverse datasets demonstrate that through joint training, the proposed EABER effectively learns shared affective motion patterns, yielding accuracy improvements of up to 7.83%. Moreover, EABER achieves superior performance in emotion recognition compared to state-of-the-art methods, and outperforms general-purpose large models, such as GPT-4o and Gemini 1.5 Pro, in explainable affective reasoning.