Multi-modal adaptive empathy assessment in online dyadic interaction using bi-directional multi-layer perceptron-mixer and dynamic weights fusion

Empathy modeling in online interactions presents a significant challenge due to its dynamic, context-sensitive, and bidirectional nature. To address these complexities, we propose the Bi-directional MLP-Mixer (Bi-Mixer) and dynamic weights fusion model—a novel neural architecture that captures temporal dependencies in both forward and reverse directions and performs dynamic multi-modal fusion using a cross-attention mechanism. The model adaptively reweights input from visual, audio, text, and biological modalities based on contextual cues, enabling more accurate real-time empathy prediction. To support the training and evaluation of our proposed model, we build the Multi-modal online interaction EMPathy (Multi-EMP) dataset, consisting of unscripted dyadic conversations recorded through online video conferencing. The dataset includes four synchronized modalities: video, audio, text, and bio-signals such as electrodermal activity, blood volume pulse, temperature, and metabolic equivalent of task. It enables dual empathy assessment based on both explicit self-reports and computed emotional alignment between speaker and listener. Our contributions include: (1) the Bi-Mixer and dynamic weights fusion model for bidirectional and adaptive multi-modal representation learning, (2) the release of a comprehensive multi-modal dataset for naturalistic empathy research, and (3) a dual-perspective framework for empathy evaluation.