Unified Multimodal Representation Learning With Prototype Calibration for Cross-Subject Few-Shot Emotion Recognition

Multimodal cross-subject emotion recognition has gained increasing attention due to the complementary nature of physiological and behavioral signals, such as electroencephalography (EEG) and eye movements (EM). However, modality heterogeneity and subject variability remain major obstacles, especially in few-shot scenarios. Existing approaches often combine increasingly complex architectures with unsupervised domain adaptation to address these issues, but they typically require large amounts of unlabeled target data and struggle to generalize when samples are scarce. To overcome these limitations, this article proposes a unified multimodal representation learning (UMRL) framework with prototype calibration under the few-shot paradigm. Specifically, the UMRL strategy is designed from an optimization perspective rather than architectural complexity to learn a compact and discriminative joint representation. It encourages compact representations by jointly minimizing cross-modal inconsistency and penalizing modality dominance, ensuring that complementary cues are effectively integrated in a unified embedding space. Nevertheless, emotional prototypes constructed from a few labeled samples of novel subjects are often unstable and sensitive to subject-dependent variability. To further alleviate this issue, this article introduces an emotional prototype calibration mechanism that aligns novel emotional prototypes with semantic priors learned from seen subjects to suppress noise and improve cross-subject generalization. Extensive experiments on multiple multimodal emotion recognition benchmarks with EEG and EM demonstrate that the proposed framework achieves state-of-the-art performance and delivers consistent robustness under few-shot conditions.