Enhancing Multimodal Facial Expression Recognition with Derived Modality Transformations Using Novel Convolutional Neural Network-Based Models

Facial Expression Recognition (FER) is a crucial task in enhancing human-computer interaction but remains challenging due to expression variability, environmental complexities, and the limitations of single-modal and existing multimodal approaches. To address these challenges, this study examines the constraints of multimodality in FER, particularly under conditions of limited data and the ineffectiveness of basic augmentations in improving model performance. We propose leveraging modality-derived transformations within modality-specific streams to enhance FER robustness and accuracy. Specifically, we investigate the application of Hematoxylin and Eosin (H&E) staining-inspired transformations for visible and infrared (IR) modalities. These transformations improve contrast and texture, enhancing feature representation. Additionally, we introduce a novel convolutional neural network (CNN)-based multimodal fusion framework that integrates both original and derived modalities, leveraging complementary information to improve performance. To refine the fusion process, we developed a Modality Attention Module that performs weighted modality fusion using attention mechanisms, prioritizing the most informative modalities. Comprehensive evaluations on the NVIE and VIRI datasets demonstrated state-of-the-art accuracies of 88.22% and 86.66%, respectively. These results significantly outperform individual modalities and previous methods, validating the scalability and effectiveness of our approach in real-world FER applications. This approach is particularly effective in capturing complex emotion-related features across varied conditions.