MultimodalMamba: An Efficient DHCNN–FAM–Mamba Architecture for Human Activity Recognition

Human activity recognition (HAR) research has largely focused on improving accuracy, with less attention to computational efficiency and hardware compatibility. Recently, Mamba has emerged as a hardware-aware sequence modeling algorithm that enables efficient training and inference across various domains, including vision and language. Inspired by this, we present MultimodalMamba, a hybrid neural architecture that integrates Dilated Hierarchical CNNs (DHCNN), Frequency Attention Module (FAM), and Mamba. The DHCNN-FAM-Mamba block serves as the core architectural block, unifying hierarchical CNN-based local encoding, frequency-domain attention, and lightweight Mamba-based global sequence modeling. We validate MultimodalMamba on three public multimodal benchmark datasets, HWU-USP, Berkeley-MHAD and UTD-MHAD, comprising diverse sensing modalities. Our model surpasses state-of-the-art vision transformers, Vision Mamba, and CNN-based approaches while requiring fewer FLOPs and parameters. MultimodalMamba achieves 99.1% accuracy and 98.1% F1-score (F1) on HWU-USP, 98.4% accuracy and 97.5% F1 on Berkeley-MHAD, and 97.7% accuracy and 96.3% F1 on UTD-MHAD, establishing new benchmarks across all datasets. Multi-run and Leave- One-Subject-Out (LOSO) evaluations confirm the model’s stability and cross-subject robustness. To our knowledge, this is the first work to explore a DHCNN-FAM-Mamba hybrid for efficient multimodal HAR. Finally, we deployed the proposed model on multiple edge computing platforms, including the Raspberry Pi and NVIDIA Jetson Nano.