TriNet-HAR: A trimodal framework for human activity recognition

Human Activity Recognition (HAR) is vital in healthcare, security, and human-computer interaction. Traditional unimodal or bimodal approaches often fail to capture the full complexity of human actions. To overcome this, TriNet-HAR, a novel trimodal deep learning framework is proposed that integrates depth maps, 3D skeletal joints, and inertial signals via a unified, explainable architecture. Each modality is processed using a dedicated module: Convolutional Neural Network (CNN) for depth, joint-wise attention layers mimicking Graph-Convolution-Network (GCN) for skeletons, and positional encoding-enhanced Long Short-Term Memory (LSTM) for Inertial Measurement Unit (IMU) data. A multi-head self-attention and cross-attention fusion strategy refines inter-modality dependencies. SHapley Additive exPlanations (SHAP) and Gradient-weighted Class Activation Mapping (Grad-CAM) are employed for interpretability. Evaluated on the University of Texas at Dallas Multimodal Human Action Dataset (UTD-MHAD) dataset, TriNet-HAR achieves 98% accuracy, outperforming unimodal and bimodal baselines in precision, recall, and F1-score, demonstrating superior robustness, adaptability, and transparency.