Hierarchical Unsupervised Feature Extraction and Fusion of Multisensor for Human Activity Recognition

Deep learning methods are widely used for Human Activity Recognition (HAR) with Inertial Measurement Unit (IMU) sensors placed on various parts of the human body. In such systems, analyzing data from multiple IMU sensors presents a significant challenge. Fusing outputs from individual sensors, as well as integrating information across sensors, is a complex task that often poses challenges for simple one-stage models, particularly in explicitly capturing distinct intra-sensor and inter-sensor. In this work, we propose a comprehensive Hierarchical Unsupervised Fusion (HUF) method for feature extraction and fusion from IMU sensor data, using a hybrid architecture of Convolutional Neural Networks (CNNs) and Autoencoders (AEs). First, a stacked CNN-AE encodes short-time signals into high-dimensional feature representations. Second, a dedicated CNN-AE performs intra-sensory fusion by combining features within each sensor. Finally, a third CNN-AE—preceded by a custom-designed Attention Mechanism (AM)—fuses feature across all sensors for inter-sensory fusion. The hierarchical structure enables the attention mechanism to calibrate sensor contributions based on activity type, thereby enhancing the fusion process. The proposed HUF method achieves 96.8% accuracy on the UCI-HAR dataset—2.8% higher than state-of-the-art supervised methods. It also achieves 92.9% and 87.6% accuracy on the DaLiAc and Parkinson’s disease gait (MDFG-PD) datasets, respectively. These results further improve to 94.8% and 89.0% when the AM is integrated into the HUF framework.