MAD-Fusion: Modality-Aware Dynamic Fusion in Identification of Activities of Daily Living

Activities of daily living (ADL) identification with wearables has significant implications in healthy lifestyle management and offers an important sensor-based supervised learning research benchmark. Most ADL studies use single-modality (i.e., single sensor type like smartwatch only or earbuds only) data while multi-sensor data fusion studies using early-stage fusion of modalities from multi-sensors and multi-devices are emerging. To improve classification performance and model interpretability, leveraging early-stage fusion, late-stage fusion and individual modalities, we introduced novel modality-aware dynamic fusion (MAD-Fusion) models for multi-sensor data-fusion-based ADL identification. Based on early-stage fusion, we incorporated conformal prediction for uncertainty quantification, uncertainty late-stage fusion for cross-modality interpretability, and multi-modal strength-aware classification module. Trained on 36 independent subjects and tested on 4 independent subjects from Samsung ADL dataset with multi-sensor (accelerometers and gyroscopes) and multi-device (earbuds and smartwatch), MAD-Fusion not only achieved the state-of-the-art classification performance (accuracy: 0.9504, F1-score: 0.9142), but also enabled better interpretability of contributions and uncertainties from different modalities. The additional contribution of each building block is validated systematically. Furthermore, we validated MAD-Fusion’s superior performance on two public datasets in multi-sensor single-device settings (UCI-HAR and USC-HAD datasets). On all three datasets, MAD-Fusion manifested statistically significantly superiority comparing against baselines of single-modality, early-stage fusion and late-stage fusion (p< 0.001). To conclude, the novel MAD-Fusion models improve the classification performance, uncertainty quantification and interpretability for the ADL identification and can be applied to broader supervised learning areas requiring high performance, rigorous uncertainty quantification and model interpretability.