Causal-Inspired Fourier Representation Learning for Wearable IMUs and Egocentric Action Recognition

Inertial Measurement Units (IMUs) can capture intricate kinematic behaviors, thereby enhancing the performance of human action recognition methods. Consequently, this technology has recently garnered considerable attention within this domain. However, existing methods either encounter limitations in instance-level visual representation due to self-occlusion or fail to fully utilize the potential of complex kinematic information, making it challenging to adequately capture the intricate relationships between the two data sources. In this paper, we tackle this issue by addressing the problem through causal learning and Fourier learning. Specifically, we introduce a novel framework calledCausal-Inspired Fourier Representation Learning (CIFRL)for Wearable IMUs and Egocentric Action Recognition, which aims to enhance cross-modal feature alignment. The framework consists of two key components: (1) Temporal Causal Modeling (TCM), designed for video interpretation from a causal perspective; (2) Spectral-Temporal Learning (STL), which aims to decompose the inertial data using Fourier representation and align cross-modal features. We evaluate our proposed framework on the WEAR and CMU-MMAC benchmarks. Empirical results demonstrate the superior performance of our CIFRL approach compared to state-of-the-art methods. Our code is available at https://github.com/Adrianos1219/CIFRL.