Arm Motion Classification Based on Radar–Camera Cross-Learning with a Generative Model

In this paper, we classify human arm movements using point clouds obtained from a millimeter-wave multiple-input multiple-output (MIMO) radar integrated with radar–camera cross-learning. When a radar receives signals reflected by the entire human body, delineating the specific details of arm movements is challenging. Our approach involves skeletonization of human point clouds measured by a MIMO radar and then processing them using diverse deep learning models. To enhance the skeletonization of the point cloud model, cross-learning between radar and camera systems is implemented. The point clouds obtained from the radar are trained on camera data, since they offer a higher resolution. Two training methods are investigated in this study. The first method utilizes a two-dimensional convolutional neural network (2D-CNN) regression model to extract the arm angles of the skeleton model for determining the class of arm motion. The second method employs an autoencoder (AE) along with data augmentation, performed using a stable diffusion model, to enhance the robustness of feature extraction. The feasibility of the proposed feature extraction methods is validated through experimentation based on six distinct arm motions, resulting in arm motion classification accuracies of 95.23% for the 2D-CNN method and 98.1% for the AE-based method. These outcomes underscore the efficacy of the proposed techniques, which show significant promise for application in detailed human motion classification using radar.