Human Gesture Classification Through Keypoint-Integrated Deep Learning on Millimeter-Wave Radar

The capability to sense and classify targets through radar instead of optical sensors offers the advantage of non-line-of-sight operation, low battery consumption, and privacy, placing integrated sensing and communications at the core of 6G networks. Thanks to their fine spatial resolution, commercial millimeter-wave radios have already been repurposed for target classification – in particular for human gesturing, the topic of this paper – through deep learning trained directly on radar data as input. While promising, to achieve robust classification across a wide variety of humans and scenes, thousands to tens of thousands of training exemplars are needed. Such databases are unavailable to date, and developing application-specific databases may require prohibitive durations, from months to years. To alleviate the burden of training, we integratekeypoints– a set of 3D points tagged to the salient body parts, generated from a camera / lidar system as an add-on to our in-house 28 GHz radar system – as an intermediate step of the training process, between the radar input and the classification output. The keypoints exploit rich databases of millions of exemplars of different humans and scenes, that are publicly available. By integrating keypoints, we can achieve 90% classification accuracy with fewer than 100 exemplars. In addition, we can operate at distances up to 2.5 m versus the near-field operation (within 50 cm) of most other systems and we are the first to consider 16 gesture classes across both forearms rather than just a few classes across a single forearm.