A Device Control System Using User-Defined Full-Body Gestures with HoloLens2

This paper presents a novel system that allows users to operate devices through full-body gestures they define themselves. Gesture-based control systems are seeing widespread application for controlling a diverse range of devices. However, most existing systems rely on a predefined set of gestures. This fundamental limitation restricts not only the number of possible operations but also the overall system flexibility. The proposed system overcomes the limitation. The proposed system comprises two primary phases. In the initial definition phase, a user wearing a HoloLens2 defines a new gesture by directly manipulating the posture of a virtual avatar. This approach facilitates a highly visual and intuitive method for gesture creation. Throughout this definition process, an external camera system captures the user’s movements and computes the skeletal joint data. Then, in the subsequent operation phase, the user can control devices by performing the defined gesture without needing to wear the HoloLens2. For recognition, the system relies solely on the external camera system. This results in a highly adaptable and extensible framework for gesture-based interaction.