Can We Infer Object Pose Changes from Hand Movements?

Some Augmented Reality applications require tracking physical objects to anchor virtual elements to them. Despite significant progress in computer vision, achieving robust tracking of manipulated objects remains challenging, notably due to occlusions caused by the hands. Yet the hands carry valuable information: the way an object is grasped and moved is reflected in their shape and motion. We explore the following question: can we infer changes to an object’s position and orientation from the shape and movements of the hand manipulating it? We investigate this general approach, which we call Object-from-Hand (ObHa), building three probes, drawing on insights from experimental psychology research on grasping, on our own empirical studies, and on an analysis of the extensive HOT3D dataset of everyday object manipulations. We then discuss the approach’s potential either as a complement to vision-based pose tracking solutions or as a coarse standalone pose tracking solution.