ViTaDex: Vision–Tactile Fusion for 6-D Object-in-Hand Pose Estimation in Dexterous Anthropomorphic Manipulation

Object-in-hand 6-D pose estimation is critical for dexterous manipulation and assembly tasks. Existing object-in-hand pose estimation methods neglect the varying significance of tactile information from distinct hand regions during feature extraction and employ simplistic concatenation for vision–tactile fusion, which fails to capture the complementary relationships between these modalities. Moreover, prevailing object-in-hand datasets predominantly rely on predefined motion trajectories and single-point fingertip contact information, which inadequately represent real-world scenarios. To address these limitations, we propose the vision–tactile adaptive fusion network. Specifically, we design a novel tactile-gated adaptive graph convolution to dynamically model multiregional tactile features across the hand. We also introduce a vision–tactile cross-attention gated fusion module that facilitates efficient integration of multimodal sensory information. In addition, we construct the Vision–Tactile Object-in-Hand Dexterous Manipulation dataset based on human demonstration for 6-D pose estimation. The dataset employs a teleoperation system that combines an exoskeleton hand with a pose tracking system to replicate human dexterous behaviors and encompasses high-density tactile data spanning the entire hand. Extensive experiments demonstrate the robustness of our framework against occlusions.