ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation

Dexterous manipulation is a cornerstone capability for robotic systems aiming to interact with the physical world in a human-like manner.Although vision-based methods have advanced rapidly, tactile sensing remains crucial for fine-grained control-particularly in unstructured or visually occluded settings.We present ViTacFormer, a representationlearning approach that couples a cross-attention encoder to fuse high-resolution vision and touch with an autoregressive tactileprediction head that anticipates future contact signals.Building on this architecture, we devise an easy-to-challenging curriculum that steadily refines the visual-tactile latent space, boosting both accuracy and robustness.The learned cross-modal representation drives imitation learning for multi-fingered hands, enabling precise and adaptive manipulation.Across a suite of challenging real-world benchmarks, our method achieves approximately 50% higher success rates than prior state-of-the-art systems.To our knowledge, it is also the first to autonomously complete longhorizon dexterous manipulation tasks that demand highly precise control with an anthropomorphic hand-successfully executing up to 11 sequential stages and sustaining continuous operation for 2.5 minutes.