NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control

Grasping accuracy is a critical prerequisite for precise object manipulation that may require careful alignment between the robot hand and an object. Neural descriptor fields (NDFs) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDFs alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Alternatively, tactile sensing enables more precise contact, but existing approaches limit policies to simple predefined contact geometries. In this work, we introduce NeuralTouch, a multimodal framework that integrates NDFs and tactile sensing to enable accurate generalizable grasping through gentle physical interaction. Our approach leverages NDFs to implicitly represent the target contact geometry, from which a deep reinforcement learning policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and with zero-shot transfer to real-world manipulation tasks, such as peg-out/in-hole and bottle lid opening. Our results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic grasp-based manipulation.