A self-supervised learning approach to acquire representation of concave object manipulation with sparse tactile sensing

Tactile sensing is essential for improving robotic manipulation, particularly when handling transparent or deformable objects. However, effectively leveraging tactile observations remains a key challenge. In this study, we propose a semi-self-supervised complementary learning framework that integrates visual input with sparse tactile data collected through probing actions. Unlike previous approaches that rely on high-resolution tactile sensors or detailed 3D reconstructions, our method employs sparse tactile sensing to construct object representations via unsupervised learning. The proposed framework enables both complementary and independent recognition through vision and tactile perception, allowing the robot to perform additional probing actions to verify whether its hand has actually reached inside an object. We trained and evaluated our method on opening exploration tasks involving semi-transparent and deformable objects, using a relatively small real-world dataset collected with a robotic hand equipped with a simple tactile sensor.