Complementary visual localization and tactile mapping approach for robotic perception of millimeter-sized objects with irregular surfaces

Humanoid robots and human-machine interaction technologies are essential for perceiving and manipulating millimeter-scale objects with irregular surfaces in extreme environments, such as outer space, radioactive zones, and hazardous sites with explosive ordnance, where human access is restricted. A vision-based perception approach provides spatial and positional information about objects but relying solely on it for robot manipulation poses challenges due to limitations in detectable object size, as well as sensitivity to external factors such as focusing issues, occlusion, and lighting conditions. In contrast, tactile perception offers valuable information about aspects that are difficult to discern visually, including an object's shape, surface characteristics, and the forces involved during contact. This study presents a complementary visual localization and tactile mapping framework that allows robots to effectively perceive small objects with irregular surfaces in visually restricted environments. The proposed method draws inspiration from the sequential vision-tactile sensory processing observed in humans when handling small objects with irregular surfaces. It employs an RGB-Depth camera for visual perception and a soft pressure sensor array, made using inkjet printing, for tactile perception. We demonstrate the feasibility of implementing a sensory substitution to detect the size and location of objects through visual perception, as well as identify object surfaces and reconstruct their three-dimensional profiles using tactile scanning, particularly in environments where visual information is limited. This study provides a technological foundation for enhancing the autonomy and adaptability of humanoid robots in unpredictable and unstructured environments, particularly to support precise robot manipulation in such conditions.