Vision–Tactile Sensor Fusion System for Fabric Sorting and Robotic Grasping in Textile Recycling

Sorting discarded fabrics is a critical yet challenging task in textile recycling due to the diversity of material types and surface textures. We present a vision–tactile robotic system leveraging multi-modal sensing to enable accurate fabric recognition and adaptive grasping. The system employs a stereo RGB camera with MobileNet-SSD on the Myriad X chip for coarse object detection and 3D localization, achieving a mean average precision (mAP50) of 93.50% at 23 FPS. For fine-grained texture classification, tactile images are processed by a lightweight MobileNetv3-Textile model on NVIDIA Jetson Orin, achieving 27.3 FPS with 8.5 ms inference latency. Two complementary datasets were constructed: a visual dataset with 20 fabric categories for appearance-based classification, and a tactile dataset with 191 categories capturing weaving patterns for precise texture discrimination. Sensor fusion is performed in real time, integrating visual and tactile modalities to enhance recognition accuracy and grasp reliability. A resource-constrained control unit manages tactile processing, gripper force modulation via optical flow, and sensor coordination. Experimental evaluation demonstrates that the proposed multi-modal sensing approach significantly improves perception robustness and operational efficiency, providing a scalable solution for automated fabric handling in recycling.We release the dataset in https://github.com/AumnceLi/Visual-tactile-fabric-dataset.git.