TacSLAM: Tactile-Visual Simultaneous Localization and Mapping for Robotic Manipulation in Heavily Cluttered Environments

Robotic manipulation in heavily cluttered environments—warehouse shelving, disaster rubble, dense surgical fields—demands accurate 6-DoF object and end-effector localization, yet visual simultaneous localization and mapping (SLAM) degrades precisely when clutter causes severe occlusion, contact-induced motion blur, and feature sparsity. We introduce TacSLAM, a tightly-coupled tactile-visual SLAM framework that augments camera-based localization with high-resolution tactile sensing from a GelSight-style fingertip sensor mounted on the manipulator. TacSLAM contributes three novel modules: (i) a Tactile Feature Front-End (TFFE) that extracts geometric micro-features (ridges, edges, curvature) from tactile imprints and registers them as landmark observations in the SLAM factor graph; (ii) a Contact-Aware Loop Closure (CALC) mechanism that recognizes previously touched surfaces via a learned tactile descriptor, enabling loop closures even when the camera is fully occluded; and (iii) a Grasp-Conditioned Local Mapping (GCLM) module that builds object-centric submaps triggered by grasp events, fusing tactile geometry with visual depth for high-fidelity contact-surface reconstruction. Evaluated on the YCB-Video benchmark, a new UC-Clutter dataset of 32 manipulation sequences with synchronized tactile–visual–motion-capture ground truth, and real-robot trials on a Franka Emika Panda, TacSLAM reduces end-effector ATE by 47% over ORB-SLAM3 and 31% over SuperPoint-SLAM3 in heavily occluded sequences, while improving grasp success rate from 68% to 89% by providing sub-millimetre contact-surface geometry. The system runs at 20 Hz on a single workstation GPU.