ContactFusion: Cross-Modal Attention for Occlusion-Robust Hand–Object Contact Detection

Detecting hand–object contact is critical for applications in robotic manipulation, augmented reality, and ergonomic analysis, yet remains challenging due to self-occlusion, subtle contact cues, and ambiguous depth boundaries. Existing vision-based approaches often degrade under occlusion, while multi-sensor solutions require specialized hardware.We propose ContactFusion, a cross-modal attention framework that predicts dense, per-vertex hand–object contact maps by jointly reasoning over RGB-D appearance and estimated hand mesh geometry. Our method introduces three key components: (1) a Hierarchical Mesh Encoder that captures multi-scale geometric context, (2) a Cross-Modal Gated Attention module that adaptively fuses visual and geometric cues on a per-vertex basis, and (3) an Occlusion-Aware Contact Head that explicitly models vertex visibility to maintain reliable predictions under heavy occlusion.We evaluate ContactFusion on ContactPose, DexYCB, and OccContact, a new dataset with controlled occlusion levels that we introduce. Across all benchmarks, ContactFusion consistently outperforms prior methods, achieving state-of-the-art contact prediction accuracy and demonstrating significantly improved robustness as occlusion increases. Our results show that effective cross-modal fusion within the visual domain can substantially narrow the gap between passive vision-based contact estimation and instrumented sensing approaches.