Multimodal Zero-Shot Industrial DefectSegmentation via Visual–Textual–Geometry Fusion

Industrial surface defect segmentation plays a vital role in quality assurance and production safety. However, industrial inspection systems often confront critical challenges, including scarce defect samples, multimodal data gaps, and poor generalization capabilities. To overcome these limitations, this paper proposes a multimodal zero-shot defect semantic segmentation framework. It leverages textual guidance as a semantic bridge to align and fuse RGB visual features with geometric features from normal maps, enabling accurate segmentation of previously unseen defect categories. Specifically, local surface normals are first estimated from the input point cloud using Principal Component Analysis (PCA), which are then projected and encoded into structured 2D normal maps. A text-guided cross-modal fusion module based on cross-attention employs a hierarchical, multi-path architecture to achieve deep semantic alignment among textual, visual, and geometric modalities, thereby significantly enhancing the model’s generalization capability in zero-shot scenarios. Furthermore, a prototype aggregation strategy is designed to reinforce texture representation in the multimodal feature space while suppressing background noise. Experimental results demonstrate that our method outperforms the latest zeroshot segmentation approaches on two industrial datasets. It also achieves performance comparable to state-of-the-art few-shot segmentation techniques, providing a robust and scalable solution for multimodal zero-shot industrial surface defect segmentation.