PipeCLIP: Defect-Conditioned and Cross-Focus-Driven Vision-Language Model for Video-Based Sewer Defect Inspection

In recent years, vision-language models (VLMs), such as CLIP, have excelled in the visual domain due to the availability of vast paired image-text data. However, directly adapting CLIP to the sewer defect inspection achieves a poor performance, since the vocabularies of defect category are highly “unfamiliar” to CLIP, resulting in the weak representations among the defect categories in the text embedding space. Besides, the substantial differences of pipe characteristics are challenging to align the multiple pairs of visual and text features across multi-focus segments. We propose PipeCLIP for adapting the text information to the video-based multi-label sewer defect classification, which is the first method to integrate a CLIP-based model into the sewer defect inspection. First, expert prior descriptions (EPD) are introduced to differentiate the distinctions between defect category in the text embedding space. Second, defect-attribute coupling (DAC) prompt is proposed to strengthen the coupling relationship between the sewer defects and pipe multi-attributes. The two prompts proposed above together with the traditional category prompt, collectively constitute defect-conditioned text prompt (DecTP) for CLIP. Then, cross-focus temporal (CFT) module is designed to integrate feature information from different focal length, strengthening the visual-text alignment across multi-focus segments. Extensive experiments are conducted on the public benchmark, in which the superiority of PipeCLIP is demonstrated compared with the state-of-the-art methods. Code is available at: https://anonymous.4open.science/r/PipeCLIP-0925.