LVLMs as inspectors: An agentic framework for category-level structural defect annotation

: Automated structural defect annotation is a critical yet challenging prerequisite for infrastructure inspection, particularly under conditions of limited labeled data, severe class imbalance, and heterogeneous visual appearances. Existing learning-based or semi-automatic annotation methods often rely on manually labeled samples, human-in-the-loop correction, or task-specific model fine-tuning, which limits their scalability and applicability in practical inspection scenarios. To address this problem, we propose Agent-based Defect Pattern Tagger (ADPT), a training-free, zero-shot reasoning-based annotation workflow that organizes agentic large vision-language model (LVLM) inference via prompt engineering into a multi-stage reasoning and verification process. ADPT integrates semantic pattern matching with an iterative self-questioning refinement mechanism to structure and consolidate LVLM outputs into standardized defect labels, without assuming human-provided labels or in-loop supervision during annotation generation. Extensive experiments on both class-balanced and class-imbalanced datasets demonstrate that ADPT achieves up to 98% accuracy in binary defect discrimination and up to 90% mean F1-score across four defect categories under balanced settings, with 89% mean F1-score under class-imbalanced conditions. These results indicate that ADPT can substantially reduce manual annotation effort while providing structured, reusable annotations that support downstream tasks such as dataset bootstrapping, transfer learning, and domain adaptation in structural damage assessment.