Integrating LLMs With Computer Vision: A Geometrically Invariant System for First Notice of Loss in Car Insurance

ABSTRACT Automating multi‐stage decision workflows in regulated industrial domains demands intelligent systems capable of orchestrating heterogeneous evidence sources, such as uncontrolled photographs, free‐text narratives and structured forms, under strict operational and compliance constraints. We present FADES, a production‐grade expert system that integrates conversational AI with computer vision to automate end‐to‐end claim evidence interpretation and vehicle damage assessment. The system addresses three core challenges: structured data acquisition via state‐machine‐controlled conversational agents powered by tool‐enabled Large Language Models (LLMs); robust damage localization through modern object detection; and deterministic severity estimation via panel‐aware geometric reasoning and rule‐based logic. Our methodology combines backend‐enforced slot‐filling for First Notice of Loss (FNOL) data collection, ensuring compliance with the Italian CAI standard, with a multi‐stage damage analysis pipeline that reconciles detection outputs with vehicle panel segmentation. We systematically evaluated diffusion‐based (DiffusionDet) and transformer‐based (RF‐DETR, D‐FINE) detectors on 13,353 real accident images comprising 41,057 annotations across four damage classes. RF‐DETR Large achieves ( over DiffusionDet), () and , while satisfying real‐time processing requirements. Operational validation demonstrates that the integrated system reduces claim processing time from 30 to 12 min, achieving time‐to‐completion under 7 min at a sustained throughput of 45 cycles per day. Cross‐dataset experiments reveal substantial domain gaps ( on CarDD), confirming the necessity of insurance‐specific training data. Micro‐ablation analysis shows full pipeline integration outperforms partial implementations by in end‐to‐end efficiency. These results provide empirical evidence that hybrid deterministic‐generative architectures deliver the reliability and auditability required for regulated industrial applications.