DefectTwin: When LLM Meets Digital Twin for Railway Defect Inspection

A Digital Twin (DT) replicates objects, processes, or systems in real-time within cyberspace for various applications, such as railway defect inspection. Such an application can benefit from DTs when integrated with Artificial Intelligence (AI) by enabling advanced data analysis, predictive maintenance, and other capabilities. Defect inspection can integrate pre-trained Large Language Models (LLMs) into DT ecosystem as its AI component to simulate defects, thereby reducing the reliance on extensive real-world defect data. Mobile-friendly tools that leverage this AI integration can help railway operators identify and understand defects directly from video streams, images, or text. Motivated by this scope, we propose the DefectTwin ecosystem and its mobile application for railway defect inspection. DefectTwin employs a multimodal AI pipeline to analyze both seen and unseen visual defects. The multimodal processing integrates diverse input types to generate user-friendly responses, while an instant feedback mechanism (instaUF) reinforces this with real-time Quality-of-Experience (QoE) insights. We evaluated the proposed ecosystem on a mobile device and obtained better performance compared to traditional image generation tools with precision between 0.76 and 0.93 for text, image, and video inputs.