Integrating Large Language Models (LLMs) with Autonomous Aerial Drone Robotics and Computer Vision for Contextual Adaptive Construction Site Safety Management and Risk Assessment

Integrating large language models (LLMs) and robotics offers transformative potential for enhancing construction site safety monitoring, real-time risk assessment, and situational response. This research proposes an intelligent drone-based system that leverages real-time object detection and contextual image analysis with advanced LLM reasoning capabilities for construction site supervision. A fine-tuned YOLOv11n model (using transfer learning) was developed for detecting 10 different construction site safety-related classes. In critical safety violations (e.g., “No Hardhat”), the corresponding image frame is sent to CLIP (Contrastive Language-Image Pretraining) for generating image-based descriptions. These data are processed by a fine-tuned LLM to generate construction-specific textual prompts, which are converted to audio and broadcast via a drone-mounted speaker. The system operates autonomously using a D* planning algorithm. Detection, response generation, and navigation capabilities were evaluated in a simulated environment using Webots, and the pipeline from object segmentation to audio generation was ported to a real-world drone.