REBA-VLM: Enhanced risk mitigation in construction via Pose guided visual instruction tuning

Ergonomic risk assessment helps improve safety and reduce injuries in construction work.The Rapid Entire Body Assessment (REBA) method is widely used, but it is usually done manually.This makes it slow, subjective, and hard to apply, especially on large construction sites.Though vision-based methods do exist, many tend to yield inaccurate assessments, especially incorrect REBA scores, and do not explain why the posture causes the risk.This work proposes an automated, humanunderstandable REBA assessment method based on a vision-language model.The system takes a single image of workers and produces REBA score, problem identification, and actionable suggestions in simple language.We fine-tuned an existing visionlanguage model (VLM), namely Qwen2-VL-2B, using Low-Rank Adaptation (LoRA) to efficiently utilize limited computational resources.During dataset preparation, we chose REBAPose (a visionbased method built on new annotations of three keypoints to get more accurate REBA score), and applied on Moving Objects in Construction Sites (MOCS) Dataset to obtain REBA scores and joint angles.Gemini 2.5 Flash generated clean, consistent textual labels and explanations from posture-related cues to prepare the dataset.Then, we created a finetuned Qwen VLM model using that dataset.We tested the method on unseen images and compared it with common baselines.We reported score agreements and whether explanation points to the same body parts that mainly drive the REBA score.As dataset is trained on MOCS, our method is robust to construction scenes with cluttered backgrounds and different camera views, making it useful in realworld conditions and helping site engineers.