The Effect of Ontology-Based Prompt Structuring on Image–Text Semantic Alignment for Construction Safety Hazard Recognition

This study examines the effect of ontology-based structured text prompts on image–text semantic alignment in multimodal models, using construction site scenes as a case study. Unlike prior research that has focused on object detection followed by risk judgment, this study targets the situation recognition stage preceding risk assessment, investigating how textual structure influences image–text alignment. A representative high-altitude construction work scene was selected, and key semantic elements were structured using an ontology. Ontology-based prompts and unstructured natural language prompts generated by large multimodal language models were applied under identical image conditions, and their semantic alignment scores produced by the CLIP model were compared. The results show that ontology-based prompts generally achieve higher and more stable semantic alignment scores than unstructured prompts, particularly in upper-ranking ranges. These findings indicate that structuring text inputs around core semantic elements can support more consistent scene interpretation in multimodal models. This study suggests that ontologies can serve not only as post hoc reasoning tools but also as input-level frameworks for structuring situation perception, providing methodological support for extending construction safety research beyond object-centric approaches.