Comparing Fully Supervised and Zero-Shot Models for Semantic Segmentation of Building Materials

Automatic building material identification using computer vision has gained attention for applications in construction, facilities, and waste management. Existing studies focused on using image classification techniques for material identification, which is less robust in complex environments and loses geometry information. To provide finer granularity for material identification, this study focuses on using semantic segmentation for automated material identification, which provides pixel-level labelling. We collect and annotate a dataset of 1,008 images, training a Swin Transformer and DDRNet, which achieved mean intersection over union (mIoU) scores of 65.37% and 72.59%, respectively. To benchmark against fully supervised methods, we also introduce a zero-shot open-vocabulary segmentation pipeline, which segments materials without task-specific training. This method attained a mIoU of 15.95% across 12 classes, highlighting the persistent domain gap in generalized models. Our work compares two semantic segmentation strategies and supports future applications in automated scan-to-BIM, material flow analysis, and related tasks.