AI-driven analysis of animal cleanliness: A data-fusion model using RGB and thermal imaging

• RGB-Thermal imaging was used to evaluate dairy cow cleanliness automatically. • A two-stage machine learning pipeline (segmentation + score prediction) was developed. • Thermal input boosted segmentation accuracy over RGB alone (mIoU + 48 %) • CNN outperformed MLR (on dirt pixel counts) for predicting human cleanliness scores. • The approach supports interpretable, objective monitoring in livestock barn environment. Accurate assessment of dairy cow cleanliness is essential for ensuring animal welfare, maintaining udder health, and optimising milk production. Traditional visual inspections are subjective and often fail to distinguish dirt from natural coat patterns, especially in spotted breeds. This research investigates the applicability of a two-stage approach for automated cleanliness evaluation, consisting of (i) semantic segmentation of dirt areas on cow coats and (ii) regression from the resulting masks to numerical cleanliness scores. The first stage was implemented using the U-Net and DeepLabV3 architectures, which were trained on either RGB-only or RGB-Thermal (RGB-T) images. Incorporating thermal information significantly improved segmentation accuracy: U-Net achieved a mean Intersection over Union (mIoU) of 0.5244 on RGB-T images, compared to 0.3537 on RGB images, while DeepLabV3 on RGB-T images reached an mIoU of 0.5049. The second stage compared two regression strategies: multiple linear regression (MLR) on the number of pixels classified as dirt, and convolutional neural networks (CNNs) trained directly on the masks. CNN-based regression consistently outperformed MLR, with the best performance obtained by combining RGB-T segmentation and CNN regression (DeepLabV3 + CNN: MAPE 23.05 %; U-Net + CNN: MAPE 25.24 %). These findings support the feasibility of a two-stage RGB-T-based approach for objective cleanliness evaluation, highlighting the benefits of thermal information for segmentation and CNNs for score prediction.