DMS-Net: A dynamic multi-scale multi-modal fusion approach for sheep behavior recognition

• visual-skeleton dual-modal collaborative architecture for accurate sheep behavior recognition in complex scenarios. • Dynamic efficient computation strategy cuts redundant calculations while maintaining recognition performance. • Cross-modal fusion boosts robustness of fine-grained behavior recognition under occlusion conditions. • Multimodal complementary mechanism supports precision livestock farming and animal welfare. Sheep behavior is the critical feature for health and welfare assessment. Yet, sheep behavior recognition in complex barn environments encounters formidable challenges, primarily stemming from variable lighting and sheep overlapping interference. To address this issue, this paper proposes an intelligent monitoring framework for restricted sheep populations and constructs a dynamic multi-scale and multi-modal network (DMS-Net) dedicated to the accurate recognition of four key sheep behaviors: standing, lying, kneeling and sitting, which seeks to balance model lightweight and multi-scale feature representation capabilities in complex scenarios. The core approach of this study is a vision-skeleton dual-modal fusion structure. For the visual modality, lightweight MobileNetV3 serves as the backbone network, with dynamic structure convolution (DSC) replacing fixed kernel convolution to adaptively adjust local response modes; meanwhile, Effective Channel Attention (ECA) and Multi-scale Pool Spatial Pyramid (MPSPP) are integrated for channel calibration and cross-scale receptive field expansion, respectively. For the skeleton modality, the Multi-scale Graph Convolutional Network (MS-GCN) is introduced, which extracts multi-granularity joint features via parallel multi-branch modules and generates a dual-scale adjacency matrix to capture both local and global joint dependencies. Experimental results indicate that DMS-Net achieves an accuracy of 98.89% in complex scenarios, with overall performance improved by 6.21% compared to traditional single-modal methods without skeleton fusion, and effectively mitigates the high behavioral misclassification rate in real agricultural environments.