Toward smart health monitoring: multimodal sensing and intelligent disease diagnosis in poultry and livestock

Multimodal sensing provides valuable data and knowledge for intelligent disease diagnosis. Intelligent disease diagnosis is evolving from text-based understanding toward multimodal fusion and knowledge-driven reasoning. Future efforts should prioritize devices for definite disease diagnosis and diagnostic frameworks integrating prescription generation for intelligent decision support. Health management of livestock and poultry production is critical for ensuring food safety, animal welfare, and production sustainability. Farm scales expanded rapidly in China over the last two decades. For example, the capacity of a single laying hen house increased from thousands to hundreds of thousands. Large-scale farm typically refers to an annual slaughtered volume of >100,000 birds or >50,000 pigs. As such, the prevention and control of animal diseases have become increasingly complex. For instance, African Swine Fever (ASF), which can have a mortality rate of up to 100%, exemplifies the devastating impact of animal epidemics. The largest ASF outbreak in China in 2018 led to the death of 43.46 million pigs, causing an estimated US$14.5 billion in indirect economic losses (You et al., 2021). However, many poultry and livestock farms often suffer from limited diagnostic infrastructures and professional expertise. Veterinarians are usually responsible for multiple farms, which may increase the risk of cross-infection and substantial economic loss. In addition, the shortage of licensed veterinarians further increases the difficulty of timely health management (Yang et al., 2025). Recent advances in artificial intelligence (AI) and smart sensing offer promising solutions and enable automated health monitoring and intelligent diagnostic systems, which marks a crucial step toward the digital transformation of livestock health management. The evolution of intelligent disease diagnosis has progressed through three milestones, ie, the Database Matching System (DMS), the Expert System (ES), and the Large Language Model (LLM). The DMS, originating in the 1960s, marked the beginning of intelligent diagnosis. It functions by calculating similarity scores through keyword matching and weighted symptom scoring to suggest suspected diseases. While effective for common, well-documented diseases, the DMS accuracy declines in cases involving atypical symptoms, co-infections, or emerging diseases (Houe et al., 2011). Today, database-driven systems are incorporated into cloud-based diagnostic platforms and veterinary knowledge graphs, continuing to serve as useful tools for rapid, simple, and low-cost screening on small- and medium-scale farms. Building upon the DMS, the ES are developed to emulate specialists' reasoning processes through if-then rules. The MYCIN system, introduced by Stanford University in the 1970s for diagnosing human infectious diseases, exemplified this approach (Saeidnia and Nilashi, 2025). In animal management, for example, the ES has been used to diagnose diseases such as blue ear in pigs, Newcastle disease in poultry, respiratory disease in dairy cattle. While these systems provide interpretable reasoning chains, they lack self-learning capability and adaptability, making them less effective for complex or uncertain scenarios. By the 2020s, the field has transitioned from rule-based reasoning to data-driven learning. The advent of LLMs such as Generative Pre-trained Transformer (GPT) and Pathways Language Model (PaLM) leverages massive datasets and deep learning architectures to autonomously acquire knowledge and detect subtle correlations across modalities (Jin et al., 2025). Although LLMs exhibited broad coverage and strong cross-domain integration ability, their “black-box” nature limits interpretability and complicates the validation of knowledge sources. Effective intelligent diagnosis in poultry and livestock health relies fundamentally on the reliability of sensed data. Traditional approaches have often relied on single-modality data due to its simplicity, cost-effectiveness, and ease of deployment. While these approaches can provide meaningful insights under controlled conditions, and form the basis of early database matching and expert system-based diagnostics, they suffer from limitations in robustness and accuracy due to environmental interference, noise, or biological variability. For instance, infrared thermal imaging may misinterpret surface heat changes caused by airflow as fever, and RGB cameras may struggle to estimate body weight under poor lighting or occlusion (Liu et al., 2023). To overcome these constraints, recent research emphasizes multimodal sensing as a powerful approach by integrating heterogeneous data sources. The advantages of multimodal sensing over single-modality approaches are threefold. First, redundancy increases resilience, allowing one modality to compensate when another is compromised. Second, complementarity allows each sensor to contribute unique insights to the same health indicator. Third, robustness is achieved through integrating multiple data streams, enabling diagnostic models to generalize across varying farm environments and animal populations. Therefore, this review will focus on the applications of multimodal fusion in health information perception of farm animals, summarizing recent achievements in physiological condition assessment, behavior recognition and abnormal sound analysis. Afterwards, multimodal fusion-based technologies of diagnosis models will be introduced (Figure 1). Finally, the challenges and limitations of intelligent diagnosis of livestock and poultry, along with future perspectives, will be discussed to provide insights into the revolution of farm animal health management. Multimodal sensing and intelligent disease diagnosis in poultry and livestock. The deep-learning based disease diagnosis model was adapted from Yang (2025). LLM donated Large Language Model. Multimodal sensing enhances animal health monitoring by automatic physiological condition assessment, behavior recognition and abnormal sound analysis. As sensors become increasingly common in livestock and poultry farms, multiple data modalities can be collected simultaneously. Typically, information contained in different data modalities varies and is specific to particular scenarios. However, in farm environments, the application scenario may pose challenges due to many factors, such as changing illumination, background noise, and limited image resolutions, making it unreliable to rely on a single data modality for decision-making (Yin et al., 2023; Ma et al., 2025). Multi-modality fusion integrates multiple sources of data, such as environmental factors, images, and audio, providing a more comprehensive understanding of the environment in livestock and poultry farms compared with a single data modality and enabling more accurate results in challenging farming environments (Kalamkar, 2023; Tang et al., 2023). Therefore, it is necessary to employ multimodal fusion to enhance the animal health monitoring and disease diagnosis. According to the fusion level, the deep learning-based multimodal data fusion can be divided into data-level fusion, feature-level fusion, and decision-level fusion (Tang et al., 2023), as shown in Figure 2. Multimodal fusion framework, taking RGB-D fusion as an example (Fu et al., 2022). (a) Early-fusion. (b) Late-fusion. (c) Middle-fusion. CNN donated Convolutional Neural Network, and Concat donated concatenate‌. Among the sensors used in livestock and poultry farms, cameras are one of the most widely used devices significantly contributing to the non-contact perception and visible ­livestock management by computer vision (Li et al., 2022a). Generally, image modalities used in computer vision for livestock ­management include Red-Green-Blue (RGB), depth (Figure 3), and thermal (Figure 4) images. Given the low cost and high accessibility, RGB images are one of the most common image modalities, offering rich color and texture information with a high spatial resolution (Ma et al., 2023). As a result, RGB images are widely adopted in various applications in livestock and poultry farms. Nevertheless, the long-existing challenges inherent in complicated farming conditions, such as changing illumination and clutter backgrounds (Lamping et al., 2022; Li et al., 2022a), need to be addressed before the RGB image-based methods can achieve improved performance (Liu et al., 2022). Paired RGB (left) and Depth (right) images (He et al., 2023). A Microsoft Azure Kinect DK camera was adopted to simultaneously collect the paired RGB-D images. The camera was horizontal to the ground at a height of 2.3 m above the feeding passageway. Paired thermal (top) and RGB (bottom) images of six different regions on pig body surface (Xie et al., 2023). A: forehead, eyes, and nose. B: ear root, and back. C: anus. The thermal and visible light images could be obtained at the same time by using an infrared thermal imaging camera. Recent advancements in imaging technology and hardware have made depth images readily available through consumer-level RGB-D cameras, effectively complementing the RGB images. In a depth image, each pixel represents the distance from the camera to the corresponding object, which favors the 3D reconstruction of animals (Li et al., 2022b). Furthermore, depth images provide shape information, which helps extract animal objects from clutter backgrounds (Xu et al., 2022). In most cases, depth images are combined with RGB images since RGB images can remedy the defects of coarse resolution and details (Liu et al., 2022; He et al., 2023). Typically, RGB images have high spatial resolution along with rich color and texture information (Ma et al., 2023), whereas they are highly susceptible to illumination variations and lack structure information (Liu et al., 2022; Xu et al., 2022). In contrast, depth images are robust to illumination changes and provide substantial shape and 3D structure details (Xu et al., 2022; He et al., 2023), while they lack fine object details (Liu et al., 2022; He et al., 2023). The RGB-D fusion leverages the complementary strengths of both modalities, enabling a more comprehensive understanding of animal health and improving the estimation of physiological traits, such as pig body weight (He et al., 2023), body size (Li et al., 2022b), posture (Xu et al., 2023) and appearance (Lamping et al., 2022). Thermal images can also serve as a complementary modality, especially under low illumination and complex background conditions due to the robustness against lighting variations (Wu et al., 2023). Thermal images measure the animal surface temperature (Cai et al., 2023; Xie et al., 2023), which is a key indicator of animal health and welfare (Cai et al., 2023). Consequently, thermal images are widely used in livestock and poultry farms to detect animal temperature and diseases (Xie et al., 2023). Additionally, thermal images can also aid in extracting animals from clutter backgrounds, as animals typically have higher temperatures than objects in the backgrounds (Zhong and Yang, 2022). Although thermal images have shown great potential in livestock and poultry farms, the high cost of the thermal infrared camera significantly hinders their use. Besides, image alignment is also required for fusion between RGB and thermal images. Notably, in applications where thermal images are used to measure animal surface temperatures, the accuracy may be influenced by the ambient temperature and humidity, measuring distance and angle, and emissivity of measuring parts (Xie et al., 2023). In addition to camera, acoustic sensors offer a valuable means of linking animal vocalizations to health, welfare, and environmental impact (Ma et al., 2025). In livestock and poultry farms, acoustic technologies have been developed to detect a multitude of traits and behaviors, such as coughing (Shen et al., 2022; Ma et al., 2025), body weight (Fontana et al., 2017), diseases (Cuan et al., 2020), and feeding behavior (Liao et al., 2022). Given the complicated acoustic environment in livestock and poultry farms (Shen et al., 2022; Ma et al., 2025), the sensing models relying solely on acoustic features may not achieve satisfactory accuracy. Instead, fusing the acoustic features with other powerful features can improve the detection accuracy (Shen et al., 2022; Ma et al., 2025), as shown in Figure 5. Although promising results have been reported (Shen et al., 2022; Ma et al., 2025), it is worth noting that the features mentioned above are derived from the same data as the acoustic features. Therefore, the fusion of acoustic data and other data modalities is yet to be explored. Structure of the multimodal pig audio representation and fusion framework (Ma et al., 2025). A was a given audio clip, A1 and A2 were two randomly cropped segments, A2′ was the spectral representation of A2⁠. z1 and p1 were the output of A1⁠, z2 and p2 were the output of A2′⁠, Stop grad indicated the stop gradient operation and MLP donated multilayer perceptron. Traditional diagnostic methods relying on manual observation and laboratory testing often suffer from low efficiency, high costs, and limited accuracy. With the advancement in natural language processing, computer vision, audio signal analysis, and knowledge graph, researchers (Hoang et al., 2023; Mustapoevich et al., 2023; Wang et al., 2024; Yu et al., 2024; Li et al., 2025) have developed intelligent diagnostic models that integrate multimodal and knowledge-driven approaches to improve accuracy, interpretability, and responding speed. Text-based intelligent diagnosis models form the foundation of this field. Yu et al. (2024) developed a Bidirectional encoder representation from transformers-Bidirectional long short-term memory network-Conditional random field (BERT-BiLSTM-CRF) model that effectively handled sparse domain texts with an F1 Score (a widely used performance metric in machine learning) of 96.38%. Wang et al. (2024) integrated BERT semantic vectors with knowledge embeddings to analyze 11,401 laying-hen cases, significantly improving diagnostic accuracy in complex textual contexts. Similar knowledge-driven text learning methods have also shown robustness in scenes where the number of training samples was limited (Wang et al., 2023). Although these models offer strong semantic comprehension, they are still limited by such as data text and poor (Yang et al., 2025). To overcome the limitations of diagnosis multimodal diagnosis models that integrate and information provide a Li et al. an for image and data Yang et al. developed a multimodal disease model that integrates CNN and BERT through improving by The fusion for diagnosis further this approach by through semantic a of Although challenges in data multimodal and these that integrating image and text data enhances robustness and diagnostic The of audio such as and provides information for disease diagnosis. Mustapoevich et al. reasoning to diseases, the potential of data in Yang incorporated and into a model for and used a to image, and text features (Figure an of in respiratory disease While these models enable early they still data and to model structure of Yang (2025). donated vision models reasoning and domain knowledge into intelligent diagnosis. Wang et al. (2024) the encoder representation from disease knowledge model (Figure for laying-hen disease diagnosis. The model integrated knowledge into and enhances with a of and by et al. introduced the which incorporated embeddings for disease reasoning. frameworks between textual data and biological allowing disease reasoning and improved model structure of (Wang et al., research a from text-based understanding toward multimodal fusion and reasoning. challenges in across modalities, and Building multimodal datasets and reasoning into diagnostic systems will be crucial for interpretable and intelligent diagnosis for livestock and poultry diseases. Recent advances in LLMs such as et al., and et al., 2023) have led to in and multimodal As a result, livestock and poultry disease diagnosis is from rule-based systems to However, LLMs from to veterinary and livestock health limited data, complex symptom across multiple and the need for interpretable and by and (2024) marked the to language models such as and a veterinary The results that achieved a accuracy of on veterinary yet still veterinary a crucial that LLMs lack the domain and reliability required for professional veterinary diagnosis their strong and reasoning With the advancement of research has from improving model reasoning toward such as et al., this this et al. a framework for disease integrating and generation disease were in ie, and specific symptom before a through decision diagnostic veterinary and diagnostic accuracy above The marks a from to where multiple to that many livestock diseases through and textual research multimodal et al. an intelligent diagnostic framework for infectious diseases, integrating textual symptom with image The a for textual and an improved for the model such as and for decision was to textual datasets and improving model The multimodal fusion achieved an accuracy of both and The results that integration combined with data can enhance diagnostic robustness in veterinary research has toward and et al. developed the (Figure the language model for disease diagnosis. upon with integrated a veterinary knowledge over of was with knowledge using before by the enabling and The performance to models such as and and high when with diseases accuracy, introduced interpretable diagnostic the between generation and veterinary The framework of et al., 2025). The of the was a as the model output of was in The above a of evolution from LLM to The field has progressed from to from to multimodal and toward these advances will veterinary and livestock health management from text understanding to and knowledge-driven intelligent disease diagnosis. intelligent diagnosis of poultry and livestock diseases still critical challenges (Figure First, the and of and datasets a (Ma et al., 2025). data often expert which is and to be et al., Ma et al., 2025). or samples are due to the low of and the typically adopted in production disease are made available to over data and data and model Second, high are common in livestock and poultry production systems, making health such as physiological condition assessment, behavior and abnormal sound analysis, Finally, sensing technologies such as infrared depth cameras can be for small- and medium-scale farms, the of Besides, the and of the perception and diagnostic models in animal management systems, and environmental conditions both and model robustness and (Ma et al., 2025). of intelligent diagnosis of poultry and livestock diseases. The of and sensing technologies is for the of intelligent diagnostic The integration of and devices can further while enabling that are to small- and In addition, since many diseases physiological or abnormal and suspected cases into a and a critical for further For example, intelligent of images and the of detection devices may with disease prescription generation into diagnostic frameworks represents the step toward intelligent decision allowing systems to not diseases also and management evolution will smart livestock health management from a to a and in are poultry and livestock health management. Given the complex farming environments, multimodal which advantages of robustness and high accuracy compared with single data modalities, has shown potential in animal physiological condition assessment, behavior recognition and abnormal sound analysis, providing valuable data and knowledge for intelligent disease diagnosis. With the of LLMs and intelligent disease diagnosis is a by multimodal fusion and enabling monitoring and management for animal However, data and more robust intelligent diagnosis Future efforts should prioritize devices for definite disease diagnosis and diagnostic frameworks integrating prescription generation for intelligent decision support. Ma a in from China University in He as an with of from to as a with of and China He is a of by China for and He has in intelligent perception of multimodal information for livestock and poultry, and intelligent disease the intelligent perception approach for behavior and physiological traits in complex farm He has more than Yang is an in the of Structure and at the of and China from the of at University of research on the of livestock farming technologies and intelligent systems for animal research include smart monitoring of poultry health assessment, automatic welfare and systems for disease Yang has as the or on research from the of China has more than research and and to the of also on the of and on livestock production of to and livestock farming by integrating sensing artificial and solutions to improve animal health, welfare, and production Yu a in and from the China University in research in at the of and has been an in and of China in research on animal livestock health environment and intelligent and for livestock farms, and prevention has in intelligent diagnosis models and systems for livestock and poultry diseases, as as intelligent disease early also to the and application of especially for the in livestock and poultry farms. is a in and at the of and China with research on multimodal intelligent perception and health monitoring of livestock and The research on management and early disease of pigs. The has in research a of the and and a key of the taking for multimodal data and The research include intelligent weight estimation under conditions, and The has one on livestock and poultry sound detection and has for two to pig weight estimation is an at the of and of and from the University of and of the in two in and research on livestock animal behavior and of two by the and under the one key of research in lighting of poultry, and in poultry and over research is an at the of and China He is a of the and as a of the of the of and a of the Building research and focus on control of livestock and poultry environments, and in livestock and smart farming and the has led or in over research and has more than as or corresponding in and has over Yu a in and from the China University in He has for the of and and since 2018 has been an in the for as the of the of and Yu has in animal audio perception intelligent disease diagnosis and early technology for livestock and He has also to the and application of farming technology for a in has been in the and animal for over three on in and By to the has led the in the and laying and the contributing to a in China poultry transformation through digital intelligence the smart By integrating information technology with has a digital for the poultry providing comprehensive solutions for its digital and intelligent transformation and of the Wang is a of the of Structure and of and China He is the of the of in Structure and of and of and of the of the of research include animal and environment and management in animal intelligent sensing of animal behavior and as as smart and early systems for animal diseases. He has over and to the of and to the and of for the of the of China the of China University and and China for the of this research for the of this was for by the of The in this are of the and not the or of the of the or the of The or of Ma Yang Yu Yu and Wang