Lameness detection in dairy cows using pose estimation and bidirectional LSTMs

• We combine pose-estimation and Bidirectional Long-Short-Term Memory (BLSTM) neural networks. • Learning temporal motion features from the keypoint trajectories removes the need for manual features engineering. • BLSTM classifiers outperform SVMs and manually-designed locomotion features. • BLSTM classifiers can detect lameness with as little as one second of video data. Lameness is a prevalent health condition in dairy cattle that negatively impacts welfare and productivity. Automatic lameness detection would allow for timely detection and treatment with limited labor. This study presents a lameness detection approach that combines pose estimation and Bidirectional Long-Short-Term Memory (BLSTM) neural networks. Combining pose-estimation and BLSTMs offers the following advantages: markerless pose-estimation, elimination of manual feature engineering by learning temporal motion features from the keypoint trajectories, and working with short sequences and small training datasets. Motion sequences of nine keypoints (located on the cows’ hooves, head and back) were extracted from videos of walking cows with the T-LEAP pose estimation model. Keypoint sequences were then used as an input to a BLSTM classifier that was trained to perform binary lameness classification. We experimented with different model architectures and with sequences lengths of one, two and three seconds. The best model architecture consisted of three BLSTM layers of size 128, and the experiments revealed that our BLSTM classifier could detect lameness with as little as one second of video data. Additionally, we performed a direct comparison against an established method based on manually-designed locomotion features and an SVM classifier. The established method achieved a classification accuracy of 80%. On the same dataset, our BLSTM classifier outperformed the established method and achieved an accuracy of 85%.