Mouse Lockbox Dataset: Behavior Recognition for Mice Solving Lockboxes

Abstract Machine learning and computer vision methods have a major impact on the study of natural animal behavior, as they enable the (semi-) automatic analysis of vast amounts of video data. Such large-scale analysis is essential not only to mere behavioral research but its applicability spans across a large range of disciplines. Mice are the standard mammalian model system in most behavioral research fields, but the datasets available today to refine such methods focus either on isolated or social behaviors. In contrast, datasets in which animals interact with a physical apparatus, which is highly relevant across disciplines that study learning, are as of yet unavailable. In this work, we present a video dataset of individual mice solving (multi-step) mechanical puzzles, so-called lockboxes. The more than 110 hours of total playtime show their behavior recorded from three different perspectives. As a benchmark for frame-level action classification methods, we provide human-annotated labels for all videos of two different mice, equaling 13% of our dataset. Our action labels provide two levels of complexity, requiring the classification of both what the mouse is doing as well as which object is targeted. Additionally, as an initial comparison against the human-annotated labels, we used two different keypoint (pose) tracking-based action classification frameworks as well as an autoencoder-based framework, which illustrates the challenges of automated labeling of fine-grained behaviors, such as the manipulation of objects. We hope that our work will help accelerate the advancement of automated action and behavior classification in a task-driven environment. Our dataset, including videos and human-annotated labels, is publicly available at https://doi.org/10.14279/depositonce-23850 .