Real-Time Segmentation and Classification of Birdsong Syllables for Learning Experiments

Abstract Songbirds are essential animal models for studying neuronal and behavioral mechanisms of learned vocalizations. Bengalese finch ( Lonchura striata domestica ) songs contain a limited number of acoustically distinct syllable types, which are combined into variable sequences. This makes them ideal to investigate the composition of vocal sequences. Many closed-loop experiments require the online recognition of a specific target syllable while the bird is singing, for example when introducing specific song modifications through reinforcement learning. In this protocol, a specific target syllable is covered with a short burst of white noise, which masks auditory feedback of the bird’s own song, and leads the bird to avoid the targeted syllable in future song renditions. Existing tools for this learning protocol require manual creation of spectral templates and have limited flexibility to adapt to new experiments. We here present Moove (Marking Online using Only the Onsets of Vocal Elements), a novel approach to real-time syllable segmentation and classification of Bengalese finch songs. A convolution-based audio encoder with a simple multi-layer perceptron is used for onset and offset segmentation of individual syllables, and syllable classification is performed by a convolutional neural network. Crucially Moove classifies syllables using only acoustic information from the first part of the syllable, while the syllable is still ongoing. This allows reinforcement or punishment to be applied with high accuracy and short latency after syllable onset, enabling effective operant conditioning experiments. The fast online annotation of all recorded syllables allows targeting syllables based on their sequence context, while keeping classification itself free from influence by acoustic information from surrounding syllables. To validate this tool, we conducted a learning experiment on one adult male Bengalese finch. The bird learned to avoid the targeted syllable sequence with comparable outcomes to previous learning experiments. Our results show that Moove can correctly segment and classify Bengalese finch syllables in real time for learning experiments. Moove could be used for other closed-loop experiments such as manipulation of auditory feedback, song-triggered neuronal microstimulation, optogenetics, or reward delivery. This makes Moove a crucial tool for future investigations of birdsong sequencing.