Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play

Videogames provide a promising framework to understand brain activity in a rich, engaging, and active environment, in contrast to mostly passive tasks currently dominating the field, such as image viewing. Analyzing videogames neuroimaging data is, however, challenging, and relies on time-intensive manual annotations of game events, based on somewhat arbitrary rules. Here, we introduce an innovative approach using Artificial Neural networks (ANN) and brain encoding techniques to generate activation maps associated with videogame behavior using functional magnetic resonance imaging (fMRI). As individual behavior is highly variable across subjects in complex environments, we hypothesized that ANNs need to account for subject-specific behavior to capture brain dynamics properly. In this study, we used data collected while subjects played Shinobi III: Return of the Ninja Master (an action-platformer released by Sega in 1993), an action-platformer videogame. Using imitation learning, we trained an ANN to play the game while closely replicating the unique gameplay style of individual participants. We found that hidden layers of our imitation learning model successfully encoded task-relevant neural representations, and predicted individual brain dynamics with higher accuracy than models trained on other subjects' gameplay. Individual-specific models also outperformed several baselines to predict brain activity, such as pixel inputs, or button presses. The highest correlations between layer activations and brain signals were observed in biologically plausible brain areas, that is, somatosensory, attention, and visual networks. This work thus demonstrates that subject-specific imitation models can be trained from scratch and improve brain encoding in an active naturalistic task. Our framework builds on a commercial videogame of unprecedented complexity in the fMRI brain encoding literature. We used a flexible game emulator that supports a broad range of commercial videogames, opening new naturalistic interactive environments for cognitive neuroscience.