Neural prediction decorrelation reveals that adversarial robustness substantially improves DNN prediction accuracy across the entire human auditory cortex

Abstract Sensory neuroscientists seek to model the neural computations that encode complex stimuli. Distinct encoding models often make similar predictions for natural stimuli such as speech, posing a challenge for model comparison. We developed a method to synthesize stimuli that decorrelate model predictions across a neural population, termed neural prediction decorrelation (NPD). Using fMRI responses to NPD sounds, we compared standard and adversarially robust deep neural network models of human auditory cortex. Prediction accuracy for NPD sounds was substantially better for the adversarially robust model in every region tested, an effect completely masked with natural sounds. Population responses to natural and synthesized NPD sounds shared an interpretable low-dimensional organization that was reproduced by the robust encoding model. NPD provides a general approach for comparing encoding models and reveals that adversarial robustness expands the predictive power of DNNs beyond natural stimuli, which is likely critical for targeting population activity through stimulus synthesis.