Multilevel Multimodal Visual Neural Decoding for Generalized Visual Object Recognition

The decoding of human visual neural signals has garnered significant attention due to its potential to reveal the underlying cognitive mechanisms of the brain. However, existing visual neural decoding methods mainly focus on instance-level multimodal semantic correspondences, often neglecting local-level and concept-level supervisory information. These limitations substantially weaken their generalizability to downstream tasks. In this study, we propose a multi-level multimodal visual neural decoding (MMVND) framework that leverages multi-level semantic correspondences between paired image stimuli and brain activities to address these limitations. Specifically, we initially employ the instance-level alignment module to capture the correspondence between brain activity-image pairs. Subsequently, a cross-attention mechanism is utilized to learn the matching between tokens from different modalities. The local-level alignment module further exploits the local semantic information based on matched tokens to establish fine-grained local correspondences. To capture concept-level correspondences, we design a momentum update strategy to generate prototypes for different modalities. The concept-level alignment module then utilizes the prototype-based semantic information to capture the concept-level multimodal correspondences. Synergistic learning across multi-level multimodal semantic correspondences significantly enhances the performance of generalized visual object recognition. Extensive experiments conducted on publicly available visual neural datasets demonstrate the superiority of the proposed multi-level multimodal learning framework.