Mind-VAD: Brain-Inspired fMRI-to-Video Precise Reconstruction via Cross-Modal Autoregressive Diffusion

Decoding visual information from brain activity is important and challenging. Existing studies have successfully reconstructed static images from fMRI signals, but fMRI-based dynamic visual reconstruction still has limitations: the low temporal resolution of fMRI limits the coherence of the video, the lack of fine-grained alignment of brain with visual features at different scales, and the deviation of diffusion modeling from the logic of visual encoding. Research has shown that the visual cortex cognitive system streams perceptual and semantic information to different brain regions for processing. Inspired by this, we propose Mind-VAD, a novel dynamic visual reconstruction paradigm that redefines the brain decoding process as cross-modal generation guided by visual stream. Specifically, we design a time-sensitive encoder to accurately achieve temporal localization, and learn different dynamic visual representations of fMRI through an autoencoder with different scales. Then proceed with full-sequence, faster video autoregressive diffusion generation through a coarse-to-fine process. We include the latest AIGC Benchmark in evaluation. Extensive experiments show that Mind-VAD realized accurate and smooth reconstruction, achieves SOTA performance (87.3% classification, 27.5% semantic improvement) on several downstream tasks and more demanding metrics. Overall, Mind-VAD provides a neural interpretable and powerful framework for visual decoding and its potential applications.