Visual Information Reconstruction Using Brain Signals: Challenges of a Multimodal Dataset and Model

This study explores an alternative possibility for brain-computer interface (BCI) technology to suppo rt real-time activities such as online learning for the visually impaired. Acknowledging the limitations of existing technologies, which primarily focus on replacing hand functions in visually-enabled environments, the study raises the need for research into image generation and visual information reconstructio n using brainwavesto visualize a user’s thoughts. To this end, a multimodal dataset including speech-text pairs and corresponding brain signals was constructed, and the validity of the model was evaluated. Data were collected from three groups of subjects—the visually impaired, those with visual impairments, and those with normal vision—using an EPOC X-14 channel wireless EEG headset. The experimental results showed high training accuracy in signal-to-text alignment, but confirmed that the model’s generalization ability was limited due to the small size of the dataset. This study demonstrates the potential of this approach while emphasizing the need for future research to build large-scale datasets and delve into complex contextual understanding.