Hybrid EEG-fNIRS phoneme classification based on imagined and perceived speech

Introduction: Individuals affected by severe motor impairments often have no means of communicating with others. To build an intuitive speech prosthesis, imagined speech brain-computer interface research began to prosper with numerous studies attempting to classify imagined speech from brain signals. While unimodal neuroimaging techniques, such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have been widely used, multimodal approaches combining two or more of them remain scarce. Methods: In this study offline phoneme decoding based on hybrid EEG-fNIRS data was performed. Twenty-two right-handed participants performed imagined and perceived speech trials encompassing four phonemes /a/,/i/,/b/ and /k/. Features in the form of power spectral densities and mean hemoglobin concentration changes were extracted from EEG and fNIRS data, respectively. Features were ranked according to the mutual information criterion relative to the target vector, and the optimal number of features to include was determined through optimization via 10-fold cross-validation. Results: Hybrid classification yielded accuracy scores of 77.29% and 76.05% regarding imagined and perceived speech, respectively. In both conditions, hybrid and EEG-based classification performances did not differ significantly, while fNIRS based phoneme discrimination produced lower accuracies. Discussion: This study represents an innovative phoneme decoding attempt based on multimodal EEG-fNIRS data, both in terms of imagined speech and perception. Four-class imagined speech classification was primarily driven by EEG features yet outperformed comparable previous studies.