Glottal-Aware Speech-to-EMG Synthesis via Sparse-Adaptive Fusion of Speech Units

Silent speech decoding utilizes electromyography (EMG) signals to generate speech without vocalization. However, acquiring EMG datasets for silent speech is challenging because of factors such as electrode configuration (unipolar vs. bipolar), inconsistent electrode placement, inter-session and inter-device variability, and overall discomfort of the subjects. An alternative approach to address these challenges considers the inverse problem of speech-to-EMG synthesis, thereby enabling the generation of EMG signals from available speech data. In this paper, we propose a novel multimodal fusion framework that integrates electroglottography (EGG) and speech signals to enhance the representation of soft speech units. We introduce a glottal-aware EGG encoder based on a ConvNeXt-LSTM architecture, which extracts key glottal features such as the derivative EGG, glottal opening/closing points, and peak-to-peak amplitude differences. We further optimize EMG feature extraction using a ConvNext-inspired EMG encoder, replacing traditional ResNet blocks and transformer modules with FNet, which leverages Fourier token mixing for improved feature learning. To enhance EMG synthesis, we introduce a sparse fusion technique that iteratively fuses stronger neural nodes, allowing weaker nodes to adapt in later iterations. Our synthesis process is driven by a STE-GAN - based generative model, incorporating our glottal-aware fusion strategy, optimized EGG encoder, and enhanced EMG encoder to improve EMG signal quality and realism. For evaluation, we generated our novel multimodal dataset containing synchronized EGG, EMG, and speech signals to validate our methodology and compare our results with publicly available EMG-voice datasets. Extensive experiments and comparative studies with state-of-theart algorithms confirm that our method significantly improves EMG synthesis quality and speech feature transduction.