VowelVC: A neural vowel-centric voice conversion framework with incremental memory learning for few-shot speaker adaptation

This paper presents VowelVC, a neural voice conversion framework that integrates vowel-centric acoustic modeling with modern deep learning architectures for robust few-shot speaker adaptation. The system addresses data scarcity in voice conversion by leveraging the concentration of speaker-specific acoustic information in vowel formant structures and steady-state spectral characteristics. VowelVC comprises five key components: a neural vowel feature extractor learning compact vowel-specific representations, a dynamic memory bank with differentiable attention-based retrieval storing vowel prototypes, an enhanced content encoder conditioning on memory-augmented vowel features, a conditional generator supporting both adversarial and diffusion-based synthesis, and an incremental adaptation module enabling single-utterance speaker adaptation through constrained parameter updates. The training methodology employs three-stage curriculum learning progressively building general acoustic models, refining vowel-specific transformations, and developing incremental adaptation capabilities. Extensive experiments on CMU ARCTIC corpus (4 speakers, 1,132 utterances each) with rigorous statistical analysis demonstrate that VowelVC achieves mean Log Spectral Distance of $7.89 \pm 1.24$ dB compared to $14.93 \pm 2.18$ dB for VAE-GAN baselines (47.1\% reduction, $p < 0.001$, Cohen's $d = 4.12$), and Mean Opinion Score improvement from $3.31 \pm 0.82$ to $3.88 \pm 0.69$ (17.2\% increase, $p < 0.001$, $N = 20$ listeners) with only one to three adaptation utterances. The framework maintains real-time inference capability (RTF $< 0.85$ on NVIDIA V100 GPU) with memory footprint of 2.8 GB for GAN variant and 4.1 GB for diffusion variant, enabling deployment in resource-constrained environments.