Deep Speech Representation Learning for Research in Phonetics

The speech signal is a rich source of information that conveys not only linguistic but also extra/paralinguistic information, such as the speaker’s identity, gender, emotional state, age. However, those traits are hidden in complex, non-transparent variations of the speech signal. With recent progress in speech synthesis and voice conversion caused by the advent of deep generative modeling, we argued at the time of writing the proposal that synthesized speech can (again) become a valuable tool for research in phonetics. The overarching goal of this project was thus to explore the potential of deep generative modeling of individual dimensions of speech as a tool to support research in phonetics. While the initial choice to adopt an encoder-decoder architecture with the targeted manipulation done in the latent embedding space turned out to be appropriate, progress in the field led to a continuous adjustment of the work packages to the current state of the art. At first, the basic architectural choice for voice conversion was the variational autoencoder (VAE). We extended the factorized VAE, which we had developed at the time of writing the proposal [O1], to enable adaptation of the speaking rate to that of the target speaker. This adaptation led to higher voice similarity between a synthesized and target speaker [1]. However, more recent generative models were reported to be superior. In a joint effort with the project TRR318 (project C06), we thus developed a synthesis system based on normalizing flows, leading to higher quality speech samples. [2], [3]. Disentangling sources of variation of the speech signal was considered an important task to be able to generate speech signals with desired properties. While the disentanglement of high-level factors of variation, i.e., content vs speaker induced variations [1], and even a separation of speaker from environment or emotion related variations was possible [4], a more fine-grained disentanglement of voice properties turned out to be infeasible [5]. Furthermore, the quality of the artificially produced speech samples needed to be adequately assessed. To address this issue, we carried out user studies to determine the dimensions of perceived quality and similarity of utterances produced by Text-to-Speech (TTS) and voice conversion (VC) systems[1], [6], [7]. Furthermore, we developed an approach towards the prediction mean opinion scores (MOSs) at a higher temporal resolution to pinpoint problematic areas within the signal. [8]. The main project results can be summarized as follows • We developed a fully unsupervised any-to-any voice conversion system with the speaking rate adapted to the target speaker’s rate. It clearly outperformed the then state-of-the-art any-to-any VC system AutoVC [O2]. • We investigated the speaker embedding vector computed by a VAE w.r.t. disentanglement, both supervised (using acoustic features as proxies for the true but unknown factors of variation), and unsupervised (using information-theoretic measures). We found that disentanglement could be improved by wisely choosing the parameters of a β-VAE and a Total Correlation VAE (TCVAE). • We distinguished new perceptual dimensions for modern state-of-the-art speech synthesis models and investigated their correlation to acoustic parameters as well as the ability to elicit them in the time domain. • As a first step towards automatic subjective speech quality assessment with frame-level resolution we developed a predictor to find segments of low speech quality.