Learning to efficiently generate and evaluating the quality of ambisonic room impulse responses for virtual spatial audio

The rise of immersive communication technologies, such as gaming, Virtual Reality and Augmented Reality (VR and AR), teleconferencing, and multimedia entertainment, has fueled significant advancements in spatial audio research, central to improving the quality of three-dimensional (3D) immersive user experiences. As VR, AR, and other immersive media applications continue to grow, creating realistic, interactive auditory environments has become paramount to enhancing user immersion and interaction. Spatial audio, which aims to recreate or synthesize a 3D auditory environment, plays a vital role in achieving this realism by preserving the directional attributes of sound sources and their acoustic characteristics. The acoustic characteristics of a room are represented by the Room Impulse Response (RIR), which is combined with sound source signals to produce a realistic audio experience for a listener. The directional attributes can be represented using mathematical models such as spherical harmonics, which forms the basis for Ambisonics, a key technology for spatial audio and is the focus of this thesis.First Order Ambisonics (FOA) refers to Ambisonics recorded using a four channel microphone that is typically converted to B format and is a practical approach to spatial audio recording and reproduction. The thesis investigates FOA representations of RIRs and their use in synthesizing realistic 3D soundscapes. However, creating accurate FOA RIRs for all potential positions in a virtual environment is computationally intensive and impractical for large-scale applications. Therefore, methods like compression and interpolation are explored to mitigate the storage and transmission costs associated with RIR datasets. The research investigates how these techniques impact spatial audio quality, particularly focusing on how RIR interpolation and compression affect the localization and spatial perception of sound. By analyzing the impacts of these processes, the study identifies effective methods for maintaining high-quality spatial audio reproduction, even under constraints.The work introduces and validates new objective quality metrics, such as the Directional RIR Objective Quality Metric (DROQM) and Kolmogorov-Smirnov RIR Metric (KSRIR), as scalable alternatives to subjective listening tests. These metrics assess the perceptual quality of FOA RIRs without requiring sound sources, allowing for efficient, large-scale evaluation of spatial audio quality. By aligning closely with human auditory perception, these metrics reduce the need for labor-intensive evaluations and offer a powerful tool for optimizing spatial audio synthesis methods, including machine learning (ML)-based techniques.The thesis first investigates ML techniques for generating single channel (mono) RIRs using CGAN-based neural network approaches that integrate both perceptually and physically relevant acoustic parameters into the training process. Loss functions based on Reverberation Time (RT), Early Decay Time (EDT), Multi-Resolution Short-Time Fourier Transform (MRSTFT), and time-domain metrics such as Mean Squared Error (MSE) are used to guide the network in replicating the acoustic properties of real environments. The research compares fixed and adaptive weighting strategies for balancing these losses during training. Results show that adaptive weightings improve accuracy and stability by dynamically adjusting the influence of each loss component. This work also highlights the superior performance of MRSTFT over conventional metrics like MSE, particularly in preserving spectral detail. Combinations of MRSTFT with RT30 and EDT yield robust results across evaluation metrics, supporting the use of multi-objective loss functions. The study presents one of the first systematic evaluations of CGAN training guided by acoustic parameters, demonstrating that this approach enhances model performance in the context of spatial audio. By learning from measured RIR datasets, the proposed CGAN models can generate responses specific to various source and receiver positions, making them well-suited for dynamic VR environments. This method allows the reproduction of realistic acoustic behavior without explicit room modeling, enabling more flexible and accurate spatial audio rendering.This thesis then extends these approaches to FOA RIRs, presenting a novel CGAN-based architecture for FOA RIR generation, conditioned on source and receiver positions and room dimensions. The method integrates a set of loss functions based on MRSTFT, RT30, EDT, Direct-to-Reverberant Ratio (DRR), Energy Decay Relief (EDR), Direction-of-Arrival (DOA), and Intensity Vector Relevance Map (IVRM), using adaptive weightings during training to enhance synthesis accuracy and stability. These losses are designed to capture essential acoustic characteristics and significantly improve the quality of the generated RIRs. The combination of MRSTFT, RT30, EDT, and DRR provides robust, balanced performance across multiple evaluation metrics, demonstrating the effectiveness of multi-objective supervision. Incorporating EDR loss further improves azimuthal localization and helps preserve spatial cues, even without an explicit DOA loss. Both EDT and EDR contribute to more stable training and higher model accuracy compared to configurations using other acoustic parameters alone. To strengthen spatial consistency, DOA and IVRM losses are applied to maintain directional coherence across Ambisonic channels. These additions significantly enhance DOA estimation, particularly in the elevation dimension. Overall, the results underscore the importance of selecting loss functions according to specific application requirements, whether emphasizing temporal, spectral, or spatial fidelity, to support diverse spatial audio rendering and room acoustics modeling tasks.The research concludes by proposing several future directions aimed at advancing the field. Key areas of focus include the generation of Higher-Order Ambisonics, the development of real-time processing techniques for VR, and the design of more efficient neural network architectures. Additionally, creating more sophisticated loss functions is proposed to enhance the accuracy of RIR simulations. By refining ML-based methods and developing advanced objective quality metrics, future work aims to improve the realism, accuracy, and scalability of spatial audio systems, ultimately enhancing the interactivity and immersive experience in VR and AR applications.