VT-NeRF: view transformation and EM optimization for robust novel view synthesis from low-variation camera trajectories

Neural Radiance Fields (NeRF)-based methods have shown great success in novel view synthesis by learning scene representations from multi-view images. However, in scenarios with Low-variation Camera Trajectories (LCT), such as those found in autonomous driving, the limited variation in camera poses often leads to overfitting to the narrow set of captured views. This overfitting significantly hampers the model's ability to generalize to novel views with large deviations in perspective. To address this challenge, we propose VT-NeRF, a novel method that enhances the generalization capability of NeRF by introducing a view transformation-based approach. By performing view transformations to simulate images from different angles, our method effectively expands the angular coverage and mitigates the overfitting issue inherent in LCT-based training. Furthermore, we introduce an Expectation-Maximization (EM) training strategy to alleviate the noise introduced by the view transformations, ensuring accurate scene representation even under challenging conditions. Experimental results on public datasets demonstrate that VT-NeRF significantly outperforms existing methods, achieving 24.3% improvement in PSNR, 19.7% in SSIM, and 10.3% in LPIPS for views with perspectives far beyond the captured images.