SLNeRF: Joint Optimization of Structured Light and NeRF

Recently, based on multi-view stereo (MVS) methods, utilizing stereo prior information to guide novel-view synthesis has become an important approach to addressing the generalization problem of neural radiance fields (NeRF). However, this approach faces challenges in handling certain difficult scenarios, such as weak texture regions, where it struggles to effectively extract geometric features information. As a result, it encounters limitations in stereo image feature representation and inaccuracies in prior depth estimation. To tackle these problems, we propose the first generalizable novel-view synthesis method that jointly optimizes structured light and NeRF. Considering that active stereo methods based on structured light optimization can enhance geometric feature extraction capability in weak texture regions through the addition of a texture layer, we integrate active stereo vision into novel-view synthesis. To this end, we propose a novel framework, dubbed SLNeRF. First, we design a structured light generation scheme based on Fourier transform and establish a differentiable imaging model using geometric optics and the Lambertian model to generate active stereo images. Then, we obtain stereo image features, as well as prior depth information through the feature extractor, which are used to construct 3D feature volumes. Finally, we accomplish the novel-view synthesis task through a neural renderer. Compared to state-of-the-art generalizable NeRF methods, our method reports encouraging results on public datasets as well as in real-world scenarios.