Combining Gaussian and Pixel Representation for Light Field View Reconstruction

Light field (LF) benefits various applications due to its rich spatial and angular information. To address the technical limitation in terms of imaging resolution, LF view reconstruction becomes a research hotspot. However, relevant methods mainly focus on pixel representation modeling on image plane but ignore the importance of scene geometry modeling. Inspired by powerful geometry description ability embedded in 3D Gaussian Splatting, we construct a network called LFGaussian to perform generalizable LF view reconstruction in this paper. Specifically, owing to the unique composition of cross-view Gaussian attribute deviation under 4D LF imaging setting, we propose disparity-guided feed-forward 2D Gaussian propagation with novel Gaussian primitive definition, subtly implementing Gaussian unprojection-projection operation in camera parameter-free case. On this basis, we introduce a dual-branch workflow including Gaussian representation rendering and pixel representation upsampling to create features of target views from two different levels, which complement each other to jointly realize geometric structure consistency as well as texture detail consistency across all target views. Besides, for the pursuit of high-efficient and high-quality Gaussian representation rendering, we design sub-sampling Gaussian decoding to alleviate Gaussian redundancy and leverage Gaussian splitting to allocate additional Gaussians for complex geometry regions identified by disparity gradient. Experimental results show that the proposed LFGaussian achieves superior performance compared with state-of-the-art methods on both real-world and synthetic LF datasets, proving the effectiveness of introducing Gaussian representation for LF view reconstruction. Furthermore, our LFGaussian supports arbitrary-scale reconstruction, showing high flexibility for the upsampling scale factor.