Single-view neural illumination estimation and editing for dynamic light field display

Abstract Light field rendering is widely applied to virtual reality (VR), augmented reality (AR), mixed reality (MR) and extended reality (XR). For photorealistic light field displays, it requires a dense view sampling of the scene. However, in dynamic immersive interactions, the available observations are often too sparse to synthesize the complete light field required for a high-fidelity display. Therefore, it poses a huge challenge for generating photometrically consistent views between the virtual and real world. Here, we introduce a neural illumination estimation and editing framework for adaptive light field synthesis. The proposed method can explicitly encode intrinsic parameters of illumination from one single sampling view, which is used for a hybrid-guided generative network to synthesize photometrically plausible dense views of the scene under the guidance of a complete rendering model. It deconstructs the baked-in lighting to enable consistent and high-fidelity relighting from any viewpoint. Our method estimates and edits illumination with only 0.2397 W m −2 irradiance error and 7.02 ∘ angular deviation, yielding synthesized images with an average 17.0% improvement in PSNR and a 51.2% reduction in LPIPS. This work presents a practical pathway towards truly interactive and adaptive digital light fields, enabling photorealistic content generation for the next generation of near-eye displays and computational imaging systems.