4DM: 4D photorealistic mapping from monocular video with 3D Gaussian splatting

Recovering dynamic 4D photorealistic scenes from monocular video without prior camera parameters is an essential yet challenging task. To address limitations in current methods regarding reconstruction accuracy and dynamic consistency, we introduce 4DM, a new framework that reconstruct 4D photorealistic scenes from monocular video with 3D Gaussian Splatting. 4DM employs solely 2D priors, namely depth and optical flow, to transform a video into a 4D scene. 4DM first divides the video into static and moving parts, then alternately optimizes the camera poses and the dynamics of the 3D Gaussian points. For newly added frames, a two-step process is employed. Initially, image gradient-based quad-tree sampling is utilized to minimize redundant sampling. Subsequently, K-Nearest Neighbor is employed to densify the new Gaussian points. The integration of these technologies has enabled 4DM to overcome the limitations of conventional video-based 4D restoration methods. Extensive experiments conducted on the DAVIS dataset demonstrate substantial improvements in visual fidelity compared to state-of-the-art approaches.