Photometric Virtual Visual Servoing based on Gaussian Splatting

We present a novel approach that integrates photometric Image-Based Virtual Visual Servoing (IBVVS) with Gaussian Splatting (GS), a recent and efficient 3D representation for photo-realistic view synthesis. In our framework, the servoing process is performed entirely in simulation using a GS model trained on a sparse set of images from a scene wheras unseen images serve as target views for photometric IBVVS. At each iteration a rendered image from the GS model simulate the camera’s current view, and the pixel-wise intensity error between the rendered and target images is used to compute control commands for camera pose optimization. This framework removes the need for externally acquired explicit 3D geometry or precomputed dense depth maps from traditional sensors, since depth information is implicitly obtained from the GS representation and used directly in the control loop. The method enables virtual servoing toward novel views that were not captured during training. Experimental results demonstrate accurate and smooth convergence, highlighting the potential of learned view synthesis for 3D camera tracking and visual servoing applications.