RGB-D Fusion for Wide Field of View User Feedback in Teleoperation Context

Effective teleoperation involves immersive and responsive visual feedback to support depth perception and spatial understanding to achieve precise control. Standard camera views naturally constrain the operator’s Field of View (FoV) of the remote scene, especially in cluttered or dynamic scenarios. We present a real-time RGB-D fusion system that expands the operator’s FoV by employing immersive 3D reconstruction. Our system incorporates the Azure Kinect sensor into Unreal Engine using the Robot Operating System (ROS) communication, rendering live sensor information onto a spherical mesh. This allows for smooth, wide-FoV rendering of the scene with greater peripheral context and depth continuity. In contrast to planar or depth-free systems, the proposed method is enhanced by live depth retranscription for more interactive teleoperation, leading to better scene understanding. This architecture lays the basis for flexible, high-fidelity remote interaction for robotics applications. All our developments and implementations are publicly available at https://github.com/isri-aist/RGB-D_Fusion.