Ultra-Low Latency Video Synthesis of Split-Screen Processing for Real-time Remote Sessions

Real-time remote music sessions require not only ultra-low latency audio but also synchronized video exchange among performers to achieve a sense of unity. Conventional web conferencing systems are unsuitable because of excessive video delay caused by codec processing, network jitter, and split-screen rendering. In particular, simply transmitting the video of one performer or conductor is insufficient, since musicians must mutually observe each other’s gestures and instrumental movements. To address this, we propose an ultra-low latency video synthesis of split-screen processing technology that reduces video processing delay to approximately 10 ms by employing a stream-based frame allocation strategy. Unlike conventional frame-synchronization methods, the proposed approach minimizes frame waiting time by processing video streams in arrival order. A prototype system was implemented on an FPGA and evaluated through both simulations and real-world experiments. Professional orchestra musicians performed remote ensemble sessions across four performance sites in three different cities connected via an optical network. The results showed that the proposed system enabled musicians to perform without perceivable video delays, significantly improving synchronization, timing accuracy, and musical expression compared with high-delay ( ≈ 150 ms) and audio-only conditions. These findings demonstrate that low-delay multi-view video is essential for remote ensembles and suggest broader applicability to remote education, rehabilitation, and collaborative performances.