MagicLiveRoom: a livestream system for VTubers to seamlessly interact with real-world objects

With the growing popularity of VTuber live streaming, there is increasing interest in enhancing the realism and interactivity of virtual avatars. Existing VTuber live streaming systems mainly focus on visual performance, with limited integration of real-world object interaction. To address this issue, we propose MagicLiveRoom, a system that unifies computer vision and image processing techniques within VTuber live streaming. Our contributions are as follows: (1) we design a novel integration framework that seamlessly brings real-world objects into virtual performance by reconstructing human-object interactions on avatars; and (2) classify interaction types and address occlusion issues to maintain visual coherence in a 3D-avatar context. We evaluate our system by having a user hold different objects during live streaming, with reconstructed avatar-object interaction results. The results demonstrate that MagicLiveRoom enables convincing and immersive virtual-real interaction for VTuber live performances.