Facial Synchronization System for an Android Avatar with an Immersive Interface to Reproduce the Operator’s Intended Expressions

To enable natural non-verbal communication during immersive avatar control, it is essential to accurately reproduce an operator’s intended facial expressions on an android avatar. This study proposes a facial synchronization system that converts facial parameters captured from head-mounted displays (HMDs), into the corresponding expressions on an android avatar. In conventional systems, actuator commands controlling facial movements are typically mapped linearly to facial parameters. However, the correspondence between android facial actions and facial parameters cannot be clearly defined. We developed a motor command mapping model (the "developer model") that generates the developer’s intended avatar expressions based on facial parameter data and actuator commands collected through repeated facial mimicry. We then proposed and evaluated a "personalized model," which adapts the developer model to individual operators to better replicate their intended expressions. In a participant experiment comparing three models (conventional, developer, and personalized), no significant differences were observed. However, the developer model—despite lacking personalization—received evaluations comparable to the conventional model, supporting the effectiveness of the proposed approach. In contrast, the personalized model received mixed evaluations, with many participants rating it lower than the other two models, indicating a need to improve the personalization process. This study offers insights for developing more effective facial synchronization systems for immersive android avatars.