Engagement Annotation in Human-Robot Dialogue across Audio and Visual Modalities

For social robots used in educational applications, such as learning companion robots, maintaining student engagement is critical. There is a need for such robots to estimate engagement in real-time. This study examines dialogue data between a Nao robot and middle school students interacting conversationally while solving math problems. We collect annotations of perceived engagement, seeking to characterize human perception of engagement with the robot. Because robots that perform real-time engagement tracking do not have consistent access to clear video and audio data, we analyze perception of engagement across varying modalities. Specifically, we compare three settings: full access to audiovisual data, access to only the video data, and access to only the audio data. Our results indicate that without access to audio data, perceptions of level of engagement are lower for low-engagement segments, and without access to video data perceptions are higher for high-engagement segments.