Effects of facial self-touch behavior on the perceived intensity and naturalness of emotion expressions in virtual agents and social robots

This study explores whether facial self-touches corresponding to emotions enhance the perceived intensity of emotions in robots with limited facial expressions. We asked participants to perform facial self-touch corresponding to vignettes to collect facial self-touch samples. To model the relationship between each facial self-touch and the emotion perceived by humans, we conducted a survey that people mapped the emotions perceived from the facial self-touches onto a two-dimensional emotional space composed of arousal and valence axes. By applying kernel density estimation to the data, we obtained the two-dimensional probability distributions of arousal and valence for each facial self-touch. Using these two-dimensional probability distributions for each facial self-touch, we developed a method to probabilistically select a plausible facial self-touch based on an arousal and valence value. We conducted two experiments – one using virtual agents and one using robots – to verify whether this method naturally enhances emotional expression. The results showed that the proposed method enhanced the intensity of emotions and the naturalness of emotional expressions in both virtual agents and robots than the control method that selected facial self-touch randomly. In particular, for disgust and fear, the proposed method demonstrated larger effect size compared to other emotions. These results provide valuable insights for optimizing robot design and refining emotional expression methods.