Human Modalities in Human–Robot Interaction: A Review from Multimodal and Foundation Model Perspectives

The field of human-robot interaction (HRI) has experienced remarkable growth, fueled by the increasing integration of robotic technologies into various aspects of human life. As robots become more ubiquitous, the need for effective and natural communication between humans and robots is paramount. Human modalities, which encompass the diverse ways in which humans perceive, interpret, and express information, play a crucial role in enabling seamless and efficient HRI. This review explores the state-of-the-art in core human modalities, analyzing their strengths and limitations. Furthermore, this paper explores the realm of multimodal interaction, characterized by the concurrent employment of various modalities to enrich and fortify the interaction process. The discussion extends to the potentialities and challenges inherent in multimodal HRI. Additionally, we examine how foundation models revolutionize the semantic interpretation and fusion of human modalities.