Modalities of interaction

The previous two chapters were brief overviews of the human information processing system and different knowledge representation techniques in artificial intelligence. This chapter will explore how the two fields work together to exploit human sensory and motor systems to facilitate information processing through different modalities of interaction. We will introduce different modalities of interaction using eye gaze tracking, gesture recognition, speech recognition, AR/VR displays, haptics and spatial audio interfaces. For each of these topics, we will initially learn the definition, then the advantages and disadvantages, followed by implementation principles and technical challenges towards deployment. We will also highlight present and prospective use cases for each modality and the basic concepts of developing a multimodal interactive system.