Context-Aware Embodied Question Answering Agent With Large Language Model

Embodied Question Answering (EQA) requires agents to explore the environment actively and gather reasonable visual information to answer the questions raised by users. Despite the significant progress attained, previous works lack a comprehensive understanding of the observed environment. Particularly, in the question answering task, little attention is paid to the temporal context in multiple observations and the semantic understanding of the environment. Meanwhile, the navigator struggles when combining global spatial context and individual observations to comprehend the geometric environment. In this paper, we propose Context-aware Embodied Question Answering (CaEQA) to address the issues from two aspects. Firstly, to integrate temporal context into image embeddings obtained from the local observations at the navigation endpoint, we propose a local temporal context-aware visual question answering module (LTC-VQA). Specifically, we utilize the large language model in the question answering module to enhance the temporal and semantic understanding of complex unseen environments by introducing general knowledge obtained in the pretraining stage. Secondly, to model the complex interdependence within the overall navigation procedure, we propose a global-geometric-guided navigation module (G3-Nav), facilitating the interpretation of global historical information. In G3-Nav, we use a transformer that incorporates RGB images, depth maps and actions from all past positions to provide geometric hints from inter-image and intra-image perspectives. Experimental results on the Matterport3D and House3D datasets demonstrate that our method outperforms the SoTA methods in terms of navigation, question answering and the overall EQA system.