MuDoC: A Multimodal Document-grounded Conversational AI System

MuDoC is a MUltimodal DOcument-grounded Conversational AI system for multimedia learning. MuDoC generates responses interleaved with text and images in a format similar to news or blog articles. Based on GPT-4o, it uses past chat as context to understand user queries and responds based on retrieved visuals and text from documents. Its user interface (UI) allows seamless navigation to image and text sources, and enables verification of AI generated content. This feature enables learners to verify the generated content, thereby fostering trust in AI systems for information seeking. Our work is a step towards creating trustworthy multimodal intelligent documents and employing them in learning environments. Some examples of conversations can be viewed at https://tinyurl.com/MuDoCSamples.