A multimodal-input-enabled adaptive large language model question answering system

Summarise with VIRAT
Authors: Liwei Guo
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Large language models (LLMs) have radically transformed the model of human-computer interaction, yet the vast majority of question answering systems remain far too imited to handle multimodal input, give fexibility to their models, and offer strong reasoning for complex queries. To overcome these drawbacks, the design, implementation, and evaluation of the adaptive question answering system with multimodal input support are outlined herein in this dissertation. This study has developed a hybrid architecture of a system, combining a Next.js coordination layer and dedicated Python microservices. The architecture is capable of running a full multimodal content extraction ipeline that processes a wide variety of file types, including documents (e.g., PDF, DOCX, PPTX), images, audio, and video with the help of the latest models, including PaddleOCR, OpenAI Whisper, and BLIP. Functional analysis of this module has validated its effectiveness, achieving a Bag-of-Words F1-score of 0.832 on complex document extraction. We have developed a three-stage sophisticated hybrid intent recognition system that will intelligently direct user queries. This system is a hybrid of a finely tuned XLM-RoBERTa-large version as a high-speed domain expert and a GPT-4o Mini LLM version as a general reorderer. A classification accuracy of 93.88% was attained based on the dynamic weighted decision fusion logic with encoder confidence and entropy, which was far higher than that of the pure LLM baseline (approximately 75%) and the accuracy of the use of a fine-tuned encoder alone (89.69%). Furthermore, this study proposes and implements a novel collaboration paradigm, the “Thinker-Answerer.” By programmatically decoupling the reasoning (Chain-of-Thought generation) and synthesis (answer generation) stages, this paradigm automates user workflow processing. In qualitative case studies, it has been shown that unlike single-model techniques, the paradigm produces logically strong yet analytically rich solutions to complex queries. In summary, the dissertation offers a complete and tested prototype of one of the next-generation conversational artificial intelligence systems. It successfully integrates advanced multimodal processing technologies, high-precision intent recognition capabilities, and an innovative collaborative reasoning system.

Fields of Study & Subjects

Topic Modeling Multimodal Machine Learning Applications Natural Language Processing Techniques

Publication details

Journal / source
DR-NTU (Nanyang Technological University)
Publication date
1 January 2026
Language
EN
Document type
Thesis