Toward Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering

Recent methods to enhance Vision-Language Models (VLMs) for Visual Question Answering (VQA) have focused on strengthening their inference capabilities, enabling them to tackle VQA tasks independently rather than merely as aids to Large Language Models (LLMs). However, these approaches often ignore the rich commonsense knowledge inside the given VQA image sampled from the real world, limiting the full potential of VLMs. Inspired by the human top-down reasoning process, i.e., systematically exploring relevant issues to derive a comprehensive answer, this work introduces a novel, explainable multi-agent collaboration framework by leveraging the expansive knowledge of LLMs to enhance the capabilities of VLMs themselves. Our framework comprises three agents, i.e.,Responder,Seeker, andIntegrator, to collaboratively answer the given VQA question by seeking its relevant issues and generating the final answer in such a top-down reasoning process. The VLM-basedResponderagent generates the answer candidates for the question and responds to other relevant issues. TheSeekeragent, primarily based on LLM, identifies relevant issues related to the question to inform theResponderagent and constructs a Multi-View Knowledge Base (MVKB) for the given visual scene by leveraging the build-in world knowledge of LLM. TheIntegratoragent combines knowledge from theSeekeragent and theResponderagent to produce the final VQA answer. Extensive and comprehensive evaluations on diverse VQA datasets with a variety of VLMs demonstrate the superior performance and interpretability of our framework over the baseline method, e.g., 5.7% improvement on VQA-RAD and 5.2% on Winoground in the zero-shot setting without extra training cost.