MMToT: Multi-Modal Token-of-Thought Reasoning for Large Models

With the development of Large models (LMs), recent methods tend to leverage them to complete various downstream tasks, like VQA and image caption. A typical method is Chain-of-Thought (CoT) prompting, which improves the reasoning abilities of LMs by providing intermediate steps. However, existing CoT prompts have two main drawbacks: 1) They typically represent CoT through multiple sentences, which often introduce irrelevant textual or visual context that may confuse LMs. 2) The current CoT methods fail to consider how the contributions of different tokens vary for answer inference. For this, we propose the Multi-Modal Token-of-Thought (MMToT), a novel token-level prompt method to improve LMs' multi-modal reasoning capabilities. Furthermore, MMToT stress on two strengths against CoT: 1) To prevent LMs from being affected by irrelevant contexts, we propose to extract explicit multi-modal tokens rather than sentences to construct MMToT, enhancing the reliability of generated answers. 2) To ensure that LMs prioritize tokens with high contribution scores during answer generation, we propose a confident decision-making module to evaluate and integrate each token's contribution in MMToT. Compared to existing methods, the proposed MMToT demonstrates superior performance on Science-QA, MATH, OKVQA, and VQA-introspect datasets. Furthermore, ablation studies and visualization results validate the effectiveness and interpretability of MMToT.