Testing Explainability of Chain of Thought for Large Language Models

Large Language Models (LLMs) have demonstrated superior abilities in complex tasks such as text generation, reasoning, and question answering. However, the explainability of LLMs becomes weak as the parameters and complexity of LLMs increase. Chains of Thought (CoTs) guide the model to perform step-by-step reasoning and effectively enhance its reasoning ability. The multi-step rationales verbalized in a CoT are widely regarded as the explanation of the model itself. This paper proposes an automated approach to testing the behavioral sensitivity of responses to self-cited evidence in CoTs from sufficiency and necessity perspectives under context intervention. Specifically, we intervene in the reasoning chain by changing the input context and measure the behavioral consistency as a proxy for the faithfulness of the CoT. We test the CoT rationales of mainstream open-source LLMs on multi-hop question-answering tasks. The experimental results show that the self-stated reasoning chain is insufficient and unnecessary. The CoT cannot fully explain the behavior of LLMs.