Cognitive Behaviours of Large Language Models

Large language models (LLMs) have reshaped natural language processing, demonstrating remarkable reasoning across diverse tasks. Their rapid real-world integration has fostered growing reliance on their perceived cognitive abilities, with users expecting models to adapt to evolving information, perform consistently across languages, and handle precise computations. However, a central question remains: to what extent do LLMs exhibit consistent cognitive behaviors under dynamic, diverse real-world conditions? This thesis explores LLM use cases in practical settings requiring adaptability, inclusivity across multilingual users, and computational precision. First, we study LLMs’ cognitive limitations in revising inferences when presented with new, contradictory, or clarifying information. Existing benchmarks assume complete context, neglecting information’s dynamic nature. We introduce a sequential evaluation framework simulating scenarios requiring belief revision, revealing that strong LLMs often cannot revise prior beliefs. We identify a fundamental trade-off: models adept at updating underperform when no revision is necessary, and vice versa—exposing key tensions in model adaptability. Second, we analyze whether LLM cognitive behaviors remain consistent across languages. Despite training on large multilingual corpora, LLM capabilities remain uneven across languages. Analyzing internal representations, we uncover inconsistent interlingual alignment and show that monolingual fine-tuning further disrupts this alignment. To mitigate this, we propose a selective freezing strategy during fine-tuning that preserves interlingual representation alignment and improves cross-lingual generalization. Third, we address LLM limitations in tasks requiring precise computation, particularly in LLM-based judges for domains demanding mathematical exactness. Programmatic tools retain advantages over purely language-based reasoning. We develop a multi-turn reinforcement learning framework enabling LLM judges to selectively invoke an external Python interpreter as part of their reasoning process. This framework enables judges to learn beneficial tool invocation strategies, delegating sub-problems requiring exact computation to code execution. Judges integrate execution outputs into their chain-of-thought reasoning, mitigating errors inherent to text-based computation and enhancing judgment accuracy.