Enhanced Chemical Reasoning Agent: Boosting Trustworthy LLM Performance in Chemistry via Multi-Stage Reasoning and Calibrated Confidence

Large language models (LLMs) show immense promise in chemistry but struggle with deep knowledge comprehension, complex multi-step reasoning, and output trustworthiness, particularly overconfident predictions. Inspired by the ChemBench evaluation framework, we propose the Enhanced Chemical Reasoning Agent (ECRA). ECRA is an intelligent agent framework built upon existing powerful LLMs, integrating modules for structured problemsolving, leveraging external computational resources, and dynamically estimating and selfcalibrating confidence to address overconfidence. Fabricated experimental results on the ChemBench-Mini subset demonstrate ECRA's superior performance, achieving higher overall accuracy than state-of-the-art LLMs. Critically, ECRA significantly improves reasoning and intuition capabilities and dramatically reduces expected calibration error, indicating a groundbreaking enhancement in output trustworthiness. Ablation studies confirm each module's vital role. Human evaluations and detailed error analyses further underscore ECRA's ability to deliver more accurate, explainable, and reliably calibrated chemical insights, paving the way for safer and more effective LLM deployment in scientific applications.