Medbench V4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents

Background: Recent advances in medical large language models (LLMs), multimodal models, and agents demand evaluation frameworks that reflect real clinical workflows and safety constraints.Methods: We present MedBench v4, a nationwide, cloud-based benchmarking infrastructure comprising over 700,000 expert-curated tasks spanning 24 primary and 91 secondary specialties. The platform features dedicated tracks for LLMs, multimodal models, and agents. Items undergo multi-stage refinement and multi-round review by clinicians from more than 500 institutions. We evaluated 15 frontier models, using an LLM-as-a-judge approach—calibrated to human ratings—to score open-ended responses.Results: Base LLMs reached a mean overall score of 54.1/100 (best: Claude Sonnet 4.5, 62.5/100), but performance in safety and ethics remained low (18.4/100). Multimodal models performed worse overall (mean 47.5/100; best: GPT-5, 54.9/100), demonstrating solid perception yet weaker cross-modal reasoning. Conversely, agents built on the same backbones substantially improved end-to-end performance (mean 79.8/100), with Claude Sonnet 4.5–based agents achieving up to 85.3/100 overall and 88.9/100 on safety tasks.Conclusions: MedBench v4 reveals persisting gaps in multimodal reasoning and safety for base models, while showing that governance-aware agentic orchestration can markedly enhance benchmarked clinical readiness without sacrificing capability. By aligning tasks with Chinese clinical guidelines and regulatory priorities, the platform offers a practical reference for hospitals, developers, and policymakers auditing medical AI.​