Can We Automate Scientific Reasoning in Closed-Loop Experiments using Large Language Models?

We present a detailed benchmarking study of hybrid large language model (LLM) / Bayesian optimization (BO) frameworks for accelerating scientific discovery through closed-loop experiments. The study evaluates five modern LLMs (o4-mini, o3, gpt-5-mini, gpt-5, and gemini-2.5-flash) as optimizers within the BORA (Bayesian Optimisation Research Assistant) framework, comparing their performance against BO-only and LLM-only approaches across: 10-dimensional photocatalytic hydrogen evolution optimization (chemistry problem) 7-dimensional physics-based pétanque simulation (physics problem)