Can AI Conduct Autonomous Scientific Research? Case Studies on Two Real-World Tasks

Abstract Recent advances in artificial intelligence (AI) have prompted claims about autonomous “AI scientists,” yet systematic evaluations of these capabilities remain scarce. This exploratory study investigates whether current AI frameworks can execute scientific research tasks beyond isolated demonstrations. We tested eight open-source AI frameworks (Agent Laboratory, AutoGen, BabyAGI, GPT Researcher, MOOSE-Chem2, SciAgents, SciMON, and Virtual Lab) on two tasks that aimed to reproduce research on algorithm development from recent papers in uncertainty quantification and protein interaction discovery. In our evaluation, no framework completed a full research cycle from literature understanding through computational execution to validated results and scientific paper writing. While all systems showed competence in conceptual tasks such as planning and summarization, they consistently failed at robust implementation. Every framework produced sophisticated hallucinations. Deployment proved demanding, requiring substantial debugging and technical expertise, which undermines common claims about the democratization of science with AI. Despite these limitations, the frameworks showed promise as research assistants for methodological planning and ideation under careful human supervision. Our findings suggest that the explored AI systems cannot yet autonomously conduct scientific research, but may provide real value for specific subtasks within the research workflow. We offer preliminary observations to help researchers and developers better understand the gap between advertised and actual capabilities of AI in science.