Automating Research Intelligence: Advanced-AI-Generated vs Manually Designed Pipelines

Automated research intelligence systems are increasingly needed to track fast-moving scientific domains where key developments first appear in preprints, news, and social media rather than curated citation databases. At the same time, agentic AI frameworks promise to synthesize such systems automatically from natural language specifications. In this paper we empirically compare advanced-AIgenerated pipelines against manually designed pipelines on the same research intelligence tasks. We first implement two independent manual systems for narrow technical subfields, with multi-source collection of thousands documents, embedding and clustering of highquality items into coherent topics. Both pipelines run end-to-end with predictable cost, robust error handling, and outputs that are useful to expert readers. We then evaluate several AI frameworks, including AutoGen, AutoGPT, OpenAgents, OpenHands, Claude and Gemini, asking them to construct comparable pipelines. While the frameworks that are still functional rapidly produce large volumes of professional-looking code at low direct API cost, none yields a fully functional pipeline without extensive human intervention; common failures include incorrect API usage, scientifically invalid data handling, brittle execution, and opaque failure modes. Debugging and partial rewrites require tens to hundreds of hours, comparable to or exceeding the effort needed to build the manual pipelines from scratch. We conclude that current AI code-generation and agent frameworks are better viewed as assistive components within deterministic, manually engineered backbones rather than as autonomous pipeline builders, and we outline design principles for such hybrid research intelligence systems.