Are Large Language Models (LLMs) Good at Research Ideation? Evidence from Chemical Engineering

Large Language Models (LLMs) are increasingly used to generate research ideas, yet systematic evidence of their effectiveness for scientific ideation remains limited. We study this question in the domain of chemical engineering by evaluating research ideas generated by three state-of-the-art LLMs across seven qualitative dimensions-Novelty, Feasibility, Simplicity, Overlapping, Relevance, Impact, Clarity. Using expert human annotations and an LLMas-a-judge framework, we analyze both absolute performance and alignment between evaluation paradigms. Human evaluations place most LLM-generated ideas in the mid-range of the Likert scale, indicating that they are generally plausible and relevant but rarely exceptional in novelty or clarity. In contrast, LLM-as-a-judge consistently assigns substantially higher scores, particularly for impact, relevance, and clarity, revealing systematic overestimation. Distributional and instance-level analysis further shows that LLM-based evaluation captures coarse quality signals but is poorly calibrated on subjective dimensions. These results suggest that while LLMs can effectively assist in research ideation, their perceived quality is highly dependent on the evaluation framework (metrics and their definitions), and LLM-based judging should not be used as a standalone substitute for human assessment.