Can Large Language Models Generate Higher Order Thinking Math Problems? An Evaluation Study

Higher-order thinking skills (HOTS) are essential for fostering students’ critical abilities and are linked to successful educational outcomes. In mathematics, access to high-quality HOTS problems is particularly important for skill development, yet creating such problems is challenging and time-consuming. Given the advanced mathematical capabilities of state-of-the-art large language models (LLMs), this paper presents a human and computational evaluation of GPT-4o’s ability to generate mathematical HOTS problems in algebra and logic. We examine the quality and versatility of problems produced under different prompting strategies, including a proposed method named prompt inference using problems as exemplars. Experimental results show that GPT-4o can generate high-quality HOTS problems, with prompt inference yielding the most effective results, highlighting its potential as a tool for developing this type of educational material. Limitations and pedagogical considerations are analyzed and presented