LLMs as useful training tools for mathematics teachers: opportunities or hallucinations?

Abstract We explore the extent to which answers to mathematics word problems generated by AI chatbots are comparable to students’ answers, and whether interaction with them is useful for replicating classroom discourse. We use a large language model (LLM) to solve ten standard textbook exercises 100 times for each question and analyse the answers to identify tasks suitable for further interaction through teacher prompts. Based on this analysis, we derive suitable teacher interventions, in the form of follow-up prompts, and determine their impact on the LLM output. We show that LLM-generated answers can indeed resemble students’ answers and argue that these answers can be used with preservice educators as they learn to intervene productively in student learning. However, we also found that generic follow-up prompts can be more effective than specific prompts based on pedagogical content knowledge in generating a correct LLM response. Taken together, these findings suggest that LLMs may be well suited to modelling students’ mathematical solutions, but less suited to modelling how students naturally respond to instructional prompts. This sheds light on the suitability of LLM training in teacher education, but also on modern reasoning models that can improve the reliability of LLM answers.