Using LLM to Generate Questions and Answers for Statistics Education: Is It Adequate?

As artificial intelligence (AI) continues to play a more prominent role in learning and instruction, educators have begun to call attention to the accuracy and reliability of such technology. In this study, we evaluated the performance of GPT-3.5 in generating questions and solutions for Advanced Placement (AP) Statistics. A total of 157 learning attributes were gathered from the AP Statistics curriculum, classified into four main sections covering 16 topic areas. For each attribute, two levels of prompting were implemented – Level 1, which simply provided a description of the attribute, and Level 2, which assigned ChatGPT a persona. Upon question and solution generation, we analyzed GPT-3.5’s responses for accuracy, identifying errors for each attribute and comparing the proportion of errors across sections, topics, and levels of prompting. Results showed that 29% of questions and solutions contained errors for Level 2, compared to 23% for Level 1. GPT-3.5 was most effective at generating solutions for Section 2 (Sampling and Experimentation), while there was no section that had the most consistently accurate generated questions. Overall, the study underscores the need to carefully evaluate the accuracy and reliability of large language models’ ability to generate questions and answers, especially those involving computational or visual output. Despite these concerns, however, the study provides insight into the potential for AI to enhance instruction and learning for the AP Statistics curriculum, given that educators are aware of its limitations.