AI Literacy of Teachers: Prompt Engineering and Model Selection as Predictors of AI-Feedback Quality

Feedback is central to promoting pre-service teachers’ professional skills. Large language models (LLMs) such as ChatGPT are increasingly used to provide feedback. Studies show that LLM feedback can be of higher quality than that provided by experts. It is unclear which factors predict feedback quality. Two quasi-experimental studies investigated which prompt characteristics improve LLM feedback and whether prompt quality or model choice has the greater influence. Pre-service teachers formulated learning goals, for which various LLMs generated feedback. In Study 1 (N = 240 feedbacks), different prompts were tested with several LLMs. The best combinations were used again in Study 2 (N = 345 feedbacks) to test their predictive power for feedback quality. Hierarchical regressions show that both the choice of LLM and prompt quality are significant predictors. Just a few targeted principles of prompt engineering are sufficient to generate high-quality feedback. Domain-specific language is particularly effective. The selection of suitable LLMs and prompting skills should therefore be fostered as relevant characteristics of AI literacy.