A Rubric for Evaluating Prompt Design in LLM-based Feedback: Pilot Validation

Large language models (LLMs) are increasingly used to generate formative feedback, but the pedagogical quality of the prompts that guide these models remains underexamined. This paper evaluates the reliability and practical applicability of the Pedagogical Prompt Engineering (PPE) rubric, designed to assess the pedagogical quality of LLM prompts. The study followed a two-phase mixed-method design. In Phase 1, three computer science educators independently rated 22 prompts using ten criteria; Fleiss’ Kappa indicated almost perfect agreement (κ = 0.858). In Phase 2, eight CS1 instructors applied the rubric within a realistic feedback generation workflow and evaluated it across six usability dimensions. Quantitative ratings and reflective comments showed that the rubric is clear, usable and pedagogically meaningful in practice. The findings provide initial evidence that the PPE rubric is a reliable and practical tool to evaluate and improve the prompt design of LLM-based formative feedback.