Bot or Prof: Comparing the Characteristics of AI-generated and Instructor-written Responses to Student Queries in an Introductory Programming Course Forum

One promising application of Generative Artificial Intelligence (GenAI) in large introductory computing courses is to power automated tools for responding to student queries on online course forums, which can greatly reduce instructor workload and improve the timeliness of responses to students. However, there is a need to carefully evaluate the suitability and effectiveness of AI-generated responses. We address this by comparing bot responses, which are automatically generated using GenAI, and instructor responses, which are manually written by instructors, to student queries on an online forum for a large introductory programming course. Using a rubric derived from previously identified characteristics for effective feedback in programming education, we assess responses on qualities such as helpfulness and conciseness. We find that while bot responses excel in some areas, such as timeliness and comprehensiveness, they are often verbose and not always effective in handling administrative queries or in capturing the interpersonal aspects of human communication. Our findings suggest that AI-generated responses could be immediately published to students, but only as constructive starting points that instructors should review, comment on, or revise to ensure their accuracy and usefulness. This hybrid approach has the potential to create more effective learning environments that leverage the strengths of both automated and human feedback.