Evaluating large language models for generating programming questions from code

Abstract E-learning platforms demand effective question-generation methods, and large language models have made this easier. While recent studies focused on generating questions from text, no prior research has evaluated their ability to generate questions from code. This work presents a framework to evaluate the models in question generation from code using average scores, win rates, and Elo ratings. Python, C++, and Java code were used to prompt models. The evaluation includes automatic metrics and human evaluators. Findings show GPT-4-0314, GPT-4-0613, and GPT-3.5-turbo-0613 outperform others, with GPT-4-0314 leading in score (8.86), win rate (1.000), and Elo rating, 2616.