Exploring the use of a customised chatbot on time-limited Python tasks in undergraduate physics

Abstract Generative artificial intelligence (GenAI) is transforming how novices learn to programme, yet evidence of effective and equitable use in physics education remains limited. We report an exploratory controlled study ( N = 27) in which undergraduates completed a 45 min, 16-item Python task under one of three support conditions: Internet (standard web resources), GPT (ChatGPT-3.5), or Custom-GPT (a hint-based variant configured to scaffold reasoning while avoiding runnable code). The primary outcome was percentage score; secondary outcomes were items attempted and accuracy on attempted items, complemented by a brief post-task survey, follow-up interviews, and chatbot log analysis. Mean scores were 67 ± 4%, 74 ± 5%, and 79 ± 3% for Internet , GPT , and Custom-GPT respectively. No pairwise comparison reached statistical significance after correction for multiple comparisons; however, effect size analysis revealed a large effect favouring Custom-GPT over Internet (Cliff’s δ = 0.51[−0.01, 0.91], though the confidence interval narrowly includes zero). Students attempted more questions with GPT and Custom-GPT (both 15/16) than with Internet (13/16), while accuracy on attempted items was comparable across groups. Descriptively, a novice-support pattern emerged: among students with