Engineering Students' Critical Engagement With ChatGPT: Effects of Output Accuracy on Answers and Confidence in a Real-World Probability Context

Contribution:This study investigates how the accuracy of artificial intelligence (AI)-generated explanations affects electrical engineering students’ solution correctness, confidence, inquiry behavior, and reasons for accepting or rejecting ChatGPT’s answers when solving a real-world probability problem grounded in Bayes’ theorem and susceptible to base-rate neglect. Unlike prior controlled or guided research, it captures authentic, unstructured student–AI interactions in genuine learning contexts.Background:Although students are generally aware that large language models can produce errors, little is known about their ability to recognize such errors or the extent to which ChatGPT’s feedback shapes their confidence during problem-solving in real-world contexts.Research Questions:This study explores: 1) how engineering students solve a real-life problem with ChatGPT shaping their accuracy; 2) how using ChatGPT affects students’ confidence in their own answers; and 3) what types of questions students ask ChatGPT and why they accept or reject its responses.Methodology:Using a within-subjects design, N = 107 students solved a real-world task, first unaided and then with ChatGPT. We analyzed ChatGPT’s accuracy, students’ solution accuracy, written explanations, self-reported confidence, interaction logs, and the reasons students accepted or rejected ChatGPT’s answers.Findings:Unaided accuracy was 14%. With ChatGPT, accuracy rose to 62% when its guidance was correct but dropped to 6% when it was incorrect. Few students used verification-oriented prompts, and many accepted plausible but incorrect answers without checking. Exposure to erroneous outputs increased confidence in wrong solutions, revealing a low level of critical engagement and highlighting the importance of integrating AI literacy with probability instruction.