From Artifact to Instrument: Upper-Level University Student–GenAI Interactions During Problem Solving in an Advanced Mathematics Course

Abstract The rapid emergence of generative artificial intelligence has created new opportunities and challenges for mathematics education, particularly in advanced courses where reasoning, verification, and conceptual understanding are central. While a growing body of research has examined the capabilities, limitations, and educational implications of generative AI, less is known about how students construct their own self-imposed guardrails when using these tools for advanced mathematical problem solving. Guided by the theoretical lenses of Instrumental Genesis and Epistemic Vigilance, this qualitative study investigates how upper-level undergraduate and graduate students in a Numerical Linear Algebra course interacted with multiple GenAI systems during complex problem-solving tasks, culminating in a novel GenAI Cognitive Adaptation Framework. Data included six group project reports (totaling 21 students) and post-project survey responses from 15 participants. Qualitative thematic analysis of students’ experiences and reflections revealed three major themes across this framework. First, students developed strategic prompting practices that helped transform GenAI from a conversational artifact into a mathematical instrument. Second, students navigated trust and verification and positioned GenAI as a fallible assistant rather than an authoritative source, emphasizing verification, independent reasoning, and critical evaluation. Third, students actively navigated tool constraints , adapting their workflows to account for the utility gaps and equity divides present across free and paid platforms. By demonstrating how mathematically mature learners construct internal cognitive guardrails, this study offers a theoretically grounded account of trust, authority, and mathematical justification in AI-mediated environments, providing implications for instructional design in advanced science, technology, engineering, and mathematics (STEM) disciplines.