Evidence of Validity for the Artificial Intelligence Competence and Literacy Test (CAIA) in Spanish University Students

This study presents the development and psychometric validation of the Artificial Intelligence Competence and Literacy Test (CAIA), designed to assess artificial intelligence (AI) literacy and competence in higher education contexts. As AI becomes increasingly integrated into academic and professional environments, reliable instruments are needed to evaluate individuals’ conceptual understanding, ethical awareness, and applied competencies related to AI. The instrument was administered to a sample of 510 university students from several faculties at a Spanish university. Exploratory and confirmatory factor analyses were conducted using a cross-validation design. Results supported a multidimensional structure, in the final 18-item version of the instrument, composed of two correlated factors (critical—conceptual AI literacy and creative—applied AI competence) and a second-order hierarchical model representing a global CAIA score. Model fit indices were acceptable-to-good, and reliability estimates, including ordinal coefficients and measurement error indicators, showed adequate precision for both individual and group-level interpretation. Evidence of construct validity was further supported through convergent and discriminant analyses, as well as hypothesis testing across academic subgroups. The findings suggest that the CAIA provides a theoretically grounded and psychometrically robust instrument for assessing AI-related competencies in higher education. The instrument may support research, curriculum design, and evaluation of educational initiatives aimed at promoting informed, critical, and responsible engagement with artificial intelligence in digitally mediated learning environments.