Supplemental Material for Applied Competence at the Bachelor’s Level: A Reliability Study of Undergraduate Trainees in Cognitive and Achievement Test Administration

Reliability EstimatesThe primary analysis estimated interrater reliability following, where possible, the methods outlined by the test publishers.The WISC-V Technical and Interpretive Manual specifies the use of intraclass correlations (ICC), while the GORT-5 reports product-moment correlations between two raters and multiple administrations.The Woodcock Johnson-IV Tests of Achievement authors provide no mention or estimate of interrater reliability.We adopted ICCs as our primary index because they are the most widely recommended for continuous or ordinal scoring of standardized test items and can be applied consistently across measures.We sought additional information as needed within their cited sources (McGraw & Wong, 1996; Shrout & Fleiss, 1979) and more recent papers (Koo and Li, 2016).We used a "Case 3" design for total score reliability [ICC(3k)] and item-level reliability [ICC(3,1)], as "each target is rated by each of the same k judges, who are the only judges of interest" (Shrout & Fleiss, 1979, p. 421).This employs a two-way, random effects ANOVA to estimate intraclass correlations.We also used Krippendorff's alpha to confirm item-level ICCs (Hayes and Krippendorff, 2007; Marzi et al., 2024) because it is robust to non-interval scaling and rater imbalance.These values were found to be nearly identical to ICC in each case, though we report both to maintain thoroughness.