Reliability in Industry-Academia Assessment: A Framework for Validating Competencies in IT Education

High-stakes competency assessments involving mixed-expert panels face inherent reliability challenges. This study investigates an assessment paradox in a hackathon-style evaluation where a psychometrically sound rubric yielded unreliable outcomes. A post hoc analysis of a national IT talent program in Thailand revealed that while the assessment instrument demonstrated excellent internal consistency ($\alpha=0.912$), interrater reliability for absolute agreement was poor (ICC$=$0.276). The discrepancy was diagnosed as systemic Scale Disagreement, where experts applied divergent severity standards. To address this, a six-step protocol using Zscore standardization was applied to normalize scores. The rectified results were perceived as fair by the learners. The findings suggest that while pre-assessment rater calibration is a necessary preventive measure, a post hoc statistical rectification protocol serves as a critical safety net to ensure the validity and fairness of certification.