Psychometric evaluation of the Ethiopian National Medical Licensing Examination: evidence from classical and item response theory analyses

BACKGROUND: Licensing examinations play a crucial role in protecting the public by establishing a minimum level of competency for safe practice. Nevertheless, empirical evidence on the psychometric quality of the National Medical Licensing Examination (NMLE) is limited. This study aimed to evaluate the effectiveness of the NMLE in assessing medical graduates in Ethiopia. METHODS: This cross-sectional study assessed the psychometric validity of the NMLE using Classical Test Theory (CTT) and the two-parameter logistic (2PL) Item Response Theory (IRT) models. We assessed the difficulty of the questions, the exam's ability to distinguish between different levels of ability, and its overall reliability. We evaluated model fit using Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and likelihood ratio tests. We conducted Differential Item Functioning (DIF) using Mantel-Haenszel, Lord's test, Raju's method, and logistic regression methods, and assessed item contributions based on factor loading and communality values. We used data from the February 2024 exam taken by 1,259 candidates across 19 schools. RESULTS: The analysis revealed that the exam demonstrated high overall reliability (α = 0.87) and correctly classified borderline candidates. However, several items didn't effectively distinguish between candidates with different levels of performance (mean = 0.18), and a few questions showed potential bias against candidates from private medical colleges. CONCLUSION: Our findings suggest that the examination was appropriate for assessing minimum competency, but it needs review to address potential bias. We recommend ongoing monitoring of item quality and enhancing faculty training to improve the quality of exam items.