Do the University Clinical Aptitude Test (UCAT) scores offer incremental predictive validity for doctors’ performance in UK post-qualification practical clinical examinations? A protocol for an observational retrospective cohort study.

Abstract. Introduction: Internationally, medical schools increasingly use standardised selection assessments to select applicants. The University Clinical Aptitude Test (UCAT) is the most popular standardised medical selection assessment across Europe and Australasia. The UCAT consists of cognitive assessments and a situational judgement test. The UCAT scores have been shown to offer incremental predictive validity for medical school academic attainment: they add predictive value beyond that already provided by prior educational attainment, such as secondary school grades. However, the use of standardised medical selection assessments imposes burdens on applicants and institutions. Therefore, to justify their widespread adoption, scores from these assessments should predict doctors' future clinical competency. This can be evaluated by their performance in post-qualification practical clinical examinations. Hence, this study aims to evaluate whether UCAT scores offer incremental predictive validity for doctors’ performance in UK post-qualification practical clinical examinations. Methods and Analysis: This observational retrospective cohort study will use data from the United Kingdom Medical Education Database (UKMED). The UKMED is a national database containing pseudo-anonymised routine data on UK doctors. The population included will be those who attempted the UCAT as part of their medical school application, and subsequently attempted UK post-qualification practical clinical examinations. The predictors are the UCAT total and subtest scores. The primary outcomes are performance in the UK post-qualification practical clinical examinations, such as the Membership of the Royal College of Physicians Practical Assessment of Clinical Examination Skills (MRCP PACES). Descriptive statistics will summarise the study data. Univariable logistic regression will estimate the probability of passing post-qualification practical clinical examinations, as a binary categorical outcome variable (pass versus fail), based on UCAT scores. Multivariable logistic regression will estimate whether UCAT scores offer incremental predictive validity for passing post-qualification practical clinical examinations, by adjusting for the predictive validity offered by prior educational attainment. Similarly, univariable and multivariable linear regression will estimate the predictive validity of UCAT scores for performance in post-qualification practical clinical examination scores as a continuous interval variable. LASSO penalised regression will be used to create parsimonious multivariable models to prevent overfitting. Path analyses will be developed and tested to explore how the educational predictors relate to each other and the outcomes of interest. Missing data will be addressed using multiple imputation by chained equations. Ethics and Dissemination: As this is secondary analysis of pseudo-anonymised data, ethical approval will not be required. Results will be published in a peer-reviewed journal, presented at academic conferences, and shared with relevant policy bodies.