Objective

2

To develop a cross-institutionally generalizable predictive model based on the theoretical test scores of Clinical Medicine Undergraduate Level Test (CMULT)

and to identify students at high risk of failing the theoretical test of National Medical Licensing Examination (NMLE) two years in advance.

Methods

2

In this retrospective study

the theoretical test scores of CMULT and NMLE of 373 undergraduate clinical medicine students (five-year program

enrolled in 2017) from JinZhou medical university served as the training set

and the corresponding scores of 126 undergraduate students enrolled in the same year from Shanghai University of Medicine & Health Sciences served as the independent external validation set. For each student

the converted score (CS) and percentile rank (PR) on the CMULT theoretical test administered at the end of the second semester of the fourth undergraduate year (May 2021)

as well as the NMLE theoretical test score obtained in the first year after graduation (August 2023)

were collected. A Spearman correlation analysis was used to assess the relationship between CS/PR and NMLE theoretical test scores. A logistic regression model was constructed to predict the risk of failure in the NMLE theoretical test. Model discrimination was evaluated using the area under the receiver operating characteristic curve (AUC)

calibration was assessed using calibration plots

and net benefit was evaluated by decision curve analysis (DCA). The optimal cutoff value for CMULT theoretical test scores was determined using the Youden index.

Results

2

Both CS and PR showed moderate positive correlations with the total NMLE theoretical test score (

r

=0.61 and 0.58

respectively

P

<

0.001). The univariable model based on CS demonstrated the best predictive performance

with an AUC of 0.797 (95%CI: 0.746-0.848) in the training set and 0.895 (95%CI: 0.819-0.971) in the independent external validation set

which indicated good cross-institutional generalizability. CS cutoff of ≤175 yielded a sensitivity of 79.55%

specificity of 84.21%

positive predictive value of 82.80%

and negative predictive value of 77.80% in predicting failure in the NMLE theoretical test. The model showed good calibration (Brier score=0.13)

and DCA demonstrated that the CS-based model provided greater net b

enefit than the "treat-all" strategy across a threshold probability range of 10%-70%.

Conclusions

2

The predictive model based solely on CS of the CMULT demonstrated robust predictive performance in independent external validation. It can effectively identify students at high risk of failing the NMLE theoretical test two years in advance

providing a simple

accessible

and scalable metric for risk stratification and targeted resource allocation in medical education.