Artificial intelligence-based generation of radiology test items and evaluation of assessment effectiveness through analysis of historical test item bank data

Shi Yushu; Qin Xiaoxiao; Liao Jinsheng; Huang Qiang; Lou Haiyan; Chen Shaohua; Xiao Wenbo

doi:10.3760/cma.j.cn116021-20260129-02321

您当前的位置：

首页 >

文章列表页 >

Artificial intelligence-based generation of radiology test items and evaluation of assessment effectiveness through analysis of historical test item bank data

Digital Intelligence MedicalEducation | 更新时间：2026-07-27

- Artificial intelligence-based generation of radiology test items and evaluation of assessment effectiveness through analysis of historical test item bank data
- Chinese Journal of Medical Education Research Vol. 25, Issue 7, Pages: 779-785(2026)
- 作者机构：
  
  1.浙江大学医学院附属第一医院放射科，杭州 310003
  2.浙江大学医学院附属第一医院教学部，杭州 310003
- 作者简介：
  
  Xiao Wenbo, Email: xiaowb.111@163.com
- 基金信息：
  
  Education Reform Project of Zhejiang University School of Medicine(jgzx2025010)
- DOI：10.3760/cma.j.cn116021-20260129-02321
  CLC： R-0;R-4;R445.4
- Received：29 January 2026，
  
  Online First：01 July 2026，
  
  Published：20 July 2026
- 稿件说明：
移动端阅览
史玉书, 秦潇潇, 廖金生, 等. 基于放射学历史题库数据分析的人工智能试题生成及考核效果评价[J]. 中华医学教育探索杂志, 2026, 25(7): 779-785.

Shi Yushu, Qin Xiaoxiao, Liao Jinsheng, et al. Artificial intelligence-based generation of radiology test items and evaluation of assessment effectiveness through analysis of historical test item bank data[J]. Chinese Journal of Medical Education Research, 2026, 25(7): 779-785.
史玉书, 秦潇潇, 廖金生, 等. 基于放射学历史题库数据分析的人工智能试题生成及考核效果评价[J]. 中华医学教育探索杂志, 2026, 25(7): 779-785. DOI： 10.3760/cma.j.cn116021-20260129-02321.

Shi Yushu, Qin Xiaoxiao, Liao Jinsheng, et al. Artificial intelligence-based generation of radiology test items and evaluation of assessment effectiveness through analysis of historical test item bank data[J]. Chinese Journal of Medical Education Research, 2026, 25(7): 779-785. DOI： 10.3760/cma.j.cn116021-20260129-02321.

摘要

目的

探讨依托放射科历史题库数据的大语言模型在医学见习阶段放射影像试题智能生成中的可行性，评估其对试题难度与区分度等核心测量学参数的预测效能及实测一致性，为智能化医学命题提供实证依据。

方法

采用定量实验设计，从放射科历史题库中按学科标准抽取50道经典试题作为提示基准，将其输入通义千问（Qwen-Plus）大语言模型生成新试题，并预测其难度与区分度参数。组织56名临床医学四年级见习生进行实测，采用非参数检验、Spearman秩相关及组内相关系数（intraclass correlation coefficient，ICC）等进行统计学分析。

结果

AI生成试题难度预测总体误差较小[平均绝对误差（mean absolute error，MAE）=0.052，均方根误差（root mean square error，RMSE）=0.062

]

，区分度预测误差略大（MAE=0.105，RMSE=0.131）。AI预测值与实测值呈高度正相关（难度

=0.896，区分度

=0.791，均

0.001）。一致性检验显示，难度参数总体ICC=0.908（优秀），区分度参数总体ICC=0.801（良好）。分层分析表明，AI对容易题难度存在轻微高估，对中档题的区分度预测略偏乐观，但实测参数整体成功维持了预设的难度梯度与鉴别力分布特征，干扰项临床贴合度良好。

结论

基于大语言模型的AI试题生成技术对放射影像见习试题的难度系数和区分度预测具有良好的一致性。依托历史题库实测参数进行交叉验证，AI辅助命题具备高度的可行性与教学适用性。建议构建“AI生成—历史数据校准—专家复核”的人机协同命题新模式，以推动医学影像学考核向标准化、精准化与智能化转型。

Abstract

Objective

To evaluate the feasibility of a large language model (LLM) based on historical radiology test item bank data for the intelligent generation of radiological imaging test items for the medical clerkship stage

assess its predictive performance and agreement between predicted and empirical data for core psychometric parameters such as difficulty and discrimination

and provide empirical evidence for intelligent medical test item development.

Methods

A quantitative experimental design was adopted. Fifty classic items were extracted from a historical radiology test item bank according to subject-specific standards as baseline prompts

and were input into the Qwen-Plus LLM to generate new items and predict their difficulty and discrimination parameters. Fifty-six fourth-year clinical medicine clerkship students completed the generated test. Statistical analyses included non-parametric tests

Spearman's rank correlation

and intraclass correlation coefficient (ICC).

Results

The predicted difficulty of artificial intelligence (AI)-generated test items showed a relatively small overall error (mean absolute error=0.052

root mean square error=0.062)

while the predicted discrimination showed a relatively large error (mean absolute error=0.105

root mean square error=0.131). AI-predicted values were highly positively correlated with empirical values (difficulty

=0.896

discrimination

=0.791

both

0.001). ICC analysis indicated excellent agreement for difficulty (ICC=0.908) and good agreement for discrimination (ICC=0.801). Stratified analysis revealed a slight overestimation of difficulty for easy items and a slightly optimistic prediction of discrimination for medium-difficulty items. However

the empirical pa

rameters overall successfully maintained the preset difficulty gradient and discrimination distribution characteristics

with distractors demonstrating high clinical plausibility.

Conclusions

LLM-based item generation demonstrates good agreement between predicted and empirical values for the difficulty coefficient and discrimination of radiology clerkship test items. In cross-validation against the empirical data from a historical radiology test item bank

AI-assisted test item development is highly feasible and pedagogically applicable. It is recommended to construct a new human-AI collaborative test item development model of "AI generation

historical data calibration

and expert review" to advance the transformation of medical imaging assessment toward standardization

precision

and intelligence.

关键词

Keywords

references

陈欣，戴社教，赵暹，等 . 不同类型试题对医学影像学考试评价的效能分析 [J ] . 中华医学教育探索杂志， 2012 ， 11 （ 11 ）： 1160 - 1163 . DOI: 10.3760/cma.j.issn.2095-1485.2012.11.022 http://dx.doi.org/10.3760/cma.j.issn.2095-1485.2012.11.022 .

刘朋，张洁，杨大为，等 . 临床医学专业本科生影像学教学现状与改革 [J ] . 医学教育管理， 2023 ， 9 （增刊1 ）： 96 - 99 . DOI: 10.3969/j.issn.2096-045X.2023.S1.023 http://dx.doi.org/10.3969/j.issn.2096-045X.2023.S1.023 .

曲波 . 推动AI与教育深度融合构建精准学习新路径 [J ] . 中华医学教育探索杂志， 2025 ， 24 （ 10 ）： 1297 - 1300 . DOI: 10.3760/cma.j.cn116021-20250326-02133 http://dx.doi.org/10.3760/cma.j.cn116021-20250326-02133 .

徐德林，史军华，姚秋丽，等 . 医学院校试卷命题的问题分析与解决策略 [J ] . 遵义师范学院学报， 2023 ， 25 （ 3 ）： 121 - 123 .

Aster A , Laupichler MC , Rockwell-Kollmann T , et al . ChatGPT and other large language models in medical education: scoping literature review [J ] . Med Sci Educ , 2025 , 35 ( 1 ): 555 - 567 . DOI: 10.1007/s40670-024-02206-6 http://dx.doi.org/10.1007/s40670-024-02206-6 .

国家医学考试中心 . 国家医学考试中心创新性研究目录（2025—2026年度） [R ] . 北京：国家医学考试中心， 2025 .

魏千，郝瑞 . 基于AI互评的中国眼科住院医师规范化培训大语言模型应用现状分析 [J ] . 中华医学教育探索杂志， 2025 ， 24 （ 12 ）： 1585 - 1591 . DOI: 10.3760/cma.j.cn116021-20250710-02166 http://dx.doi.org/10.3760/cma.j.cn116021-20250710-02166 .

王斌，田士来，王小成，等 . 以ChatGPT为代表的人工智能技术在医学教育中的机遇与挑战 [J ] . 医学教育研究与实践， 2023 ， 31 （ 4 ）： 402 - 405 . DOI: 10.13555/j.cnki.c.m.e.2023.04.002 http://dx.doi.org/10.13555/j.cnki.c.m.e.2023.04.002 .

平措玉珍，兰亚佳，马景璇，等 . 基于经典测量理论和项目反应理论的工作内容量表（JCQ-22）评价与优化 [J ] . 现代预防医学， 2025 ， 52 （ 17 ）： 3220 - 3226 . DOI: 10.20043/j.cnki.MPM.202503385 http://dx.doi.org/10.20043/j.cnki.MPM.202503385 .

Koo TK , Li MY . A guideline of selecting and reporting intraclass correlation coefficients for reliability research [J ] . J Chiropr Med , 2016 , 15 ( 2 ): 155 - 163 . DOI: 10.1016/j.jcm.2016.02.012 http://dx.doi.org/10.1016/j.jcm.2016.02.012 .

黄广仕，周梦强，韩春梅，等 . 人工智能深度学习技术在医学考试试题难度预估中的应用研究 [J ] . 中华医学教育杂志， 2021 ， 41 （ 10 ）： 932 - 935 . DOI: 10.3760/cma.j.cn115259-20201203-01668 http://dx.doi.org/10.3760/cma.j.cn115259-20201203-01668 .

Staziaki PV , Sarangi R , Parikh UN , et al . An objective structured clinical examination for medical student radiology clerkships: reproducibility study [J ] . JMIR Med Educ , 2020 , 6 ( 1 ): e15444 . DOI: 10.2196/15444 http://dx.doi.org/10.2196/15444 .

Rezaei-Zadeh M , Cerbin-Koczorowska M . A research roadmap for AI opportunities in student assessment for medical education [J ] . BMC Med Educ , 2025 , 25 ( 1 ): 1653 . DOI: 10.1186/s12909-025-08078-7 http://dx.doi.org/10.1186/s12909-025-08078-7 .

Zhang JX , Zuo CS . GRPO-LEAD: a difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models [EB/OL ] . [ 2026-01-05 ] . https://arxiv.org/abs/2504.09696 https://arxiv.org/abs/2504.09696 .

Pereira DSM , Mourão F , Ribeiro JC , et al . ChatGPT as an item calibration tool: psychometric insights in a high-stakes examination [J ] . Med Teach , 2025 , 47 ( 4 ): 677 - 683 . DOI: 10.1080/0142159X.2024.2376205 http://dx.doi.org/10.1080/0142159X.2024.2376205 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Differences in the application of artificial intelligence tools in domestic and overseas scenarios among medical staff and their implications

Survey and reflections on resident perceptions regarding the clinical applications of AI

Research on innovation of tumor microenvironment course and practice of building an online course cluster

Practice and exploration of computer-based case simulations in pediatric clerkship teaching for eight-year clinical medicine program students

Trends in learning gains of intern nurses in obstetrics and gynecology and their correlations with learning beliefs and academic emotions

Related Author

Hu Xin

Chen Shaohua

Shi Beiyi

Deng Tianyi

Zhou Qing

Tang Dong

Qi Jiajun

Weng Xiaohong

Related Institution

Department of International Affairs, The First Affiliated Hospital, Zhejiang University School of Medicine

Department of Medical Education, The First Affiliated Hospital, Zhejiang University School of Medicine

Department of Education and Training, Affiliated Hospital of Nantong University

Department of Education and Training, The First Affiliated Hospital of Soochow University

Department of Education, Suzhou Hospital Affiliated to Nanjing Medical University

⁰