浏览全部资源
扫码关注微信
中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量
Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology
- 中华医学教育探索杂志 2025年24卷第2期 页码：145-149
- 作者机构：
1.上海交通大学医学院附属同仁医院影像科，上海 200336
2.上海交通大学医学院附属同仁医院教学科，上海 200336
3.上海交通大学医学院附属同仁医院院长办公室，上海 200336
4.上海交通大学医学院附属瑞金医院《诊断学理论与实践》编辑部，上海 200025
5.上海交通大学医学院附属瑞金医院放射科，上海 200025
- 作者简介：
姚伟武，Email：yaoweiwuhuan@163.com
- 基金信息：上海交通大学2024年“生成式人工智能+教育”专项基金(CTLD24A0021);2024年秋上海交通大学“人工智能+教育”专项基金(CTLD24A0081);上海交通大学“数智课程”建设项目［医学影像学（预防医学专用）］;上海交通大学医学院2024年本科核心课程建设项目［医学影像学（预防医学专业）］;上海交通大学医学院2023年度本科教材建设项目［医学影像学（供预防医学专业用）］;上海市同仁医院萤星计划(TRYXJH28)
DOI：10.3760/cma.j.cn116021-20231111-01961
中图分类号： R445收稿日期：2023-11-11，
纸质出版日期：2025-02-20
- 稿件说明：
移动端阅览
钟京谕, 星月, 胡扬帆, 等. 中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量[J]. 中华医学教育探索杂志, 2025, 24(2): 145-149.
Zhong Jingyu, Xing Yue, Hu Yangfan, et al. Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology[J]. Chinese journal of medical education research, 2025, 24(2): 145-149.
钟京谕, 星月, 胡扬帆, 等. 中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量[J]. 中华医学教育探索杂志, 2025, 24(2): 145-149. DOI： 10.3760/cma.j.cn116021-20231111-01961.
Zhong Jingyu, Xing Yue, Hu Yangfan, et al. Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology[J]. Chinese journal of medical education research, 2025, 24(2): 145-149. DOI： 10.3760/cma.j.cn116021-20231111-01961.
摘要
目的
2
对比中文大语言模型（large language model，LLM）和医师在放射医学中级专业技术考试中的正确率，评估中文LLM提供的答案解析质量。
方法
2
本研究分层随机抽取100题精选习题组成测试用题，网页访问文心一言要求其给出正确答案及解析，邀请15名不同经验的放射科医师参与测试。比较中文LLM及医师的答题正确率，2名医师使用5分制的半定量方法评估答案解析质量。
结果
2
文心一言的正确率为60.00%，低于医师的正确率［67.00%（64.00%，73.00%）］，两者之间的差异具有统计学意义（
W
=2.47，
P
=0.013）。文心一言提供的答案解析的字数为（196.44±99.25）字，正确和错误的答案解析的字数分别为（211.03±107.53）字和（174.55±81.84）字，差异无统计学意义（
t
=1.82，
P
=0.072）。正确答案的解析在质量评估中获得1~4分的试题数量分别为3题、9题、12题和36题，无获得5分的答案解析。
结论
2
中文LLM具备一定的医学知识和临床推理能力，能够辅助带教医师开展教学，但尚不能单独辅导住院医师学习，缺乏引导式、启发式教学的能力。
Abstract
Objective
2
To compare the accuracy of a Chinese large language model (LLM) and radiologists in Chinese middle level professional qualification examination of radiology
and evaluate the quality of answer reasoning provided by the Chinese LLM.
Methods
2
In this study
100 high-quality questions were selected using stratified random sampling to form a test set. We asked the ERNIE Bot by dialogues on the website to provide the correct answers and answer reasoning for these questions. These questions were also answered by 15 radiologists with different levels of experience. The accuracy of Chinese LLM and that of radiologists were compared. Two radiologists evaluated the quality of answer reasoning using a 5-point semi-quantitative scale.
Results
2
The accuracy of ERNIE Bot was 60.00%
which was lower than the median (interquartile) accuracy of 67.00% (64.00%
73.00%) for radiologists
and the difference was statistically significant (
W
=2.47
P
=0.013). The word count of the reasoning provided by Ernie Bot was (196.44±99.25) words
with no significant difference in word count between correct and incorrect answer reasoning
which were (211.03±107.53) words and (174.55±81.84) words
respectively (
t
=1.82
P
=0.072). Among the correct answers
the quality of reasoning was scored as follows: 1 point for 3 questions
2 points for 9 questions
3 points for 12 questions
and 4 points for 36 questions. No reasoning received a score of 5.
Conclusions
2
Chinese LLM demonstrates a certain level of medical knowledge and clinical reasoning ability
which can assist clinical teachers in educational activities. However
it is not yet able to independently tutor residents and lacks the ability of invitational and heuristic teaching.
关键词
Keywords
references
Khurana D , Koli A , Khatter K , et al . Natural language processing: state of the art, current trends and challenges [J ] . Multimed Tools Appl , 2023 , 82 ( 3 ): 3713 - 3744 . DOI: 10.1007/s11042-022-13428-4 http://dx.doi.org/10.1007/s11042-022-13428-4 .
Thirunavukarasu AJ , Ting DSJ , Elangovan K , et al . Large language models in medicine [J ] . Nat Med , 2023 , 29 ( 8 ): 1930 - 1940 . DOI: 10.1038/s41591-023-02448-8 http://dx.doi.org/10.1038/s41591-023-02448-8 .
Gilson A , Safranek CW , Huang T , et al . Correction: how does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment [J ] . JMIR Med Educ , 2024 , 10 : e57594 . DOI: 10.2196/57594 http://dx.doi.org/10.2196/57594 .
Kung TH , Cheatham M , Medenilla A , et al . Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models [J ] . PLoS Digit Health , 2023 , 2 ( 2 ): e0000198 . DOI: 10.1371/journal.pdig.0000198 http://dx.doi.org/10.1371/journal.pdig.0000198 .
St Huang R , Lu KJQ , Meaney C , et al . Assessment of resident and AI chatbot performance on the university of Toronto family medicine residency progress test: comparative study [J ] . JMIR Med Educ , 2023 , 9 : e50514 . DOI: 10.2196/50514 http://dx.doi.org/10.2196/50514 .
Bhayana R , Krishna S , Bleakney RR . Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations [J ] . Radiology , 2023 , 307 ( 5 ): e230582 . DOI: 10.1148/radiol.230582 http://dx.doi.org/10.1148/radiol.230582 .
Huh S . Are ChatGPT's knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study [J ] . J Educ Eval Health Prof , 2023 , 20 : 1 . DOI: 10.3352/jeehp.2023.20.1 http://dx.doi.org/10.3352/jeehp.2023.20.1 .
Takagi S , Watari T , Erabi A , et al . Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study [J ] . JMIR Med Educ , 2023 , 9 : e48002 . DOI: 10.2196/48002 http://dx.doi.org/10.2196/48002 .
Wang X , Gong Z , Wang G , et al . ChatGPT performs on the Chinese National Medical Licensing Examination [EB/OL ] . ( 2023-02-16 ) [ 2023-10-01 ] . https://www.researchsquare.com/article/rs-2584079/v1 https://www.researchsquare.com/article/rs-2584079/v1 .
贺樑 ， 应振宇 ， 王英英 ， 等 . 教育中的ChatGPT： 教学能力诊断研究 [J ] . 华东师范大学学报（教育科学版） ， 2023 ， 41 （ 7 ）： 162- 176. DOI: 10.16382/j.cnki.1000-5560.2023.07.015 http://dx.doi.org/10.16382/j.cnki.1000-5560.2023.07.015 .
刘斯润 ， 张水兴 . 2021全国卫生专业资格技术考试题集丛书： 放射医学精选习题集 [M ] . 北京 ： 人民卫生出版社 ， 2020 .
百度 . 文心一言 [EB/OL ] . ( 2023-08-31 ) [ 2023-09-27 ] . https://yiyan.baidu.com https://yiyan.baidu.com .
张华平 ， 李林翰 ， 李春锦 . ChatGPT中文性能测评与风险应对 [J ] . 数据分析与知识发现 ， 2023 ， 7 （ 3 ）： 16 - 25 . DOI: 10.11925/infotech.2096-3467.2023.0214 http://dx.doi.org/10.11925/infotech.2096-3467.2023.0214 .
Lyu Q , Tan J , Zapadka ME , et al . Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential [J ] . Vis Comput Ind Biomed Art , 2023 , 6 ( 1 ): 9 . DOI: 10.1186/s42492-023-00136-5 http://dx.doi.org/10.1186/s42492-023-00136-5 .
Stoehr F , Kämpgen B , Müller L , et al . Natural language processing for automatic evaluation of free-text answers-a feasibility study based on the European Diploma in radiology examination [J ] . Insights Imaging , 2023 , 14 ( 1 ): 150 . DOI: 10.1186/s13244-023-01507-5 http://dx.doi.org/10.1186/s13244-023-01507-5 .
陈勇 ， 张欢 . 线上自适应教程在医学影像学专业型硕士研究生教学实践中的应用 [J ] . 诊断学理论与实践 ， 2021 ， 20 （ 6 ）： 596- 598. DOI: 10.16150/j.1671-2870.2021.06.017 http://dx.doi.org/10.16150/j.1671-2870.2021.06.017 .
倪闽景 . 从学习进化的视角看ChatGPT/生成式人工智能对学习的影响 [J ] . 华东师范大学学报（教育科学版） ， 2023 ， 41 （ 7 ）： 151 - 161 . DOI: 10.16382/j.cnki.1000-5560.2023.07.014 http://dx.doi.org/10.16382/j.cnki.1000-5560.2023.07.014 .
李袁爽 ， 张子航 ， 王运武 . 基于ChatGPT的人机共教、共育和共创模式 [J ] . 中国医学教育技术 ， 2023 ， 37 （ 4 ）： 383 - 389， 396 . DOI: 10.13566/j.cnki.cmet.cn61-1317/g4.202304002 http://dx.doi.org/10.13566/j.cnki.cmet.cn61-1317/g4.202304002 .
袁欢欢 ， 全祉悦 ， 林祎 ， 等 . 当医学课程考核遇上生成式人工智能： 价值、挑战与未来展望 [J ] . 中国医学教育技术 ， 2023 ， 37 （ 3 ）： 254 - 259 . DOI: 10.13566/j.cnki.cmet.cn61-1317/g4.202303002 http://dx.doi.org/10.13566/j.cnki.cmet.cn61-1317/g4.202303002 .
0
浏览量
0
下载量
0
CSCD
- 网络PDF下载
- WORD下载
- Meta-XML下载
- BibTeX下载
- Endnote下载
- RefMan 下载
- NoteExpress下载
- NoteFirst下载
关联资源
相关文章
相关作者
相关机构
- 技术支持由北京北大方正电子有限公司提供 京ICP备09064830号-19
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies
