Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology

Zhong Jingyu; Xing Yue; Hu Yangfan; Min Qinghua; Zhu Caisong; Shi Dandan; Fan Xiaoyu; Chu Jingshen; Zhang Huan; Yao Weiwu

doi:10.3760/cma.j.cn116021-20231111-01961

您当前的位置：

首页 >

文章列表页 >

Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology

Digital Intelligence MedicalEducation | 更新时间：2025-02-21

- Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology
- Chinese Journal of Medical Education Research Vol. 24, Issue 2, Pages: 145-149(2025)
- 作者机构：
  
  1.上海交通大学医学院附属同仁医院影像科，上海　200336
  2.上海交通大学医学院附属同仁医院教学科，上海　200336
  3.上海交通大学医学院附属同仁医院院长办公室，上海　200336
  4.上海交通大学医学院附属瑞金医院《诊断学理论与实践》编辑部，上海　200025
  5.上海交通大学医学院附属瑞金医院放射科，上海　200025
- 作者简介：
  
  Yao Weiwu, Email: yaoweiwuhuan@163.com
- 基金信息：
  
  2024 "Generative Artificial Intelligence + Education" Fund of Shanghai Jiao Tong University(CTLD24A0021);2024 Autumn "Artificial Intelligence + Education" Fund from Shanghai Jiao Tong University(CTLD24A0081);"Digital Intelligence Courses" Construction Project from Shanghai Jiao Tong University [Medical Imaging (For Preventive Medicine)];2024 Undergraduate Core Courses Construction Project from Shanghai Jiao Tong University School of Medicine [Medical Imaging (For Preventive Medicine)];2023 Undergraduate Textbook Construction Project from Shanghai Jiao Tong University School of Medicine [Medical Imaging (For Preventive Medicine)];Yingxing Project of Shanghai Tongren Hospital(TRYXJH28)
- DOI：10.3760/cma.j.cn116021-20231111-01961
  CLC： R445
- Received：11 November 2023，
  
  Published：20 February 2025
- 稿件说明：
移动端阅览
钟京谕, 星月, 胡扬帆, 等. 中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量[J]. 中华医学教育探索杂志, 2025, 24(2): 145-149.

Zhong Jingyu, Xing Yue, Hu Yangfan, et al. Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology[J]. Chinese journal of medical education research, 2025, 24(2): 145-149.
钟京谕, 星月, 胡扬帆, 等. 中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量[J]. 中华医学教育探索杂志, 2025, 24(2): 145-149. DOI： 10.3760/cma.j.cn116021-20231111-01961.

Zhong Jingyu, Xing Yue, Hu Yangfan, et al. Accuracy and quality of answer reasoning of Chinese large language model in Chinese middle level professional qualification examination of radiology[J]. Chinese journal of medical education research, 2025, 24(2): 145-149. DOI： 10.3760/cma.j.cn116021-20231111-01961.

摘要

目的

对比中文大语言模型（large language model，LLM）和医师在放射医学中级专业技术考试中的正确率，评估中文LLM提供的答案解析质量。

方法

本研究分层随机抽取100题精选习题组成测试用题，网页访问文心一言要求其给出正确答案及解析，邀请15名不同经验的放射科医师参与测试。比较中文LLM及医师的答题正确率，2名医师使用5分制的半定量方法评估答案解析质量。

结果

文心一言的正确率为60.00%，低于医师的正确率［67.00%（64.00%，73.00%）］，两者之间的差异具有统计学意义（

=2.47，

=0.013）。文心一言提供的答案解析的字数为（196.44±99.25）字，正确和错误的答案解析的字数分别为（211.03±107.53）字和（174.55±81.84）字，差异无统计学意义（

=1.82，

=0.072）。正确答案的解析在质量评估中获得1~4分的试题数量分别为3题、9题、12题和36题，无获得5分的答案解析。

结论

中文LLM具备一定的医学知识和临床推理能力，能够辅助带教医师开展教学，但尚不能单独辅导住院医师学习，缺乏引导式、启发式教学的能力。

Abstract

Objective

To compare the accuracy of a Chinese large language model (LLM) and radiologists in Chinese middle level professional qualification examination of radiology

and evaluate the quality of answer reasoning provided by the Chinese LLM.

Methods

In this study

100 high-quality questions were selected using stratified random sampling to form a test set. We asked the ERNIE Bot by dialogues on the website to provide the correct answers and answer reasoning for these questions. These questions were also answered by 15 radiologists with different levels of experience. The accuracy of Chinese LLM and that of radiologists were compared. Two radiologists evaluated the quality of answer reasoning using a 5-point semi-quantitative scale.

Results

The accuracy of ERNIE Bot was 60.00%

which was lower than the median (interquartile) accuracy of 67.00% (64.00%

73.00%) for radiologists

and the difference was statistically significant (

=2.47

=0.013). The word count of the reasoning provided by Ernie Bot was (196.44±99.25) words

with no significant difference in word count between correct and incorrect answer reasoning

which were (211.03±107.53) words and (174.55±81.84) words

respectively (

=1.82

=0.072). Among the correct answers

the quality of reasoning was scored as follows: 1 point for 3 questions

2 points for 9 questions

3 points for 12 questions

and 4 points for 36 questions. No reasoning received a score of 5.

Conclusions

Chinese LLM demonstrates a certain level of medical knowledge and clinical reasoning ability

which can assist clinical teachers in educational activities. However

it is not yet able to independently tutor residents and lacks the ability of invitational and heuristic teaching.

关键词

Keywords

references

Khurana D , Koli A , Khatter K , et al . Natural language processing: state of the art, current trends and challenges [J ] . Multimed Tools Appl , 2023 , 82 ( 3 ): 3713 - 3744 . DOI: 10.1007/s11042-022-13428-4 http://dx.doi.org/10.1007/s11042-022-13428-4 .

Thirunavukarasu AJ , Ting DSJ , Elangovan K , et al . Large language models in medicine [J ] . Nat Med , 2023 , 29 ( 8 ): 1930 - 1940 . DOI: 10.1038/s41591-023-02448-8 http://dx.doi.org/10.1038/s41591-023-02448-8 .

Gilson A , Safranek CW , Huang T , et al . Correction: how does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment [J ] . JMIR Med Educ , 2024 , 10 : e57594 . DOI: 10.2196/57594 http://dx.doi.org/10.2196/57594 .

Kung TH , Cheatham M , Medenilla A , et al . Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models [J ] . PLoS Digit Health , 2023 , 2 ( 2 ): e0000198 . DOI: 10.1371/journal.pdig.0000198 http://dx.doi.org/10.1371/journal.pdig.0000198 .

St Huang R , Lu KJQ , Meaney C , et al . Assessment of resident and AI chatbot performance on the university of Toronto family medicine residency progress test: comparative study [J ] . JMIR Med Educ , 2023 , 9 : e50514 . DOI: 10.2196/50514 http://dx.doi.org/10.2196/50514 .

Bhayana R , Krishna S , Bleakney RR . Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations [J ] . Radiology , 2023 , 307 ( 5 ): e230582 . DOI: 10.1148/radiol.230582 http://dx.doi.org/10.1148/radiol.230582 .

Huh S . Are ChatGPT's knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study [J ] . J Educ Eval Health Prof , 2023 , 20 : 1 . DOI: 10.3352/jeehp.2023.20.1 http://dx.doi.org/10.3352/jeehp.2023.20.1 .

Takagi S , Watari T , Erabi A , et al . Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study [J ] . JMIR Med Educ , 2023 , 9 : e48002 . DOI: 10.2196/48002 http://dx.doi.org/10.2196/48002 .

Wang X , Gong Z , Wang G , et al . ChatGPT performs on the Chinese National Medical Licensing Examination [EB/OL ] . ( 2023-02-16 ) [ 2023-10-01 ] . https://www.researchsquare.com/article/rs-2584079/v1 https://www.researchsquare.com/article/rs-2584079/v1 .

贺樑，应振宇，王英英，等 . 教育中的ChatGPT：教学能力诊断研究 [J ] . 华东师范大学学报（教育科学版）， 2023 ， 41 （ 7 ）： 162- 176. DOI: 10.16382/j.cnki.1000-5560.2023.07.015 http://dx.doi.org/10.16382/j.cnki.1000-5560.2023.07.015 .

刘斯润，张水兴 . 2021全国卫生专业资格技术考试题集丛书：放射医学精选习题集 [M ] . 北京：人民卫生出版社， 2020 .

百度 . 文心一言 [EB/OL ] . ( 2023-08-31 ) [ 2023-09-27 ] . https://yiyan.baidu.com https://yiyan.baidu.com .

张华平，李林翰，李春锦 . ChatGPT中文性能测评与风险应对 [J ] . 数据分析与知识发现， 2023 ， 7 （ 3 ）： 16 - 25 . DOI: 10.11925/infotech.2096-3467.2023.0214 http://dx.doi.org/10.11925/infotech.2096-3467.2023.0214 .

Lyu Q , Tan J , Zapadka ME , et al . Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential [J ] . Vis Comput Ind Biomed Art , 2023 , 6 ( 1 ): 9 . DOI: 10.1186/s42492-023-00136-5 http://dx.doi.org/10.1186/s42492-023-00136-5 .

Stoehr F , Kämpgen B , Müller L , et al . Natural language processing for automatic evaluation of free-text answers-a feasibility study based on the European Diploma in radiology examination [J ] . Insights Imaging , 2023 , 14 ( 1 ): 150 . DOI: 10.1186/s13244-023-01507-5 http://dx.doi.org/10.1186/s13244-023-01507-5 .

陈勇，张欢 . 线上自适应教程在医学影像学专业型硕士研究生教学实践中的应用 [J ] . 诊断学理论与实践， 2021 ， 20 （ 6 ）： 596- 598. DOI: 10.16150/j.1671-2870.2021.06.017 http://dx.doi.org/10.16150/j.1671-2870.2021.06.017 .

倪闽景 . 从学习进化的视角看ChatGPT/生成式人工智能对学习的影响 [J ] . 华东师范大学学报（教育科学版）， 2023 ， 41 （ 7 ）： 151 - 161 . DOI: 10.16382/j.cnki.1000-5560.2023.07.014 http://dx.doi.org/10.16382/j.cnki.1000-5560.2023.07.014 .

李袁爽，张子航，王运武 . 基于ChatGPT的人机共教、共育和共创模式 [J ] . 中国医学教育技术， 2023 ， 37 （ 4 ）： 383 - 389， 396 . DOI: 10.13566/j.cnki.cmet.cn61-1317/g4.202304002 http://dx.doi.org/10.13566/j.cnki.cmet.cn61-1317/g4.202304002 .

袁欢欢，全祉悦，林祎，等 . 当医学课程考核遇上生成式人工智能：价值、挑战与未来展望 [J ] . 中国医学教育技术， 2023 ， 37 （ 3 ）： 254 - 259 . DOI: 10.13566/j.cnki.cmet.cn61-1317/g4.202303002 http://dx.doi.org/10.13566/j.cnki.cmet.cn61-1317/g4.202303002 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Application of blended teaching based on teaching case library in standardized residency training in department of radiology

Application of Eurorad database in the teaching of radiology in standardized residency training: A case study of teaching film-reading

Analysis of the results of clinical skills completion assessment for standardized residency training of radiology in Liaoning province in 2020

Evaluating the effectiveness of a training program for faculty supervisory specialists in standardized residency training based on adult learning theory

Competency evaluation of dermatology physicians receiving residency training based on entrustable professional activities

Related Author

Cao Minghui

Wang Dongye

Mao Jiaji

Zhang Xiang

Zheng Chushan

Shi Guangzi

Cui Haoyuan

Shen Jun

Related Institution

Department of Radiology, Sun Yat-sen Memorial Hospital, Sun Yat-sen University

中国医科大学附属盛京医院放射科

Department of Continuing Education, The Third Affiliated Hospital of Sun Yat-sen University

Department of Endocrinology, The Third Affiliated Hospital of Sun Yat-sen University

Clinical Skills Training Center, The Third Affiliated Hospital of Sun Yat-sen University

⁰