Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment

Li Li; Liu Jihai; Xie Yadian; Shi Di; Hu Xiaoyun; Chen Zhiqiao; An Haiyan

doi:10.3760/cma.j.cn116021-20230503-02099

您当前的位置：

首页 >

文章列表页 >

Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment

Medical Simulation Education | 更新时间：2025-08-08

- Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment
- Chinese Journal of Medical Education Research Vol. 24, Issue 7, Pages: 865-871(2025)
- 作者机构：
  
  1.武汉大学中南医院重症医学科/湖北省重症医学临床医学研究中心，武汉 430071
  2.中国医学科学院北京协和医院急诊科/疑难重症及罕见病全国重点实验室，北京 100730
  3.武汉大学中南医院教学办公室，武汉 430071
  4.中国医学科学院北京协和医院内科重症监护室，北京 100730
  5.武汉大学中南医院急救中心，武汉 430071
  6.北京大学人民医院麻醉科，北京 100044
- 作者简介：
  
  Li Li, Email: mrcatking@gmail.com
- 基金信息：
  
  Health Commission of Hubei Province Scientific Research Project(HBJG-250007)
- DOI：10.3760/cma.j.cn116021-20230503-02099
  CLC： R33
- Received：03 May 2023，
  
  Published：20 July 2025
- 稿件说明：
移动端阅览
李力, 刘继海, 谢亚典, 等. 提升医学模拟评价信度的12项实用建议[J]. 中华医学教育探索杂志, 2025, 24(7): 865-871.

Li Li, Liu Jihai, Xie Yadian, et al. Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment[J]. Chinese journal of medical education research, 2025, 24(7): 865-871.
李力, 刘继海, 谢亚典, 等. 提升医学模拟评价信度的12项实用建议[J]. 中华医学教育探索杂志, 2025, 24(7): 865-871. DOI： 10.3760/cma.j.cn116021-20230503-02099.

Li Li, Liu Jihai, Xie Yadian, et al. Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment[J]. Chinese journal of medical education research, 2025, 24(7): 865-871. DOI： 10.3760/cma.j.cn116021-20230503-02099.

摘要

医学模拟评价已成为衡量医护人员临床能力的重要方法。这些评价为受评者提供了在还原真实医疗场景且受控的环境中展示临床技能的机会。然而，评价组织方必须采取特定的措施，才能保障这些评价能够产生有效且可靠的结果。模拟评价的信度是模拟评价质量保障的重要组成部分。本文从评价者、模拟环境和评价工具3个关键维度梳理了提升模拟评价信度的12项实用建议：①选择具备相关专业背景和丰富模拟教学与评价经验的评价者；②为评价者提供关于评价的规范化培训；③进行模拟评分和试点评价，以达成评价者之间的共识；④防止评价者超时工作或认知负荷超载；⑤为标准化人员提供充分的培训和再培训；⑥对标准化病人的角色演绎提供即时和长期的质量保障措施；⑦确保模拟器保持一致的运行状态；⑧确保模拟为受评者提供足够的功能仿真度；⑨明确定义评价工具中的要素和项目，并控制评价条目的数量；⑩如果使用多个评价工具，确保不同工具之间的一致性；11尽可能使用具有构念效度证据的评价工具；12建立确保数据（评分）采集一致性的系统，并确保数据安全。这些策略旨在帮助医学教育工作者构建结构化的、可信的模拟考核评价方案。

Abstract

Medical simulation-based assessments have emerged as a vital method for evaluating the clinical competence of healthcare professionals. These assessments provide examinees with opportunities to practice clinical skills in a controlled environment that mirrors real-life medical scenarios. However

assessment organizers must implement specific measures to ensure that these assessments yield valid and reliable results. The reliability of simulation-based assessments is a crucial component of quality assurance. This article delineates twelve practical recommendations to enhance the reliability of medical simulation-based assessments across the three key dimensions of raters

simulation environment

and assessment tools. These recommendations include ① selecting raters with relevant professional backgrounds and extensive experience in simulation-based teaching and assessment; ②providing raters with standardized training for the assessment; ③conducting mock ratings and piloting to reach consensus among raters; ④preventing excessive work hours or cognitively overload while rating; ⑤providing standardized personnel with adequate training and re-training; ⑥implementing immediate and long-term quality assurance measures for standardized patient role performance; ⑦maintaining consistent operation of simulation equipment; ⑧guaranteeing the simulation provides adequate functional fidelity for the examinees; ⑨clearly defining each element and item in assessment tools while controlling their quantity; ⑩ensuring consistency when multiple assessment tools are used; 11using assessment tools with evidence of construct validity when possible; and 12establishing systems to ensure consistency in data (scoring) collection while safeguarding data security. These strategies aim to assist medical educators in developing structured and reliable simulation-based assessment schemes.

关键词

Keywords

references

McGaghie WC , Issenberg SB , Petrusa ER , et al . A critical review of simulation-based medical education research: 2003-2009 [J ] . Med Educ , 2010 , 44 ( 1 ): 50 - 63 . DOI: 10.1111/j.1365-2923.2009.03547.x http://dx.doi.org/10.1111/j.1365-2923.2009.03547.x .

Holmboe ES , Sherbino J , Long DM , et al . The role of assessment in competency-based medical education [J ] . Med Teach , 2010 , 32 ( 8 ): 676 - 682 . DOI: 10.3109/0142159X.2010.500704 http://dx.doi.org/10.3109/0142159X.2010.500704 .

Hawkins RE , Margolis MJ , Durning SJ , et al . Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research [J ] . Acad Med , 2010 , 85 ( 9 ): 1453 - 1461 . DOI: 10.1097/ACM.0b013e3181eac3e6 http://dx.doi.org/10.1097/ACM.0b013e3181eac3e6 .

van der VC , Schuwirth LW . Assessing professional competence: a model integrating measures of health care outcomes, performance assessment, and the assessment of knowledge [J ] . Adv Health Sci Educ Theory Pract , 2005 , 10 ( 4 ): 205 - 211 . DOI: 10.1007/978-3-319-30064-1 http://dx.doi.org/10.1007/978-3-319-30064-1 .

Kogan JR , Conforti LN , Bernabeo E , et al . How faculty members experience workplace-based assessment rater training: a qualitative study [J ] . Med Educ , 2015 , 49 ( 7 ): 692 - 708 . DOI: 10.1111/medu.12733 http://dx.doi.org/10.1111/medu.12733 .

Feldman M , Lazzara EH , Vanderbilt AA , et al . Rater training to support high-stakes simulation-based assessments [J ] . J Contin Educ Health Prof , 2012 , 32 ( 4 ): 279 - 286 . DOI: 10.1002/chp.21156 http://dx.doi.org/10.1002/chp.21156 .

Cook DA , Dupras DM , Beckman TJ , et al . Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial [J ] . J Gen Intern Med , 2009 , 24 ( 1 ): 74 - 79 . DOI: 10.1007/s11606-008-0842-3 http://dx.doi.org/10.1007/s11606-008-0842-3 .

McMullan M , Endacott R , Gray MA , et al . Portfolios and assessment of competence: a review of the literature [J ] . J Adv Nurs , 2003 , 41 ( 3 ): 283 - 294 . DOI: 10.1046/j.1365-2648.2003.02528.x http://dx.doi.org/10.1046/j.1365-2648.2003.02528.x .

Newman LR , Brodsky D , Jones RN , et al . Frame-of-reference training: establishing reliable assessment of teaching effectiveness [J ] . J Contin Educ Health Prof , 2016 , 36 ( 3 ): 206 - 210 . DOI: 10.1097/CEH.0000000000000086 http://dx.doi.org/10.1097/CEH.0000000000000086 .

Davis DA , Mazmanian PE , Fordis M , et al . Accuracy of physician self-assessment compared with observed measures of competence: a systematic review [J ] . JAMA , 2006 , 296 ( 9 ): 1094 - 1102 . DOI: 10.1001/jama.296.9.1094 http://dx.doi.org/10.1001/jama.296.9.1094 .

Kogan JR , Conforti L , Bernabeo E , et al . Opening the black box of clinical skills assessment via observation: a conceptual model [J ] . Med Educ , 2011 , 45 ( 10 ): 1048 - 1060 . DOI: 10.1111/j.1365-2923.2011.04025.x http://dx.doi.org/10.1111/j.1365-2923.2011.04025.x .

Tavares W , Eva KW . Exploring the impact of mental workload on rater-based assessments [J ] . Adv Health Sci Educ Theory Pract , 2013 , 18 ( 2 ): 291 - 303 . DOI: 10.1007/s10459-012-9370-3 http://dx.doi.org/10.1007/s10459-012-9370-3 .

Paravattil B , Wilby KJ . Optimizing assessors' mental workload in rater-based assessment: a critical narrative review [J ] . Perspect Med Educ , 2019 , 8 ( 6 ): 339 - 345 . DOI: 10.1007/s40037-019-00535-6 http://dx.doi.org/10.1007/s40037-019-00535-6 .

Cleland JA , Abe K , Rethans JJ . The use of simulated patients in medical education: AMEE Guide No 42 [J ] . Med Teach , 2009 , 31 ( 6 ): 477 - 486 . DOI: 10.1080/01421590903002821 http://dx.doi.org/10.1080/01421590903002821 .

Mirza N , Cinel J , Noyes H , et al . Simulated patient scenario development: a methodological review of validity and reliability reporting [J ] . Nurse Educ Today , 2020 , 85 : 104222 . DOI: 10.1016/j.nedt.2019.104222 http://dx.doi.org/10.1016/j.nedt.2019.104222 .

MacLean S , Geddes F , Kelly M , et al . Simulated patient training: Using inter-rater reliability to evaluate simulated patient consistency in nursing education [J ] . Nurse Educ Today , 2018 , 62 : 85 - 90 . DOI: 10.1016/j.nedt.2017.12.024 http://dx.doi.org/10.1016/j.nedt.2017.12.024 .

Shirazi M , Labaf A , Monjazebi F , et al . Assessing medical students' communication skills by the use of standardized patients: emphasizing standardized patients' quality assurance [J ] . Acad Psychiatry , 2014 , 38 ( 3 ): 354 - 360 . DOI: 10.1007/s40596-014-0066-2 http://dx.doi.org/10.1007/s40596-014-0066-2 .

Bagnasco A , Rossi S , Dasso N , et al . Basing patient safety education on real student experience: development of a multinational simulation scenario [J ] . J Prev Med Hyg , 2025 , 65 ( 4 ): E547 - E554 . DOI: 10.15167/2421-4248/jpmh2024.65.4.3110 http://dx.doi.org/10.15167/2421-4248/jpmh2024.65.4.3110 .

Elendu C , Amaechi DC , Okatta AU , et al . The impact of simulation-based training in medical education: a review [J ] . Medicine (Baltimore) , 2024 , 103 ( 27 ): e38813 . DOI: 10.1097/MD.0000000000038813 http://dx.doi.org/10.1097/MD.0000000000038813 .

Cook DA , Hatala R , Brydges R , et al . Technology-enhanced simulation for health professions education: a systematic review and meta-analysis [J ] . JAMA , 2011 , 306 ( 9 ): 978 - 988 . DOI: 10.1001/jama.2011.1234 http://dx.doi.org/10.1001/jama.2011.1234 .

Palaganas J , Raemer DB . Defining the role of simulation in health professions education: a qualitative study [J ] . Simul Healthc , 2017 , 12 ( 2 ): 90 - 97 . DOI: 10.2196/62803 http://dx.doi.org/10.2196/62803 .

Lavoie P , Michaud C , Bélisle M , et al . Learning theories and tools for the assessment of core nursing competencies in simulation: a theoretical review [J ] . J Adv Nurs , 2018 , 74 ( 2 ): 239 - 250 . DOI: 10.1111/jan.13416 http://dx.doi.org/10.1111/jan.13416 .

Mitchell S , Blanchard E , Curran V , et al . Effects of simulation fidelity on health care providers on team training-a systematic review [J ] . Simul Healthc , 2024 , 19 ( S1 ): S50 - S56 . DOI: 10.1097/SIH.0000000000000762 http://dx.doi.org/10.1097/SIH.0000000000000762 .

Downing SM . Reliability: on the reproducibility of assessment data [J ] . Med Educ , 2004 , 38 ( 9 ): 1006 - 1012 . DOI: 10.1111/j.1365-2929.2004.01932.x http://dx.doi.org/10.1111/j.1365-2929.2004.01932.x .

Downing SM . Validity: on meaningful interpretation of assessment data [J ] . Med Educ , 2003 , 37 ( 9 ): 830 - 837 . DOI: 10.1046/j.1365-2923.2003.01594.x http://dx.doi.org/10.1046/j.1365-2923.2003.01594.x .

Ilgen JS , Ma IWY , Hatala R , et al . A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment [J ] . Med Educ , 2015 , 49 ( 2 ): 161 - 173 . DOI: 10.1111/medu.12621 http://dx.doi.org/10.1111/medu.12621 .

Furman GE , Smee S , Wilson C . Quality assurance best practices for simulation-based examinations [J ] . Simul Healthc , 2010 , 5 ( 4 ): 226 - 231 . DOI: 10.1097/SIH.0b013e3181da5c93 http://dx.doi.org/10.1097/SIH.0b013e3181da5c93 .

van der Vleuten CM , Schuwirth LT , Driessen EW , et al . A model for programmatic assessment fit for purpose [J ] . Med Teach , 2012 , 34 ( 3 ): 205 - 214 . DOI: 10.3109/0142159X.2012.652239 http://dx.doi.org/10.3109/0142159X.2012.652239 .

Norcini J . On purpose: the case for alignment in assessment [J ] . Acad Med , 2023 , 98 ( 11 ): 1240 - 1242 . DOI: 10.1097/ACM.0000000000005430 http://dx.doi.org/10.1097/ACM.0000000000005430 .

Borsboom D . Educational measurement (4th Ed.) [J ] . Struct Equ Model A Multidiscip J , 2009 , 16 ( 4 ): 702 - 711 . DOI: 10.1080/10705510903206097 http://dx.doi.org/10.1080/10705510903206097 .

McGaghie WC , Hinkamp MM , Harden RM . International best practices for evaluation in the health professions [M ] . London : CRC Press , 2022 .

王佳玉，史霆，徐玲玲 . 基于Delphi-AHP法构建医学情境模拟课程实施质量评价工具的探索 [J ] . 中华医学教育杂志， 2025 ， 45 （ 3 ）： 235 - 240 . DOI: 10.3760/cma.j.cn115259-20240116-00058 http://dx.doi.org/10.3760/cma.j.cn115259-20240116-00058 .

Downing SM . Twelve steps for effective test development [M ] // Handbook of Test Development . London : Routledge , 2015 .

Johnston L , Sawyer T , Nishisaki A , et al . Neonatal intubation competency assessment tool: development and validation [J ] . Acad Pediatr , 2019 , 19 ( 2 ): 157 - 164 . DOI: 10.1016/j.acap.2018.07.008 http://dx.doi.org/10.1016/j.acap.2018.07.008 .

Stufflebeam DL . Guidelines for developing evaluation checklists: the checklists development checklist (cdc) [EB/OL ] . [ 2025-03-10 ] . https://files.wmich.edu/s3fs-public/attachments/u350/2014/guidelines_cdc.pdf https://files.wmich.edu/s3fs-public/attachments/u350/2014/guidelines_cdc.pdf .

McQueen S , McKinnon V , VanderBeek L , et al . Video-based assessment in surgical education: a scoping review [J ] . J Surg Educ , 2019 , 76 ( 6 ): 1645 - 1654 . DOI: 10.1016/j.jsurg.2019.05.013 http://dx.doi.org/10.1016/j.jsurg.2019.05.013 .

Fu Y , Zhang W , Zhang S , et al . Applying a video recording, video-based rating method in OSCEs [J ] . Med Educ Online , 2023 , 28 ( 1 ): 2187949 . DOI: 10.1080/10872981.2023.2187949 http://dx.doi.org/10.1080/10872981.2023.2187949 .

Boal MWE , Anastasiou D , Tesfai F , et al . Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review [J ] . Br J Surg , 2024 , 111 ( 1 ): znad331 . DOI: 10.1093/bjs/znad331 http://dx.doi.org/10.1093/bjs/znad331 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

基于置信职业行为构建胜任力导向的肿瘤学实习考评体系

Translation of the Maastricht Clinical Teaching Questionnaire and its reliability and validity in the standardized training of residents

Application of medical simulation in standardized training of anesthesia residents

Evaluation on the reliability and validity of the social networking using status scale among medical students

Development and Evaluation of Tobacco Control Questionnaire for Nursing Students

Related Author

Kuang Ming

Hou Tao

Ma Fei

Guo Jing

Miao Rulin

Zou Bijun

Huang Hua

Liang Yao

Related Institution

Department of Education, Shandong Provincial Qianfoshan Hospital, The First Affiliated Hospital of Shandong First Medical University

Department of Anesthesiology, The First Affiliated Hospital of Army Medical University

上海交通大学医学院学科规划处

上海市徐汇区疾病预防控制中心

⁰