提升医学模拟评价信度的12项实用建议

李力; 刘继海; 谢亚典; 史迪; 胡小芸; 陈志桥; 安海燕

doi:10.3760/cma.j.cn116021-20230503-02099

您当前的位置：

首页 >

文章列表页 >

提升医学模拟评价信度的12项实用建议

医学模拟教育 | 更新时间：2025-08-08

- 提升医学模拟评价信度的12项实用建议
- Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment
- 中华医学教育探索杂志 2025年24卷第7期页码：865-871
- 作者机构：
  
  1.武汉大学中南医院重症医学科/湖北省重症医学临床医学研究中心，武汉 430071
  2.中国医学科学院北京协和医院急诊科/疑难重症及罕见病全国重点实验室，北京 100730
  3.武汉大学中南医院教学办公室，武汉 430071
  4.中国医学科学院北京协和医院内科重症监护室，北京 100730
  5.武汉大学中南医院急救中心，武汉 430071
  6.北京大学人民医院麻醉科，北京 100044
- 作者简介：
  
  李力，Email：mrcatking@gmail.com
- 基金信息：
  
  湖北省卫生健康委员会科研基金资助(HBJG-250007)
- DOI：10.3760/cma.j.cn116021-20230503-02099
  中图分类号： R33
- 收稿日期：2023-05-03，
  
  纸质出版日期：2025-07-20
- 稿件说明：
移动端阅览
李力, 刘继海, 谢亚典, 等. 提升医学模拟评价信度的12项实用建议[J]. 中华医学教育探索杂志, 2025, 24(7): 865-871.

Li Li, Liu Jihai, Xie Yadian, et al. Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment[J]. Chinese journal of medical education research, 2025, 24(7): 865-871.
李力, 刘继海, 谢亚典, 等. 提升医学模拟评价信度的12项实用建议[J]. 中华医学教育探索杂志, 2025, 24(7): 865-871. DOI： 10.3760/cma.j.cn116021-20230503-02099.

Li Li, Liu Jihai, Xie Yadian, et al. Twelve practical recommendations for enhancing the reliability of medical simulation-based assessment[J]. Chinese journal of medical education research, 2025, 24(7): 865-871. DOI： 10.3760/cma.j.cn116021-20230503-02099.

摘要

医学模拟评价已成为衡量医护人员临床能力的重要方法。这些评价为受评者提供了在还原真实医疗场景且受控的环境中展示临床技能的机会。然而，评价组织方必须采取特定的措施，才能保障这些评价能够产生有效且可靠的结果。模拟评价的信度是模拟评价质量保障的重要组成部分。本文从评价者、模拟环境和评价工具3个关键维度梳理了提升模拟评价信度的12项实用建议：①选择具备相关专业背景和丰富模拟教学与评价经验的评价者；②为评价者提供关于评价的规范化培训；③进行模拟评分和试点评价，以达成评价者之间的共识；④防止评价者超时工作或认知负荷超载；⑤为标准化人员提供充分的培训和再培训；⑥对标准化病人的角色演绎提供即时和长期的质量保障措施；⑦确保模拟器保持一致的运行状态；⑧确保模拟为受评者提供足够的功能仿真度；⑨明确定义评价工具中的要素和项目，并控制评价条目的数量；⑩如果使用多个评价工具，确保不同工具之间的一致性；11尽可能使用具有构念效度证据的评价工具；12建立确保数据（评分）采集一致性的系统，并确保数据安全。这些策略旨在帮助医学教育工作者构建结构化的、可信的模拟考核评价方案。

Abstract

Medical simulation-based assessments have emerged as a vital method for evaluating the clinical competence of healthcare professionals. These assessments provide examinees with opportunities to practice clinical skills in a controlled environment that mirrors real-life medical scenarios. However

assessment organizers must implement specific measures to ensure that these assessments yield valid and reliable results. The reliability of simulation-based assessments is a crucial component of quality assurance. This article delineates twelve practical recommendations to enhance the reliability of medical simulation-based assessments across the three key dimensions of raters

simulation environment

and assessment tools. These recommendations include ① selecting raters with relevant professional backgrounds and extensive experience in simulation-based teaching and assessment; ②providing raters with standardized training for the assessment; ③conducting mock ratings and piloting to reach consensus among raters; ④preventing excessive work hours or cognitively overload while rating; ⑤providing standardized personnel with adequate training and re-training; ⑥implementing immediate and long-term quality assurance measures for standardized patient role performance; ⑦maintaining consistent operation of simulation equipment; ⑧guaranteeing the simulation provides adequate functional fidelity for the examinees; ⑨clearly defining each element and item in assessment tools while controlling their quantity; ⑩ensuring consistency when multiple assessment tools are used; 11using assessment tools with evidence of construct validity when possible; and 12establishing systems to ensure consistency in data (scoring) collection while safeguarding data security. These strategies aim to assist medical educators in developing structured and reliable simulation-based assessment schemes.

关键词

Keywords

references

McGaghie WC , Issenberg SB , Petrusa ER , et al . A critical review of simulation-based medical education research: 2003-2009 [J ] . Med Educ , 2010 , 44 ( 1 ): 50 - 63 . DOI: 10.1111/j.1365-2923.2009.03547.x http://dx.doi.org/10.1111/j.1365-2923.2009.03547.x .

Holmboe ES , Sherbino J , Long DM , et al . The role of assessment in competency-based medical education [J ] . Med Teach , 2010 , 32 ( 8 ): 676 - 682 . DOI: 10.3109/0142159X.2010.500704 http://dx.doi.org/10.3109/0142159X.2010.500704 .

Hawkins RE , Margolis MJ , Durning SJ , et al . Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research [J ] . Acad Med , 2010 , 85 ( 9 ): 1453 - 1461 . DOI: 10.1097/ACM.0b013e3181eac3e6 http://dx.doi.org/10.1097/ACM.0b013e3181eac3e6 .

van der VC , Schuwirth LW . Assessing professional competence: a model integrating measures of health care outcomes, performance assessment, and the assessment of knowledge [J ] . Adv Health Sci Educ Theory Pract , 2005 , 10 ( 4 ): 205 - 211 . DOI: 10.1007/978-3-319-30064-1 http://dx.doi.org/10.1007/978-3-319-30064-1 .

Kogan JR , Conforti LN , Bernabeo E , et al . How faculty members experience workplace-based assessment rater training: a qualitative study [J ] . Med Educ , 2015 , 49 ( 7 ): 692 - 708 . DOI: 10.1111/medu.12733 http://dx.doi.org/10.1111/medu.12733 .

Feldman M , Lazzara EH , Vanderbilt AA , et al . Rater training to support high-stakes simulation-based assessments [J ] . J Contin Educ Health Prof , 2012 , 32 ( 4 ): 279 - 286 . DOI: 10.1002/chp.21156 http://dx.doi.org/10.1002/chp.21156 .

Cook DA , Dupras DM , Beckman TJ , et al . Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial [J ] . J Gen Intern Med , 2009 , 24 ( 1 ): 74 - 79 . DOI: 10.1007/s11606-008-0842-3 http://dx.doi.org/10.1007/s11606-008-0842-3 .

McMullan M , Endacott R , Gray MA , et al . Portfolios and assessment of competence: a review of the literature [J ] . J Adv Nurs , 2003 , 41 ( 3 ): 283 - 294 . DOI: 10.1046/j.1365-2648.2003.02528.x http://dx.doi.org/10.1046/j.1365-2648.2003.02528.x .

Newman LR , Brodsky D , Jones RN , et al . Frame-of-reference training: establishing reliable assessment of teaching effectiveness [J ] . J Contin Educ Health Prof , 2016 , 36 ( 3 ): 206 - 210 . DOI: 10.1097/CEH.0000000000000086 http://dx.doi.org/10.1097/CEH.0000000000000086 .

Davis DA , Mazmanian PE , Fordis M , et al . Accuracy of physician self-assessment compared with observed measures of competence: a systematic review [J ] . JAMA , 2006 , 296 ( 9 ): 1094 - 1102 . DOI: 10.1001/jama.296.9.1094 http://dx.doi.org/10.1001/jama.296.9.1094 .

Kogan JR , Conforti L , Bernabeo E , et al . Opening the black box of clinical skills assessment via observation: a conceptual model [J ] . Med Educ , 2011 , 45 ( 10 ): 1048 - 1060 . DOI: 10.1111/j.1365-2923.2011.04025.x http://dx.doi.org/10.1111/j.1365-2923.2011.04025.x .

Tavares W , Eva KW . Exploring the impact of mental workload on rater-based assessments [J ] . Adv Health Sci Educ Theory Pract , 2013 , 18 ( 2 ): 291 - 303 . DOI: 10.1007/s10459-012-9370-3 http://dx.doi.org/10.1007/s10459-012-9370-3 .

Paravattil B , Wilby KJ . Optimizing assessors' mental workload in rater-based assessment: a critical narrative review [J ] . Perspect Med Educ , 2019 , 8 ( 6 ): 339 - 345 . DOI: 10.1007/s40037-019-00535-6 http://dx.doi.org/10.1007/s40037-019-00535-6 .

Cleland JA , Abe K , Rethans JJ . The use of simulated patients in medical education: AMEE Guide No 42 [J ] . Med Teach , 2009 , 31 ( 6 ): 477 - 486 . DOI: 10.1080/01421590903002821 http://dx.doi.org/10.1080/01421590903002821 .

Mirza N , Cinel J , Noyes H , et al . Simulated patient scenario development: a methodological review of validity and reliability reporting [J ] . Nurse Educ Today , 2020 , 85 : 104222 . DOI: 10.1016/j.nedt.2019.104222 http://dx.doi.org/10.1016/j.nedt.2019.104222 .

MacLean S , Geddes F , Kelly M , et al . Simulated patient training: Using inter-rater reliability to evaluate simulated patient consistency in nursing education [J ] . Nurse Educ Today , 2018 , 62 : 85 - 90 . DOI: 10.1016/j.nedt.2017.12.024 http://dx.doi.org/10.1016/j.nedt.2017.12.024 .

Shirazi M , Labaf A , Monjazebi F , et al . Assessing medical students' communication skills by the use of standardized patients: emphasizing standardized patients' quality assurance [J ] . Acad Psychiatry , 2014 , 38 ( 3 ): 354 - 360 . DOI: 10.1007/s40596-014-0066-2 http://dx.doi.org/10.1007/s40596-014-0066-2 .

Bagnasco A , Rossi S , Dasso N , et al . Basing patient safety education on real student experience: development of a multinational simulation scenario [J ] . J Prev Med Hyg , 2025 , 65 ( 4 ): E547 - E554 . DOI: 10.15167/2421-4248/jpmh2024.65.4.3110 http://dx.doi.org/10.15167/2421-4248/jpmh2024.65.4.3110 .

Elendu C , Amaechi DC , Okatta AU , et al . The impact of simulation-based training in medical education: a review [J ] . Medicine (Baltimore) , 2024 , 103 ( 27 ): e38813 . DOI: 10.1097/MD.0000000000038813 http://dx.doi.org/10.1097/MD.0000000000038813 .

Cook DA , Hatala R , Brydges R , et al . Technology-enhanced simulation for health professions education: a systematic review and meta-analysis [J ] . JAMA , 2011 , 306 ( 9 ): 978 - 988 . DOI: 10.1001/jama.2011.1234 http://dx.doi.org/10.1001/jama.2011.1234 .

Palaganas J , Raemer DB . Defining the role of simulation in health professions education: a qualitative study [J ] . Simul Healthc , 2017 , 12 ( 2 ): 90 - 97 . DOI: 10.2196/62803 http://dx.doi.org/10.2196/62803 .

Lavoie P , Michaud C , Bélisle M , et al . Learning theories and tools for the assessment of core nursing competencies in simulation: a theoretical review [J ] . J Adv Nurs , 2018 , 74 ( 2 ): 239 - 250 . DOI: 10.1111/jan.13416 http://dx.doi.org/10.1111/jan.13416 .

Mitchell S , Blanchard E , Curran V , et al . Effects of simulation fidelity on health care providers on team training-a systematic review [J ] . Simul Healthc , 2024 , 19 ( S1 ): S50 - S56 . DOI: 10.1097/SIH.0000000000000762 http://dx.doi.org/10.1097/SIH.0000000000000762 .

Downing SM . Reliability: on the reproducibility of assessment data [J ] . Med Educ , 2004 , 38 ( 9 ): 1006 - 1012 . DOI: 10.1111/j.1365-2929.2004.01932.x http://dx.doi.org/10.1111/j.1365-2929.2004.01932.x .

Downing SM . Validity: on meaningful interpretation of assessment data [J ] . Med Educ , 2003 , 37 ( 9 ): 830 - 837 . DOI: 10.1046/j.1365-2923.2003.01594.x http://dx.doi.org/10.1046/j.1365-2923.2003.01594.x .

Ilgen JS , Ma IWY , Hatala R , et al . A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment [J ] . Med Educ , 2015 , 49 ( 2 ): 161 - 173 . DOI: 10.1111/medu.12621 http://dx.doi.org/10.1111/medu.12621 .

Furman GE , Smee S , Wilson C . Quality assurance best practices for simulation-based examinations [J ] . Simul Healthc , 2010 , 5 ( 4 ): 226 - 231 . DOI: 10.1097/SIH.0b013e3181da5c93 http://dx.doi.org/10.1097/SIH.0b013e3181da5c93 .

van der Vleuten CM , Schuwirth LT , Driessen EW , et al . A model for programmatic assessment fit for purpose [J ] . Med Teach , 2012 , 34 ( 3 ): 205 - 214 . DOI: 10.3109/0142159X.2012.652239 http://dx.doi.org/10.3109/0142159X.2012.652239 .

Norcini J . On purpose: the case for alignment in assessment [J ] . Acad Med , 2023 , 98 ( 11 ): 1240 - 1242 . DOI: 10.1097/ACM.0000000000005430 http://dx.doi.org/10.1097/ACM.0000000000005430 .

Borsboom D . Educational measurement (4th Ed.) [J ] . Struct Equ Model A Multidiscip J , 2009 , 16 ( 4 ): 702 - 711 . DOI: 10.1080/10705510903206097 http://dx.doi.org/10.1080/10705510903206097 .

McGaghie WC , Hinkamp MM , Harden RM . International best practices for evaluation in the health professions [M ] . London : CRC Press , 2022 .

王佳玉，史霆，徐玲玲 . 基于Delphi-AHP法构建医学情境模拟课程实施质量评价工具的探索 [J ] . 中华医学教育杂志， 2025 ， 45 （ 3 ）： 235 - 240 . DOI: 10.3760/cma.j.cn115259-20240116-00058 http://dx.doi.org/10.3760/cma.j.cn115259-20240116-00058 .

Downing SM . Twelve steps for effective test development [M ] // Handbook of Test Development . London : Routledge , 2015 .

Johnston L , Sawyer T , Nishisaki A , et al . Neonatal intubation competency assessment tool: development and validation [J ] . Acad Pediatr , 2019 , 19 ( 2 ): 157 - 164 . DOI: 10.1016/j.acap.2018.07.008 http://dx.doi.org/10.1016/j.acap.2018.07.008 .

Stufflebeam DL . Guidelines for developing evaluation checklists: the checklists development checklist (cdc) [EB/OL ] . [ 2025-03-10 ] . https://files.wmich.edu/s3fs-public/attachments/u350/2014/guidelines_cdc.pdf https://files.wmich.edu/s3fs-public/attachments/u350/2014/guidelines_cdc.pdf .

McQueen S , McKinnon V , VanderBeek L , et al . Video-based assessment in surgical education: a scoping review [J ] . J Surg Educ , 2019 , 76 ( 6 ): 1645 - 1654 . DOI: 10.1016/j.jsurg.2019.05.013 http://dx.doi.org/10.1016/j.jsurg.2019.05.013 .

Fu Y , Zhang W , Zhang S , et al . Applying a video recording, video-based rating method in OSCEs [J ] . Med Educ Online , 2023 , 28 ( 1 ): 2187949 . DOI: 10.1080/10872981.2023.2187949 http://dx.doi.org/10.1080/10872981.2023.2187949 .

Boal MWE , Anastasiou D , Tesfai F , et al . Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review [J ] . Br J Surg , 2024 , 111 ( 1 ): znad331 . DOI: 10.1093/bjs/znad331 http://dx.doi.org/10.1093/bjs/znad331 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于置信职业行为构建胜任力导向的肿瘤学实习考评体系

临床教学问卷的汉化及其在住院医师培养过程中的信效度检验

模拟医学在麻醉住院医师规范化培训中的应用研究

社交网络使用量表在医学生中的信度和效度评估

护理专业学生控烟问卷的编制及其信效度分析