Developing a validity argument for a simulation-based model of entrustment in dispensing skills assessment framework

Introduction: Integrated assessment of multiple competencies at once, including entrustable professional activity (EPA) based assessment, is emerging as an effective approach to competency-based evaluation of health professionals. However, there is an absence of validated assessment frameworks in entry level pharmacy education. We aimed to develop an assessment framework and establish a validity argument, containing multiple sources of evidence, for use in the integrated assessment of pharmacy student's competency in all aspects of the supply of prescribed medicine(s). Methods: A two-phase prospective study was conducted. Phase 1 involved development and content validation of the Model of Entrustment in Dispensing Skills (MEDS) assessment framework using a literature review, a think-aloud study, and expert consultation. In phase 2, a pilot study was conducted with faculty and expert assessors to test the framework. Subsequent analysis involved psychometric evaluation of rating scales and usability testing. Results: Validity evidence was collected and organised across the two study phases. The MEDS framework had good evidence of content validity supported by the rigorous development and consultation process, as well as case sampling, with 88% of national practice-based competencies represented across the two simulations. Reliability coefficients were high and acceptable, supporting strong agreement across domains, students, and simulations as well as a strong correlation between the EPA and total score (spearman correlation rho 0.725, P <.001). Conclusions: This study describes a valid and rigorous approach for the implementation and interpretation of an integrated simulation-based assessment tool for determining pharmacy student's progress towards entrustment for independent medication supply practice.