Foreword: The Next Era of Assessment and Precision Education

Abstract

This supplement offers a vision for the next era of medical education assessment that centers on why assessment is done, namely, to ensure that learners achieve the learning outcomes that prepare them to provide high-quality, equitable patient care.1 Equitable care is defined as providing care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status.2 To achieve these learning outcomes, this supplement proposes an era of assessment that centers around developing and implementing a precision education (PE) system that uses data and technology to transform lifelong learning by improving personalization, efficiency, and learner agency.3 Articles in the supplement explore 3 thematic areas: 1) A vision for assessment and patient care that is personalized, supportive, high-quality, and equitable; 2) Foundations for using data and technology to transform and personalize efficient and effective assessment and lifelong learning; and 3) Examples using data and technology to transform and personalize efficient and effective assessment and learning. Answering the Call of Competency-Based Medical Education Competency-based medical education (CBME) has recently shifted from a preponderance of theoretical discussions with limited meaningful pilots4–6 to multiple large-scale national implementation efforts in several countries and specialties.7,8 At its core, CBME can be defined by 2 concepts: patient-focused and learner-centered.9–11 CBME starts with the end in mind—what do patients and populations need? With the answer to this in hand, CBME defines training outcomes that align with those patient needs and then designs curricula and assessments to ensure those training outcomes are met.10 The curricula and assessments—in fact, the whole of the training programs—should be learner-centered in CBME and place learners in control of their educational process. The environment they are learning in should help them take the next steps in their development, intentionally sequencing competencies and skills over time.9,11 Despite these lofty goals, individuals graduate from training programs with gaps in their basic ability to provide the care that patients need,12–15 resulting in low-quality16,17 and inequitable care.18 There are multiple root causes contributing to this chasm between the current state of medical education and the goals of CBME, including under-resourcing of training programs and use of assessment practices that are not learner-centered and have unwanted variability.19,20 A fundamentally different system of assessment is needed to ensure learners take the assured, successive steps in their development toward providing the care their patients will need. Envisioning the Next Era of Assessment and PE Building on prior work,21 Desai and colleagues advance a conceptual framework for PE that involves applying advanced analytics to multimodal data inputs to generate insights that can then be used to drive precision interventions.3Outcomes are assessed assiduously, gaps are reassessed, and the “cycle” repeats. These cycles can occur at multiple levels22—individuals, teams, programs, or organizations—reflecting contemporary conceptualizations of outcomes assessment and systems thinking.23–25 If PE initially sounds similar to other improvement models, such as the master adaptive learner (MAL) cycle or continuous quality improvement,26,27 that is by design. PE intentionally seeks to build upon these proven frameworks as well as systems science to improve the care learners provide through cycles of inputs, insights, interventions, and outcomes. As the nascent PE framework evolves, it will need continual enhancement and refinement from the medical education community. At the same time, PE also moves medical education in a number of bold new directions described by authors in this supplement, which we explore thematically next. High-quality equitable care through high-quality equitable learner assessment While previous eras of assessment in medical education were defined by how assessment was done, Schumacher and colleagues argue that the next era of assessment must be defined by why assessment is done: to ensure high-quality, equitable patient care.1 We must agree that high-quality care is equitable care—and make this the North Star of PE. Foregrounding health equity across all elements of the PE cycle will require critical reappraisal of systemic racism and harmful bias and inequity that are deeply ingrained in medicine.28 In our view, all inequity is unfair and unjust and must be addressed. Unfortunately, harmful bias and inequity are too often “woven into the fabric”29 of assessment efforts as well.29–32 Thus, high-quality, equitable learner assessment is needed—that is, “all [learners] have fair and impartial opportunities to learn, be evaluated, coached, graded, advanced, graduated, and selected for subsequent opportunities based on their demonstration of achievements that predict future success in the field of medicine, and that neither learning experiences nor assessments are negatively influenced by structural or interpersonal bias related to personal or social characteristics of learners or assessors.”33 Marcotte and colleagues note that transparency in assessment development and implementation is an important means to achieve this goal.34 Weaving equity into the fabric of emerging novel assessment efforts will require cocreation with learners, from concept to design and implementation to maintenance.34,35 Likewise, anchoring assessment around patient care requires we engage patients and families, yet only a few efforts have detailed meaningful engagement of patients and families in assessment efforts.36–38 To better understand how equity affects PE systems, we asked all authors contributing papers to this supplement to foreground equity and engage trainees in codevelopment of papers. This was variably achieved, which served as a reminder of ever-present limitations and difficulties likely reflecting structural challenges—underrepresentation in medical education; siloing of colleagues who promote diversity, inclusion, and belonging; and marginalization of trainees from the very work that is for them and about them. As a field, medical education has a long way to go. Rethinking our structural approach to programs of assessment and engaging diverse learners and patients in coproduction will not prevent events where harmful bias, inequity, or lack of fairness arises. An apt example would be a frontline clinician-educator entering narrative comments for clerkship students that use language perpetuating bias or inequity based on gender or race, and these comments getting propagated into students’ residency applications. To address this, Thoma and colleagues note how powerful new artificial intelligence (AI) tools are emerging that can prompt assessors in real time when potentially biased language is detected.39 Such approaches must be prioritized and resourced. Scalable precision assessment: high-density, learner-attributable performance data In the PE model, data are the essential currency. Learners and patients own their data, which should cross arbitrary silos of training or clinical care. By establishing ethical principles and enforcing interoperability standards, learner and patient data can flow across the various entities and products that exist in our fragmented medical education ecosystem.3,40 To achieve the precision necessary to drive proactive, targeted interventions that are predictive of high-quality equitable care, the density (quantity) and quality of data we collect about learners must be increased. This prospect is daunting, as existing CBME implementation efforts offer a cautionary tale of assessments resulting in learner harm or burden, rather than learner-centeredness.20,41 However, these cautionary tales further emphasize the need to identify new sources of reliable assessment data that do not rely heavily on human raters, including both “found” and fit-for-purpose data. Most importantly, the data must drive meaningful insights for learners that can be used for feedback and improvement. Kinnear and colleagues detail an instructive analogy: how professional sports (specifically Major League Baseball) went from solely relying on human scouting to using advanced technology for assessing the most granular units of individual performance.42 Pitching performance, for example, once was solely measured through scouting evaluations and game-level outcomes (like win/loss) that were multifactorial. While these measures remain important, innovations in technology have led to sophisticated pitch-level measurements of velocity, movement, spin rate, and the like. The pitcher has complete control over these novel measures (i.e., they are not affected by hitter performance, umpire calls, or ballpark configuration), and they are assessed without human input. This analogy demonstrates a model for assessment in medical training that identifies objective metrics of individual and team performance. Of course, assessing learners in the authentic clinical learning environment is vastly different from a game of baseball, where the rules, environment, and goals are all (largely) static. Nonetheless, baseball’s evolution to augment human rater-derived assessments with modern technology to capture much higher density measurements of individual performance is instructive. Emerging performance measures in medical education, such as Resident Sensitive Quality Measures (RSQMs) and TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs),43–46 take a step in this direction in focusing on measures that have significant attribution to a single learner yet remain meaningful to patient care in their own right. However, even more granular measures are also needed, such as through electronic health record (EHR) metadata, motion capture, and wearable and haptic devices. Obviously, such efforts would need to be done with transparency as well as consent, assent, or ability to opt out in applicable situations. Garibaldi and colleagues describe a multi-institutional effort to use real-time location service badges to quantify residents’ time at the bedside, in the workroom, or on rounds.40 Such technologies capture orders of magnitude more data about learners (over 95,000 hours on 43 residents in 1 year), revealing clinically meaningful inter-resident differences.47 Two papers in this supplement describe approaches for using haptics to achieve much more granular assessment of performance during existing standardized clinical tasks, such as an Objective Structured Clinical Examinations (OSCEs) on thyroid or breast palpation.48,49 While the measured forces used to palpate the breast may only be interpreted normatively in isolation, when linked to a meaningful patient outcome (e.g., positive likelihood ratio for detecting a mass), such measurements become powerful and precise tools for performance assessment and timely feedback. These efforts to increase the density and precision of assessment of individual learners are an essential building block for the next era of assessment. Additionally, as Sebok-Syer and colleagues explore, it will be essential to understand and measure interdependent team-based effects.50 Measuring how interdependence shapes learner performance throughout training could be especially impactful in both measuring team performance and identifying critical periods to durably shape learners’ future practice patterns, which appear to be driven by interdependence. Stepping out one further “ring” to understand how teams come together to account for program-level outcomes will also require fundamentally different thinking.22 AI: powering analytics to gather insights on big data Emerging high-density assessment data streams bring with them challenges in generating timely insights related to the 6Vs of “big data”: Volume, Velocity, Variety, Veracity, Variability, and Value.51 Statistical approaches that account for these increasingly complex and often nested measures with multilevel models have been recently described, as have been approaches for adjusting assessments to control for rater effects.52,53 We view these approaches as analogous to baseball’s successful development of analytics that account for team, opponent, and ballpark effects. AI will surely also be essential. Recognizing the breathtaking pace of change in generative AI and other techniques, Turner and colleagues share a vision for how AI can be used for collecting, analyzing, and intervening upon these data streams.54 Issues related to explainability of previously “black box” models are rapidly being addressed, while increased attention to the propagation of bias has led to important principles and guardrails. Thoma and colleagues further explore the promise and perils of applying AI models to “found” data, emphasizing threats related to validity, consent, and unwanted bias.39 Large language models (LLMs), on which technologies like ChatGPT are built, are advancing rapidly. We anticipate LLMs will make sophisticated analyses of previously difficult-to-analyze data types—such as audio and text—commonplace in medical education in the coming years. LLM deployment in the clinical environment will undoubtedly create new data streams that might be used for learner assessment that were previously untenable, such as patient-physician communication patterns from an ambient listening tool.55 Considering how such data can be used to deliver high-quality equitable assessments of our learners and, in turn, ensure our graduates provide high-quality equitable care will be critical. Likewise, new AI tools, such as Segment Anything,56 have transformed computer vision and may eventually augment what an “observable” professional activity is through computer-assisted video analysis.57 Theory-grounded implementation through transdisciplinary collaboration To move beyond isolated innovations in assessment and fully realize a system of precision assessment will require theory-grounded implementation efforts and transdisciplinary collaboration. Drake and colleagues describe several PE interventions and the underlying theories and digital tools that were applied to ensure the implementation efforts were grounded in proven approaches.58 In addition to the theories they review (MAL, transformative learning, and self-determination), others applicable to PE could include cognitive load theory (e.g., personalizing experiences to match an individual learner’s cognitive load capacity),59,60 situated cognition theory (e.g., incorporating context into assessment),61 and social learning theory (e.g., understanding how to assess and leverage interprofessional interactions for durable high-quality care patterns).62,63 Likewise, in addition to dashboards and nudge strategy, tools that will advance PE will include digital health devices (e.g., wearables64 and sensors65), adaptive learning platforms, virtual and augmented reality (VR/AR),66,67 gamification,68 and LLM chatbot-based tutors and coaches (e.g., https://hpe-bot.com).69 Garibaldi and colleagues draw upon building blocks for PE to illustrate how applying the PE conceptual framework can elucidate strengths and gaps in PE implementation cycles, whether at the individual or program level.40 Warm and colleagues show that programs need not use AI to achieve true PE; taking a principled approach that provides learners with performance goals related to both assessed performance and improvement over time ensures learners across the performance spectrum remain growth minded.70 Kinnear and colleagues highlight the so-called “Medici effect”71—that transformative innovation often arises from disciplinary intersectionality (i.e., transdisciplinary collaboration) where different ideas and approaches converge.42 Their analogy to player performance assessment in professional sports, though imperfect, is highly instructive and begs the question of whether medical education should be more active in engaging experts in assessment from other fields. The field of haptics has emerged out of engineering, medicine, and the science of touch; similarly, we must leverage the expertise of those doing measurement in other fields. Finally, the Canadian CBME initiative “Competency by Design” has been plagued by challenges related to logistics, workflow disruption, and assessment burnout.20,72–74 Although not explored in depth in this supplement, implementation science has multiple constructs for effecting change at the individual, team, or organization levels,75 such as the Consolidated Framework for Implementation Research (CFIR),76,77 that will be critical to apply in order for PE to move from innovation to widespread implementation. Tackling Threats to the Next Era of Assessment A number of challenges were highlighted by multiple authors in this supplement that threaten the realization of this next era of assessment and PE. The resources—both monetary and human—devoted to health professions education are inadequate. Only 2% of the global health care expenditures go toward health professions education (i.e., medicine, nursing, midwives, public health).10 Paltry funding is unlikely to lead to delivering on the bold visions set forth in this supplement. In the United States, allocating a fraction of federal funding for graduate medical education positions, which currently exceeds $16 billion per year, to medical education innovation and research would be transformational.78 The absence of such funding streams makes the prospect of developing a PE system—replete with complex data inputs, new technologies for data collection, AI analytics, and the like—frankly daunting. Unsurprisingly, existing PE innovations have generally occurred at institutions with more resources, including analytic supports and data scientists. Without concerted effort, this innovation disparity will only widen. Although PE will require front-loaded investment, we truly believe these investments will pay profound dividends to our health care system. As discussed by Richardson and colleagues, an organized approach to this next era of assessment with funding and structures similar to the Human Genome Project or Cancer Moonshot Initiative will be necessary to scale from isolated innovations to widespread implementation of PE.22 Calls to honor medicine’s social contract by ensuring the quality of medical graduates are growing louder yet no single entity “owns” the problem. Even more aspirational is the dream of multiple entities partnering together and pooling resources to address the problem. Further complicating these challenges, we believe the changes needed in medical education to realize the vision set forth in this supplement, or any other vision that seeks to better serve patients and learners, must be viewed as an adaptive challenge rather than a technical problem.79,80 A technical problem is one in which the problem and solution are clear, the work will be managed by an expert, and the is on a An adaptive on the other hand, does not a problem or challenges can be to to example, the to achieve equitable assessments and patient care, these are by and to adaptive challenges requires and new This work be done by the of experts requires concerted of and resources in the challenges by changes in and such as implementation and together with medicine, medical education, and learners and patient will be essential. Call to current system of health professions education is to deliver graduates who can deliver high-quality equitable care. The vision for the next era of assessment in this supplement requires bold As of this supplement, we the papers a to to address the gaps in our current system through a new system of PE and assessment. We on medical residency and funding and national to this vision and the necessary and must be as Implementation efforts should draw upon proven frameworks while also the adaptive challenges that will occur will profound The of our learners and the care they provide can be transformed with a to high-quality equitable assessment. we we and the We will a in of to the growing and to build a of The will be a future to provide the care patients that is and of the

Topics & Concepts