Inter‐rater Reliability of Clinical Frailty Scores for Older Patients in the Emergency Department

Abstract

Over 50 million U.S. adults 65 years and older account for >20 million emergency department (ED) visits each year.1 Increasing ED use by older adults is projected to exceed the capacity of U.S. EDs.2 The traditional ED model of care is ill-equipped to address the many complex care needs of older adults.2 To address these problems, an evolution of emergency care has developed as evidenced by consensus geriatric ED (GED) guidelines and the American College of Emergency Medicine’s Geriatric Emergency Department Accreditation (GEDA) program (https://www.acep.org/geda/).3 Although the GED guidelines recommend “routine screening for all geriatric patients for high-risk features,” there are no screening tests that effectively predict poor outcomes among older patients who seek ED care.4 The currently used screening tests measure risk of a composite of poor outcomes such as functional decline, death, and depression after an ED visit. Alternatively, frailty might be used to identify older adults who might benefit from specialized GED care. Frailty is a state of heightened vulnerability to stressors arising from impairments in multiple systems leading to declines in homeostatic reserve and resiliency. Frailty predicts poor outcomes for older adults after acute care. However, it is not a concept familiar to many ED clinicians. The Clinical Frailty Scale (CFS) is a judgment-based measure of fitness and frailty that uses pictographs and clinical descriptions to stratify older adults on a 9-point scale, with 1 defined as very fit and 9 as terminally ill.5 ED physician–assigned higher CFS is associated with hospitalization; however, it only has moderate inter-rater reliability between physicians and patients.6 The inter-rater reliability of the CFS between ED nurses and ED physicians in the United States is unknown. Recently, Ringer et al.7 demonstrated good inter-rater reliability between nurses and physicians/advanced practice providers for patients age 75 and older in a Canadian ED. There is no criterion standard for identifying frailty in the ED. However, good inter-rater reliability between nurses and an ED physician with specialized training in geriatric emergency medicine would support screening for frailty by ED nurses. The objective of this study was to estimate the inter-rater reliability of the CFS between ED nurses and an ED physician with expertise in geriatric emergency medicine. The CFS was included in the standard nursing assessment protocol for patients 65 years and older at Northwestern Memorial Hospital Emergency Department in August 2019. Prior to implementing CFS screening, ED nurses received educational materials on frailty, pocket cards with pictographs and clinical descriptions, and a short (5 minutes) educational session on frailty and the use of the CFS. The physician performing the study did not provide the training. We added pictographs and clinical descriptions into the electronic medical record to help nurses assign the appropriate score. This study was designed to estimate the inter-rater reliability of the CFS prior to the initiation of a larger trial (ClinicalTrials.gov Identifier NCT04115371). The institutional review board at Northwestern University determined that this study did not meet the definition of human subjects research. Clinical Frailty Scale assessments were completed by a nurse during bedside nursing assessment and subsequently completed by the physician who was blinded to the nurse’s assessment. Inter-rater reliability between the ED physician and nurses of the CFS was assessed based on the 9-point CFS scale using Cohen’s weighted kappa. CFS scores were also dichotomized using predefined cutoff scores of <4 (not frail) vs. ≥4 (vulnerable or frail), based on previous work demonstrating a median CFS of 3 for patients discharged home and a median CFS of 4 for hospitalized patients.6 The inter-rater reliability of this dichotomous classification was assessed using Cohen’s kappa. We also evaluated Kappa coefficients for subgroups defined by age (65–70, 71–80 and ≥80 years), sex, race (non-Hispanic white vs. all others), clinical acuity (Emergency Severity Index (ESI) categories 1–2 vs. 3–5), and time of day (nursing day shift (7:30 am to 7:30 pm) vs. overnight shift (7:30 pm to 7:30 am). Using Landis and Koch’s strength of agreement of Cohen’s kappa (κ) scores, values >0.80 indicated “Almost perfect” reliability, 0.61 to 0.80 “substantial,” 0.41 to 0.60 “moderate,” 0.21 to 0.40 “fair,” and 0.20 or below “poor.”8 We determined that a sample size of 100 would provide a two-sided 95% confidence interval (CI) with a lower bound of 0.515, which excludes κ = 0.5 We evaluated n = 100 patients aged 65 years and older, of whom 63% were female, 52% were nonwhite, and 51% were ≥75 years (the median age in this study population). One physician with expertise in geriatric emergency medicine and 38 nurses assigned CFS scores. For the entire sample and using the entire spectrum of CFS scores, the weighted kappa was 0.90 (95% CI = 0.85 to 0.94). The kappa was 0.80 (95% CI = 0.66 to 0.90) when categorizing CFS scores as <4 points (“not frail”) vs. ≥4 points (“vulnerable or frail”). In subgroup analyses, CFS inter-rater reliability was similar across time of day. We observed lower inter-rater reliability estimates across age, sex, race, and clinical acuity (ESI) subgroups. However, the subgroup reliability was better when we examined the CFS using all scores than when using <4 vs. ≥4. The minimum kappa coefficients exceeded 0.80 and 0.60, respectively (Table 1). Binary Kappa CFS < 4 vs. ≥4 (95% CI)* Weighted Kappa All CFS scores (95% CI)* Between the ED nurses and one ED physician, we observed almost perfect inter-rater reliability in the assessment of frailty using the CFS, among older adult patients seeking care in the ED. In subgroup analyses, inter-rater reliability did not vary substantially across age, sex, race, time of day, or patient acuity when the entire span of CFS scores was examined. All subgroup Kappa coefficients were “substantial” at minimum, with several categories within the “almost perfect” range. To our knowledge, this is the first report on CFS inter-rater reliability between a physician and nurses in a U.S. ED. Taken together with the Canadian study by Ringer et al.7 in patients age 75 and older, our study provides evidence supporting the inter-rater reliability between ED nurses and physicians of CFS to screening for frailty among older adults in the ED. Additionally, this study offers an analysis of the reliability associated with the CFS complementary to the analysis by Dresden et al.,6 which reported moderate reliability between physician- and patient-assigned CFS. A comparison of the results from these two studies suggests that nurses and physicians’ perspectives of patient frailty are better aligned than between physician and patient. The almost perfect inter-rater reliability observed in this study suggests that with the training and decisional support described in the methods, CFS assigned by an ED nurse closely matches that of an emergency physician with expertise in geriatric emergency medicine. Measuring frailty can identify older adults at high risk for poor outcomes, such as hospitalizations, functional decline, long-term care placement, and death.5 The CFS has been described as the most feasible measure of frailty for use in the ED.9 A previous study found that physician-assigned CFS was associated with hospitalization.6 Taken with previous findings, the results of our study demonstrating the excellent inter-rater reliability of ED nurse performed CFS suggests that it may be a useful tool to identify older patients at risk of poor outcomes and who may benefit from additional services or care coordination. Future research is needed to determine whether identifying older adults in the ED with frailty can improve outcomes. The GED guidelines suggest that older adults who are identified as high risk for poor outcomes “be referred to health care resources, both inpatient and outpatient, to help improve overall health and function.”3 Existing studies on GED programs including our own have shown that patients who are cared for through a multidisciplinary geriatric assessment and care-coordination team are less likely to be hospitalized than matched comparison patients.10 However, it is unknown whether systematically identifying vulnerable and frail older adults for additional GED services will lead to improved outcomes. Despite demonstrating that ED nurses have almost perfect inter-rater reliability with the CFS compared to an emergency physician specially trained in geriatric emergency medicine, this study has limitations. First, it is a single-site study at a top-tier “Level I” ACEP-accredited GED. The ED nurses in this study may have had additional knowledge obtained through clinical experience of identifying and referring older adults for additional ED services; therefore, the results of this study may not apply to other sites. Second, inter-rater reliability demonstrated between multiple nurses and a single physician with significant experience in geriatric emergency medicine may not be generalizable to all emergency physicians. Finally, our results do not reflect the validity of the CFS screen performed by ED nurses because there is no criterion standard for the measurement of frailty in the ED. In conclusion, inter-rater reliability of CFS between ED nurses and an ED physician with expertise in geriatric emergency medicine was near perfect for patients age 65 and older. Future studies should focus on identifying a criterion standard for the assessment of frailty among older ED patients and examine the relationship between the CFS and meaningful outcomes in this patient population, with particular attention to function and mobility.

Topics & Concepts