The Impact of Alternative Approaches to Diagnostic Yield Calculation in Studies of Bronchoscopy

Abstract

In recent years, there have been notable advancements in the guidance technologies for peripheral bronchoscopy, including newer electromagnetic navigation platforms, cone beam CT, augmented fluoroscopy, digital tomosynthesis, and robot-assisted bronchoscopy.1Chen A.C. Pastis Jr., N.J. Mahajan A.K. et al.Robotic bronchoscopy for peripheral pulmonary lesions: a multicenter pilot and feasibility study (BENEFIT).Chest. 2021; 159: 845-852Abstract Full Text Full Text PDF PubMed Scopus (42) Google Scholar, 2Fielding D.I.K. Bashirzadeh F. Son J.H. et al.First human use of a new robotic-assisted fiber optic sensing navigation system for small peripheral pulmonary nodules.Respiration. 2019; 98: 142-150Crossref PubMed Scopus (55) Google Scholar, 3Folch E.E. Pritchett M.A. Nead M.A. et al.Electromagnetic navigation bronchoscopy for peripheral pulmonary lesions: one-year results of the prospective, multicenter NAVIGATE study.J Thorac Oncol. 2019; 14: 445-458Abstract Full Text Full Text PDF PubMed Scopus (135) Google Scholar, 4Pritchett M.A. Schampaert S. de Groot J.A.H. Schirmer C.C. van der Bom I. Cone-beam CT with augmented fluoroscopy combined with electromagnetic navigation bronchoscopy for biopsy of pulmonary nodules.J Bronchology Interv Pulmonol. 2018; 25: 274-282Crossref PubMed Scopus (75) Google Scholar, 5Aboudara M. Roller L. Rickman O. et al.Improved diagnostic yield for lung nodules with digital tomosynthesis-corrected navigational bronchoscopy: initial experience with a novel adjunct.Respirology. 2020; 25: 206-213Crossref PubMed Scopus (27) Google Scholar, 6Cicenia J. Bhadra K. Sethi S. Nader D.A. Whitten P. Hogarth D.K. Augmented fluoroscopy: a new and novel navigation platform for peripheral bronchoscopy.J Bronchology Interv Pulmonol. 2021; 28: 116-123Crossref PubMed Scopus (9) Google Scholar, 7Benn B.S. Romero A.O. Lum M. Krishna G. Robotic-assisted navigation bronchoscopy as a paradigm shift in peripheral lung access.Lung. 2021; 199: 177-186Crossref PubMed Scopus (13) Google Scholar As a result, multiple recent trials evaluating the performance of these technologies have been reported. The most frequently employed assessment metric is diagnostic accuracy, frequently termed diagnostic yield (DY) in the bronchoscopy literature, yet there remains no standardized approach to measurement of this outcome in studies of bronchoscopy. The use of variable definitions of DY may influence the interpretation of study results and limit the ability to compare findings across studies. The aim of this analysis is to illustrate and quantify the impact of different methodological approaches on DY estimates. We generated a hypothetical cohort of 1,000 patients undergoing bronchoscopy by extrapolating data from a prospective study of robot-assisted bronchoscopy.1Chen A.C. Pastis Jr., N.J. Mahajan A.K. et al.Robotic bronchoscopy for peripheral pulmonary lesions: a multicenter pilot and feasibility study (BENEFIT).Chest. 2021; 159: 845-852Abstract Full Text Full Text PDF PubMed Scopus (42) Google Scholar We calculated DY, defined as the rate of true positives (TPs) (for malignancy) plus true negatives (TNs), using three alternative definitions commonly employed in the literature.1Chen A.C. Pastis Jr., N.J. Mahajan A.K. et al.Robotic bronchoscopy for peripheral pulmonary lesions: a multicenter pilot and feasibility study (BENEFIT).Chest. 2021; 159: 845-852Abstract Full Text Full Text PDF PubMed Scopus (42) Google Scholar,3Folch E.E. Pritchett M.A. Nead M.A. et al.Electromagnetic navigation bronchoscopy for peripheral pulmonary lesions: one-year results of the prospective, multicenter NAVIGATE study.J Thorac Oncol. 2019; 14: 445-458Abstract Full Text Full Text PDF PubMed Scopus (135) Google Scholar,8Ost D.E. Ernst A. Lei X. et al.Diagnostic yield and complications of bronchoscopy for peripheral lung lesions. results of the AQuIRE registry.Am J Respir Crit Care Med. 2016; 193: 68-77Crossref PubMed Scopus (248) Google Scholar In all methods, bronchoscopy findings were classified as either positive or negative for malignancy, based on pathology findings. All three methods classify a bronchoscopic procedure positive for malignancy as a TP. However, they vary in their approach to categorization of cases that are negative and handling of missing data, resulting in variability in DY estimates. To demonstrate these differences, we define cases negative for malignancy as either (1) a specific benign (SPB) diagnosis (eg, infection, granuloma); (2) a nonspecific benign (NSB) finding (eg, inflammation); or (3) a nondiagnostic (ND) result (eg, atypical cells, normal alveoli). Using this approach, the differences in the three methods are briefly summarized as follows: The primary feature of this approach is that DY estimates are determined with data available at the time of index bronchoscopy; this method does not allow for the inclusion of follow-up data. Only procedures with a specific benign diagnosis at index are considered TNs. Cases with an NSB diagnosis or a ND result are not considered as TNs. DY is calculated as (TP + SPB) divided by total procedures. The key feature of this approach is that the DY calculation allows only for the inclusion of follow-up data for cases with an NSB finding at bronchoscopy. NSB findings are assessed longitudinally and categorized as TNs only if subsequent biopsy or imaging confirms a nonmalignant diagnosis (NSBTN); cases in which a definitive diagnosis is not established because of lack of follow-up are considered nondiagnostic. ND cases at index cannot be considered as TNs in the final DY calculations regardless of results of subsequent imaging or biopsy. As in method 1, cases that result in a SPB diagnosis at index bronchoscopy are considered TNs. DY is calculated as (TP + SPB + NSBTN) divided by total procedures. In this approach, the DY calculation allows for the inclusion of follow-up data for all cases that are negative for malignancy at index bronchoscopy. Cases with either an SPB, NSB, or ND finding can be considered a TN (SPBTN + NSBTN or NDTN) if either of the following is true: a subsequent biopsy confirms a definitive nonmalignant diagnosis or there is imaging evidence of benign disease (ie, lack of lesion progression). DY is calculated as (TP + SPBTN + NSBTN + NDTN) divided by total procedures with longitudinal data and excludes index NSB, SPB, or ND cases that lack follow-up data) (labeled method 3A in Table 1). Two iterations of this approach have been used in the literature: (1) Method 3B—patients that are lost to follow-up (LTFU) are considered ND and included only in the denominator; DY is calculated as (TP + SPBTN + NSBTN + NDTN) divided by total procedures; (2) Method 3C—index SPB, NSB, or ND cases that are LTFU are considered as TNs; DY is calculated as (TP + SPBTN + NSBTN + NDTN + LTFU) divided by total procedures.Table 1Bronchoscopy Findings and Calculation of DY in the Hypothetical CohortClassification of TP, TN, and Denominator for DY CalculationFindings at Bronchoscopy (N = 1,000)No.Method 1Method 2Method 33A3B3CMalignant611611 (TP)611 (TP)611 (TP)611 (TP)611 (TP)Nonmalignant389NANANANA Specific benign (SPB)565656565656 Nonspecific benign (NSB)111aOf the 111 patients with NSB findings at bronchoscopy, 74 were categorized as TNs based on subsequent biopsy or repeat imaging (labeled NSBTN).NA74 (NSBTN)74 (NSBTN)74 (NSBTN)74 (NSBTN) Nondiagnostic (ND)222bOf the 222 patients with ND findings at bronchoscopy, 56 were determined to have benign disease based on subsequent biopsy or repeat imaging (labeled NDTN).NANA56 (NDTN)56 (NDTN)56 (NDTN)Numerator: TP + TN667741797797889Denominator: total procedures (TO)1,0001,000908cMethod 3A excludes 92 patients from the denominator that were lost to follow-up.1,0001,000Diagnostic yield, %66.7dDY (method 1) = (TP + SPB)/TO = (611 + 56)/1,000.74.1eDY (method 2) = (TP + SPB + NSBTN)/TO = (611 + 56 + 74)/1,000.87.8fDY (method 3A) = (TP + SPB + NSBTN + NDTN)/(TO-LTFU) = (611 + 56 + 74 + 56)/(1,000 − 92).79.8gDY (method 3B) = (TP + SPB + NSBTN + NDTN)/TO = (611 + 56 + 74 + 56)/(1,000).88.9hDY (method 3C) = (TP + SPB + NSBTN + NDTN + LTFU)/TO = (611 + 56 + 74 + 56 + 92)/(1,000).DY = diagnostic yield; LTFU = lost to follow-up; NA = not applicable; ND = nondiagnostic; NDTN = nondiagnostic true negative; NSB = nonspecific benign; NSBTN = nonspecific benign true negative; SPB = specific benign; TN = true negative; TO = total procedures; TP = true positive.a Of the 111 patients with NSB findings at bronchoscopy, 74 were categorized as TNs based on subsequent biopsy or repeat imaging (labeled NSBTN).b Of the 222 patients with ND findings at bronchoscopy, 56 were determined to have benign disease based on subsequent biopsy or repeat imaging (labeled NDTN).c Method 3A excludes 92 patients from the denominator that were lost to follow-up.d DY (method 1) = (TP + SPB)/TO = (611 + 56)/1,000.e DY (method 2) = (TP + SPB + NSBTN)/TO = (611 + 56 + 74)/1,000.f DY (method 3A) = (TP + SPB + NSBTN + NDTN)/(TO-LTFU) = (611 + 56 + 74 + 56)/(1,000 − 92).g DY (method 3B) = (TP + SPB + NSBTN + NDTN)/TO = (611 + 56 + 74 + 56)/(1,000).h DY (method 3C) = (TP + SPB + NSBTN + NDTN + LTFU)/TO = (611 + 56 + 74 + 56 + 92)/(1,000). Open table in a new tab DY = diagnostic yield; LTFU = lost to follow-up; NA = not applicable; ND = nondiagnostic; NDTN = nondiagnostic true negative; NSB = nonspecific benign; NSBTN = nonspecific benign true negative; SPB = specific benign; TN = true negative; TO = total procedures; TP = true positive. The classification of biopsy findings within the cohort at the time of index bronchoscopy and with longitudinal assessment are provided in Table 1. At index bronchoscopy, 61.1% (611 of 1,000) were diagnosed with malignant disease, and 38.9% (389 of 1,000) were negative for malignancy, 5.6% (56 of 1,000) of which had an SPB diagnosis. A total of 9.2% (92 of 1,000) were lost to follow-up; the remaining cases that were negative for malignancy at index were assessed at 1 year to determine a final clinical diagnosis (malignant vs benign). The DY estimates based on the three primary approaches were 66.7% (method 1), 74.1% (method 2), and 87.8% (method 3A). This evaluation demonstrates that applying different approaches to the calculation of DY results in estimates that can differ by more than 20% on an absolute basis. The variability in DY estimates is driven by the characterization and approach to classification of nonmalignant findings at bronchoscopy. Method 1 is the most conservative, allowing only definitive index results to be included in the numerator and does not require any longitudinal information to inform the reference standard for DY calculation, whereas method 2 includes certain nonspecific benign findings as TNs with appropriate follow-up information. Method 3 is the most liberal, because it allows for the greatest latitude in considering nonmalignant findings as TNs, including findings that may not reflect the pathology of the lesion of interest. The different iterations of method 3 that employ alternative approaches to the handling of missing data resulted in estimates that varied by 9%, a magnitude equivalent to the rate of cases that were lost to follow-up in this scenario. Method 3 likely results in biased estimates of DY given the inclusion of ND procedures as true negatives. Although method 1 is the most conservative, it may underestimate DY given the inability to use longitudinal information to establish whether some NSB findings represent TN procedures. Our evaluation was limited to a single simulated cohort with a cancer prevalence of approximately 72% and missing follow-up in 9% of the study sample. Additional analyses are needed to examine the impact of different methodologies of DY estimation across broader scenarios of test sensitivity, cancer prevalence, and loss to follow-up rates to advance our understanding on this topic. Use of other measures, such as sensitivity for malignancy, that may be less affected by methodological approaches to DY calculation also need to be explored. Ultimately, consensus definitions on the calculation and reporting of DY are necessary to ensure scientific and robust comparisons across different studies and technologies. In particular, standardization of the handling of NSB and ND results at index will improve the transparency and comparability of diagnostic bronchoscopy studies. Given the importance of longitudinal assessment, studies also must employ rigorous methods to capture follow-up information in patients without definitive results at index bronchoscopy; follow-up periods that allow determination of the true diagnosis among patients without definitive findings at index are critical for calculating unbiased estimates of bronchoscopy performance.9Bossuyt P.M. Cohen J.F. Gatsonis C.A. Korevaar D.A. group S. STARD 2015: updated reporting guidelines for all diagnostic accuracy studies.Ann Transl Med. 2016; 4: 85PubMed Google Scholar,10Cohen J.F. Korevaar D.A. Altman D.G. et al.STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration.BMJ Open. 2016; 6e012799Crossref Scopus (827) Google Scholar

Topics & Concepts