Does the Introduction of American College of Surgeons NSQIP Improve Outcomes? A Systematic Review of the Academic Literature

Abstract

Surgery is a central part of healthcare. It is estimated that approximately 11% of the global burden of diseases can be addressed wholly or partly by surgical care.1Ozgediz D. Jamison D. Cherian M. McQueen K. The burden of surgical conditions and access to surgical care in low- and middle-income countries.Bull World Health Organ. 2008; 86: 646-647Crossref PubMed Scopus (193) Google Scholar By its very nature, surgery is considered an invasive procedure and therefore carries inherent risks for patients. Based on World Health Organization data, the crude death rate after major surgery is 0.5% to 5%; up to 25% of surgical inpatients experience post-surgery complications, and surgical care accounts for almost half of all adverse events for inpatients in western countries.2Slawomirski L. Auraaen A. Klazinga N. The economics of patient safety: strengthening a value-based approach to reducing patient harm at national level. OECD, Paris2017http://www.oecd.org/els/health-systems/The-economics-of-patient-safety-March-2017.pdfDate accessed: July 26, 2019Google Scholar There is no shortage of data to document the significant costs associated with surgical complications.3de Vries E.N. Ramrattan M.A. Smorenburg S.M. et al.The incidence and nature of in-hospital adverse events: a systematic review.Qual Saf Health Care. 2008; 17: 216-223Crossref PubMed Scopus (927) Google Scholar, 4Van Den Bos J. Karan Rustagi K. Gray T. et al.The $17.1 billion problem: the annual cost of measurable medical errors.Health Affairs. 2011; 30: 596-603Crossref PubMed Scopus (174) Google Scholar, 5WHO Guidelines for Safe Surgery. Safe Surgery Saves Lives. 978 92 4 159855 2 World Health Organization, Geneva, Switzerland2009Google Scholar The World Health Organization estimated that half of surgery-associated harm is preventable.2Slawomirski L. Auraaen A. Klazinga N. The economics of patient safety: strengthening a value-based approach to reducing patient harm at national level. OECD, Paris2017http://www.oecd.org/els/health-systems/The-economics-of-patient-safety-March-2017.pdfDate accessed: July 26, 2019Google Scholar Since its creation in 1913, the American College of Surgeons (ACS) has been relentless in its pursuit of the improvement of health outcomes of patients undergoing surgery. One of the major achievements of the ACS was the creation, testing, and validation of a system that captures and reports risk adjusted outcomes of surgical interventions: The National Surgical Quality Improvement Program (NSQIP). The trigger for the creation of the initial version of NSQIP was a mandate in 1985 from the US federal government (US Congress of Public Law No. 99-166) to monitor surgical outcomes in 133 Veterans Affairs Hospitals.6Stremple J.F. Bross D.S. Davis C.L. McDonald G.O. Comparison of postoperative mortality in VA and private hospitals.Ann Surg. 1993; 217: 277-285Crossref PubMed Scopus (37) Google Scholar In collaboration with the ACS, the Department of Veterans Affairs conducted the National Veterans Administration Surgical Risk Study between October 1, 1991 and December 31, 1993, with the goal to define performance indicators and clinical variables and delineate how risk adjustment would be conducted. In 1994, the first version of NSQIP was introduced only in Veteran Affairs hospitals, which enabled risk-adjusted comparison of surgical complications to be made across 133 hospitals.7Stremple J.F. The historical evolution of the Department of Veterans Affairs National Surgical Quality Improvement Program.J Am Coll Surg. 2011; 213: 567-571Abstract Full Text Full Text PDF PubMed Scopus (7) Google Scholar This original version of NSQIP collected information on post-surgical outcomes and provided surgeon- and hospital-specific scorecards based on objective and rigorous analysis of the data. In light of the patient outcomes data generated by NSQIP, a number of quality improvement initiatives were put in place that led to reductions in mortality and morbidity in the VA hospital system, by 27% and 45%, respectively.8Khuri S.F. Henderson W.G. Daley J. et al.Successful implementation of the Department of Veterans Affairs' National Surgical Quality Improvement Program in the private sector: The Patient Safety in Surgery study.Ann Surg. 2008; 248: 329-336Crossref PubMed Scopus (459) Google Scholar Subsequently, the NSQIP program was expanded to include private sector hospitals, with the Patient Safety in Surgery (PSS) study, which ran in parallel in the VA and a set of non-VA hospitals in 2001 to 2004, and showed clear and definite improvements in patient outcomes in both systems.8Khuri S.F. Henderson W.G. Daley J. et al.Successful implementation of the Department of Veterans Affairs' National Surgical Quality Improvement Program in the private sector: The Patient Safety in Surgery study.Ann Surg. 2008; 248: 329-336Crossref PubMed Scopus (459) Google Scholar In 2004, NSQIP was officially branded as ACS-NSQIP. In 2005, participation in the private sector program sponsored by the American College of Surgeons (ACS-NSQIP) became available by subscription, while at the same time, the VA program separated from the private sector program and became the VA-sponsored version: VA-SQIP. Since then, the number of hospitals adopting the program has increased on a yearly basis. Over the past 13 years, the number of surgical sites that have adopted ACS-NSQIP has seen an average yearly increase of 38%, from 37 sites in 2006 to 722 sites in 2019.9American College of SurgeonsAmerican College of Surgeons National Surgical Quality Improvement Program 2019. ACS NSQIP Semiannual Report, July 12, 2019, Chicago.https://www.facs.org/quality-programs/acs-nsqipDate accessed: July 31, 2020Google Scholar While most of the surgical facilities are US-based (596, 82%), an increasing number of international sites have adopted ACS-NSQIP. To date these include Canada, Australia, and the United Kingdom, with 96, 16, and 3 hospital sites, respectively.10American College of Surgeons National Surgical Quality Improvement ProgramNSQIP participating hospitals.https://www.facs.org/quality-programs/acs-nsqip/about/participantsDate: 2020Date accessed: July 31, 2020Google Scholar With the increasing adoption of such a data-rich platform, it is not surprising that the past 13 years have seen an increase in the number of academic publications using ACS-NSQIP data. While most of the publications used ACS-NSQIP data to measure quality improvement at specific surgical sites and within surgical specialties, a significant number of studies have investigated the reliability of the ACS-NSQIP surgical risk calculator, an online tool that uses surgery-related data collected in ACS-NSQIP to predict postoperative outcomes and inform surgical decision-making. The prime objective of this review was to assess whether the introduction of ACS-NSQIP in surgical sites is associated with improved patient outcomes compared with other sites that have not adopted the ACS-NSQIP program, or the same site before the introduction of the ACS-NSQIP program. A secondary objective of our review was to estimate the costs/cost savings associated with implementation of the ACS-NSQIP program. We conducted a comprehensive systematic review of articles that assess the effectiveness of ACS-NSQIP, published between the program's inception in 2005 to December 2018. The review was carried out following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines.11Moher D. Liberati A. Tetzlaff J. Altman D.G. The PRISMA Grouppreferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement.PLoS Med. 2009; 6e1000097Crossref PubMed Scopus (30662) Google Scholar A PRISMA flow diagram depicting the strategy for selecting relevant articles is given in Figure 1. The following biomedical and health-related science databases were selected for the literature search: Medline, PsychInfo, EMBASE (all 3 via OVID), CINAHL (via EBSCO), Web of Science, Scopus, ProQuest, and the Cochrane database of systematic reviews and controlled trials. To ensure the comprehensive capture of all relevant studies, citation searching was performed on key studies and review articles to identify additional eligible studies. Databases were screened using a search strategy developed by a research librarian in partnership with the research team using "National Surgical Quality Improvement Program.mp" OR "NSQIP.mp" as search terms. To narrow our search to a feasible number of articles for screening, we added the search terms "evaluat∗" OR "assess∗" to the search. Only articles written in the English language and published or accepted for publication between 2005 and 2018 were considered. No restriction on publication format was applied. All abstracts and articles obtained were imported into EndNote reference manager to facilitate removal of duplicates. Duplicates were removed in 2 steps; identical duplicates were removed using the "find duplicate" command in EndNote. In a second step to remove duplicates, articles were checked alphabetically by first author name to facilitate identification of full publication, duplicate abstracts, and abstracts of full lengths publications with the same first author. The remaining articles were screened independently by 2 investigators (BCS, SB) by title and/or abstract to exclude records according to our selection criteria and to exclude further duplicates, such as abstracts with the same contents and abstracts of full publications with a different name of first author. All articles that met the inclusion criteria or could not be excluded based on the information available in the title and abstracts were read in full to assess eligibility and inclusion in the final review (Fig. 1, PRISMA Flow Diagram). This last round of screening involved 2 independent investigators (BCS, SB). A Cohen's kappa statistic of 0.83 reflected excellent agreement in interrater reliability.12Cohen J. A coefficient of agreement for nominal scales.Education psychologic measurement. 1960; 20: 37-46Crossref Scopus (23344) Google Scholar Discrepancies in screening were resolved by consensus. When disagreement persisted, a third investigator (MB) made the final determination. To ensure that an objective, consistent, and rigorous methodology was used to select relevant peer-reviewed articles, we adopted the PICOS framework to define our inclusion criteria. PICOS elements include: Population, Intervention, Comparison, Outcome, and Study Type.1.Population: The population of interest was all studies describing hospitals or clinics that used the ACS-NSQIP to evaluate the quality of their surgical outcome.2.Intervention: For this review, we included studies with any type of surgery that was followed and evaluated using the ACS-NSQIP program.3.Comparison: Eligible as study comparators were hospitals that did not implement ACS-NSQIP or studies comparing hospitals before and after adoption of ACS-NSQIP.4.Outcome: Relevant outcomes were quality improvement in surgery such as reduced adverse events postoperatively and improved mortality and morbidity rates. Reduced wait time and cost effectiveness could also be assessed.5.Study type: All comparative study designs (before vs after ACS-NSQIP adoption and ACS-NSQIP participating vs nonparticipating institutions) were eligible for inclusion. We applied the following exclusion criteria: Articles that only used ACS-NSQIP to identify and recruit subjects for specific research purposes; studies that use the ACS-NSQIP for validation of other programs or for validation of specific procedures; nonoriginal publications such as systematic reviews and meta-analysis evaluating the ACS-NSQIP were excluded, but the citations in these reviews were cross-checked with our selection for eligibility in our study; commentaries, letters, and abstracts without sufficient data justifying their findings; and papers that only assessed the ACS-NSQIP risk calculator. A standardized data collection form was developed for data abstraction from the final selected articles. Data were entered into an Excel database for cleaning and analysis. Data were extracted by 1 person (BCS); 2 content experts (SB, MB) then independently verified the extracted data. Not all papers reported on the start date of joining the ACS-NSQIP program. As we considered this an important variable in our analysis, we contacted authors to supply this information if it was not provided in the paper. We have indicated in Table 1 whenever the ACS-NSQIP start date was supplied by the author.Table 1Summary Characteristics of Included PapersFirst author, yLocationNumber of hospitalsDate of ACS NSQIP participation∗Identifies ACS NSQIP start dates as communicated by the author.Study periodStudy designData sourceSample sizeOutcome focusThanh23Thanh N.X. Baron T. Litvinchuk S. An economic evaluation of the National Surgical Quality Improvement Program (NSQIP) in Alberta, Canada.Ann Surg. 2019; 269: 866-872Crossref PubMed Scopus (12) Google Scholar2019Canada520152015–2017Pre-/post-QI interventionCanadian healthcare and finance department22,487Clinical impact and costVan Katwyk22Van Katwyk S. Thavorn K. Coyle D. et al.The return of investment of hospital-based surgical quality improvement programs in reducing surgical site infection at a Canadian tertiary-care hospital.Infect Control Hosp Epidemiol. 2019; : 25-132Google Scholar 2018Canada1May 2010Apr 2010–Jan 2015Pre-/post-QI interventionACS NSQIP and TOH Data WarehouseNot specifiedCost analysis of SSI prevention programNimeri24Nimeri A.A. Bautista J. Philip R. Reducing healthcare costs using ACS NSQIP-driven quality improvement projects: A success story from Sheikh Khalifa Medical City (SKMC).World J Surg. 2019; 43: 331-338Crossref PubMed Scopus (6) Google Scholar2018UAE120062009–2015Pre-/post-QI interventionACS NSQIP8,842Complication rate and cost analysisCromeens31Cromeens B.P. Lisciandro R.E. Brilli R.J. et al.Identifying adverse events in pediatric surgery: Comparing morbidity and mortality conference with the NSQIP-pediatric system.J Am Coll Surg. 2017; 224: 945-953Abstract Full Text Full Text PDF PubMed Scopus (11) Google Scholar 2017US1Pre-2010Jan 2010–Sep 2015Comparison of NSQIP-P and M&M conference dataNSQIP P and M&M conference2,973 (NSQIP), 9,603 (M&M)Adverse events post-various operationsBenlice38Benlice C. Gorgun E. Using NSQIP data for quality improvement: The Cleveland Clinic SSI experience.Semin Colon Rectal Surg. 2016; 27: 74-82Crossref Scopus (7) Google Scholar 2016US120052005–2010, Feb 2013–Feb 2015Comparison of hospital with ACS NSQIP cohort, pre-/post-QI interventionACS NSQIP2,279Exploring high SSI rate, SSI post-colorectal operationCohen27Cohen M.E. Liu Y. Ko C.Y. Hall B.L. Improved surgical outcomes for ACS-NSQIP hospitals over time: Evaluation of hospital cohorts with up to 8 years of participation.Ann Surg. 2016; 263: 267-273Crossref PubMed Scopus (142) Google Scholar 2016US515Various times between 2006-20132006–2013NSQIP participation over timeACS NSQIP2,941,845Postoperative adverse events, mortality, and SSIDeHaas34DeHaas D. Aufderheide S. Gano J. et al.Colorectal surgical site infection reduction strategies.Am J Surg. 2016; 212: 175-177Abstract Full Text Full Text PDF PubMed Scopus (12) Google Scholar 2016US1Pre-20112011 vs 2014Pre-/post-QI interventionACS NSQIP531SSI post-colorectal operationEtzioni29Etzioni D.A. Wasif N. Dueck A.C. et al.Association of hospital participation in a surgical outcome monitoring program with inpatient complications and mortality.JAMA. 2015; 313: 505-511Crossref PubMed Scopus (123) Google Scholar 2015US113, 44 NSQIP, 69 non-NSQIPPre-2009Jan 2009–Jul 2013Comparison of ACS NSQIP participation vs nonparticipationUniversity Health System Consortium345,357Adverse events and mortality post-general and vascular operationOsborne21Osborne N.H. Nicholas L.H. Ryan A.M. et al.Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries.JAMA. 2015; 313: 496-504Crossref PubMed Scopus (184) Google Scholar 2015US789,263 NSQIP, 526 non-NSQIPPre-20032003–2012Pre-/post-NSQIP, comparison of ACS NSQIP participation vs nonparticipationNational Medicare provider430,179; 1,226,479Postoperative complication and mortality, and hospital costLucas28Lucas D.J. Pawlik T.M. Quality improvement in gastrointestinal surgical oncology with American College of Surgeons National Surgical Quality Improvement Program.Surgery. 2014; 155: 593-601Abstract Full Text Full Text PDF PubMed Scopus (34) Google Scholar 2014US316Pre-2006 (n=121), 2006 (n = 62), 2007 (n = 28), 2008 (n = 26), 2009 (n = 21), 2010 (n = 58)2006–2011NSQIP participation over timeACS NSQIP participant use file76,076Complication and mortality post-gastrointestinal oncology operationTepas39Tepas 3rd, J.J. Kerwin A.J. deVilla J. Nussbaum M.S. Macro vs micro level surgical quality improvement: A regional collaborative demonstrates the case for a national NSQIP initiative.J Am Coll Surg. 2014; 218: 599-604Abstract Full Text Full Text PDF PubMed Scopus (10) Google Scholar 2014US54, 6 NSQIP, 48 'lite' versionPre-2011Oct 2011–Jul 2012Pre-/post-QI interventionACS NSQIP semiannual reports38,896Adverse events post-colorectal operationMcNelis35Lutfiyya W. Parsons D. Breen J. A colorectal "care bundle" to reduce surgical site infections in colorectal surgeries: A single-center experience.The Permanente Journal. 2012; 16: 10-16PubMed Google Scholar 2014US12010Jan 2010–July 2012Pre-/post-QI interventionACS NSQIP1,081Postoperative pneumonia, prolonged ventilation, and costCima33Cima R. Dankbar E. Lovely J. et al.Colorectal surgery surgical site infection reduction program: A National Surgical Quality Improvement Program-driven multidisciplinary single-institution experience.J Am Coll Surg. 2013; 216: 23-33Abstract Full Text Full Text PDF PubMed Scopus (154) Google Scholar2013US1Pre-20092009–2011Pre-/post-QI interventionACS NSQIP729SSI post-colorectal surgeryCompoginis40Compoginis J.M. Katz S.G. American College of Surgeons National Surgical Quality Improvement Program as a quality improvement tool: A single institution's experience with vascular surgical site infections.Am Surg. 2013; 79: 274-278Crossref PubMed Google Scholar 2013US12007Jan 2009–Dec 2010Pre-/post-QI interventionACS NSQIP478SSI post-vascular surgeryCeppa45Ceppa E.P. Pitt H.A. House M.G. et al.Reducing surgical site infections in hepatopancreatobiliary surgery.Hepato-Pancreato-Biliary J. 2013; 15: 384-391Abstract Full Text Full Text PDF Scopus (59) Google Scholar2012US12007Jan 2007–Dec 2009Pre-/post-QI interventionACS NSQIP895SSI, length of stay, readmission, and cost post-hepatopancreato-biliary operationFuchshuber37Fuchshuber P.R. Greif W. Tidwell C.R. et al.The power of the National Surgical Quality Improvement Program--achieving a zero-pneumonia rate in general surgery patients.Perm J. 2012; 16: 39-45Crossref PubMed Scopus (98) Google Scholar 2012US12006∗Identifies ACS NSQIP start dates as communicated by the author.2008–2009Comparison of hospital with ACS NSQIP cohort, pre-/post-QI interventionACS NSQIP raw and risk-adjusted data3,100Postoperative pneumonia, prolonged intubationGuillamonde-gui20Guillamondegui O.D. Gunter O.L. Hines L. et al.Using the National Surgical Quality Improvement Program and the Tennessee Surgical Quality Collaborative to improve surgical outcomes.J Am Coll Surg. 2012; 214: 709-714Abstract Full Text Full Text PDF PubMed Scopus (119) Google Scholar2012US10, TSQCPre-2009Jan 2009–Dec 2010Pre-/post-TSQC intervention; comparison of 10 TSQC hospitals with ACS NSQIP cohortACS NSQIP29,106Postoperative complication, mortality, and costLutfiyya35Lutfiyya W. Parsons D. Breen J. A colorectal "care bundle" to reduce surgical site infections in colorectal surgeries: A single-center experience.The Permanente Journal. 2012; 16: 10-16PubMed Google Scholar 2012US12006Jan 2006–Jun 2011Pre-/post-QI interventionACS NSQIP624SSI post-colorectal operationWick41Wick E.C. Hobson D.B. Bennett J.L. et al.Implementation of a surgical comprehensive unit-based safety program to reduce surgical site infections.J Am Coll Surg. 2012; 215: 193-200Abstract Full Text Full Text PDF PubMed Scopus (152) Google Scholar2012US1July 2009 (high-risk pilot NSQIP)Jul 2009–Jul 2011Pre-/post-QI interventionACS NSQIP602SSI post-colorectal et to measure surgical Comparison of the for and Quality Patient Safety and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single 2011; Full Text Full Text PDF PubMed Scopus (154) Google of NSQIP-P and NSQIP and of postoperative complication of adverse L. et of the National Surgical Quality Improvement Surg. 2011; PubMed Scopus (37) Google Scholar NSQIP and hospital data events and in general and vascular M.G. surgical site Using National Surgical Quality Improvement Program data to surgical care improvement in surgical outcomes.J Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google Scholar 2006–Jun interventionACS post-colorectal R.J. K. J. NSQIP A tool for quality Surg. PubMed Scopus Google Scholar of hospital pre-/post-QI with ACS NSQIP cohortACS from complication, mortality, length of stay, and cost in and S.M. M. program for the inpatient surgical Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google NSQIP of the B.L. K. et surgical quality improve in the American College of Surgeons National Surgical Quality Improvement An evaluation of all participating hospitals.Ann Surg. 2009; PubMed Scopus Google (n = 2006 (n = 2007 (n = NSQIP participation over timeACS mortality and et al.Implementation of the National Surgical Quality Improvement to success for and J 2009; PubMed Scopus Google Scholar of hospital pre-/post-QI to ACS NSQIP cohortACS NSQIP, of and Health of events, rate of S.F. Henderson W.G. Daley J. et al.Successful implementation of the Department of Veterans Affairs' National Surgical Quality Improvement Program in the private sector: The Patient Safety in Surgery study.Ann Surg. 2008; 248: 329-336Crossref PubMed Scopus (459) Google VA and times of ACS NSQIP participation between VA and NSQIP and ACS NSQIP mortality and morbidity after general or vascular B.L. M. Y. et and in Veterans Affairs and selected medical of the Patient Safety in Surgery Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google VA and times of ACS NSQIP participation between VA and study events and W.G. S.F. C. et of risk-adjusted postoperative mortality and morbidity in Department of Veterans Affairs hospitals and selected medical surgical in Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google Scholar VA and times of ACS NSQIP participation between VA and study and mortality in general et of risk-adjusted postoperative mortality and morbidity in Department of Affairs hospitals and selected medical surgical in Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google Scholar VA and times of ACS NSQIP participation between VA and study and mortality in vascular et of National Surgical Quality Improvement Program data as a for quality Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google and not variable for interventionACS for not postoperative adverse et of surgical complications and An of the surgical morbidity and mortality conference compared with the American College of Surgical Quality Improvement Program.J Am Coll Surg. Full Text Full Text PDF PubMed Scopus Google of NSQIP-P and M&M conference NSQIP-P and M&M complication and American College of for and Quality Patient Safety morbidity and NSQIP NSQIP private Patient Safety in quality Surgical Improvement surgical site Tennessee Surgical Quality The United Veterans ACS NSQIP start dates as communicated by the author. in a ACS, American College of for and Quality Patient Safety morbidity and NSQIP NSQIP private Patient Safety in quality Surgical Improvement surgical site Tennessee Surgical Quality The United Veterans Affairs. in the data were Data were according to study type and comparison To the studies, we used in incidence rate as the outcome We this measure it was used by most studies to on in vs quality improvement the rate was not provided in the we Data were to evaluate use and effectiveness of the ACS-NSQIP in surgical With to the economic of ACS-NSQIP, we cost analysis, analysis, and analysis, as in the studies. In the effectiveness included surgical complications by the NSQIP, and the cost effectiveness was as an cost surgical complication In the surgical complications were into a in terms and the were as a between costs of NSQIP and the cost In cost surgical complications were not but in health use in between NSQIP and or and In this review, outcomes of interest were investment costs = and return on investment = if the number of patients was we and for All were to US For studies performed in Canada, we first all to Canadian using the of of accessed: 2020Google Scholar and then into US using a rate of Canadian = US of

Topics & Concepts