ACS-NSQIP risk calculator predicts cohort but not individual risk of complication following colorectal resection

doi:10.1016/j.amjsurg.2018.11.017

The American Journal of Surgery

Volume 218, Issue 1, July 2019, Pages 131-135

The manuscript was a poster presentation at The American Society of Colon and Rectal Surgeons 2018 ASCRS Annual Scientific Meeting in Nashville TN May 19–23.

https://doi.org/10.1016/j.amjsurg.2018.11.017 Get rights and content

Highlights

•
The calculator underpredicts “serious complication” following colorectal surgery.
•
The calculator did not perform better than simply assigning patients cohort risk.
•
The calculator predicted risk better for some surgeons than for others.
•
It is harder to predict poor outcomes in lower risk patients.

Abstract

Objective

Compare the ACS-NSQIP risk calculator with institutional risk for colorectal surgery.

Methods

Actual and predicted outcomes were compared for both cohort and individuals.

Results

For the cohort, the risk calculator was accurate for 7/8 outcomes; there were more serious complications than predicted (19.4 vs 14.7%, p < 0.05). Risk calculator Brier scores and null Brier scores were comparable.

Patients

with better outcomes than predicted were current smokers (OR 4.3 95% CI 1.2–15.4), ASA ≥ 3 (OR 10.4, 95% CI 2.8–39.2), underwent total/subtotal colectomy (OR 3.5, 95% CI 1.1–12.2) or operated by Surgeon 2 (OR 2.9, 95% CI 1.4–11.6). Patients with serious complications who had low predicted risk had low ASA (OR 10.5, 95% CI 1.3–82.6), and underwent operation by Surgeon 2 (OR 11.8, 95% CI 2.5, 55.2).

Limitations

Single center study, sample size may bias subgroup analyses.

Conclusions

The ACS NSQIP calculator did not predict outcome better than sample risk.

Introduction

Complications are common in colorectal surgery, experienced by up to 40% of patients at substantial cost.¹^,² Surgeon intuition does not predict postoperative morbidity,³ so there is a clear role for clinical calculators. Accurately predicting which patients are at risk for complication is essential both patients and providers as observed versus expected performance will be increasingly used for payment.⁴^,⁵ Risk calculators have been shown to improve predictions compared to provider intuition alone.6, 7, 8, 9, 10 Risk calculators have the potential to inform shared-decision making between patients and clinicians by facilitating communication about risk.¹¹ They can also be used for global benchmarking of surgical quality.¹² Despite these benefits, implementation has been limited due to unwieldiness.⁶

The ACS-NSQIP calculator is a unique tool built using aggregate data from 2.7 million operations to create a “universal” calculator that purports to address some of the logistical shortcomings of older calculators as it is available free online and uses variables that are commonly present in administrative databases.¹³ It predicts risk of 15 major outcomes within the thirty days following surgery. In initial validation studies, the ACS-NSQIP calculator outperformed procedure-specific calculators and had a low Brier Score¹⁴^,¹⁵ (where Brier scores compare a continuous prediction with a binary outcome, so 0 is a perfect model and 1 is a model that is always wrong.) Two small single institution studies of colorectal procedures found that the ACS-NSQIP calculator accurately predicted most adverse outcomes, though each found areas where it fell short.¹⁶^,¹⁷ Critiques of the risk calculator have included that it underestimates risk for ulcerative colitis patients and following proctectomy,¹⁷^,¹⁸ and variables that may be important in assessing risk such as preoperative laboratory evaluation are not included.¹⁹ Although imperfect, the ACS-NSQIP calculator has been found to compare favorably to other similar tools²⁰ which has motivated investigators to propose modifications to improve it.²¹

Our primary goal was to assess the predictive capability of the risk calculator for patients undergoing colorectal resection on both a population level using observed to expected calculations and on an individual using a comparative Brier score. Our hypothesis was that the ACS-NSQIP calculator would perform better than simply assigning each patient the overall sample risk of the population. Our secondary goal was to analyze the subsets of patients for whom the calculator performs best and worst in order to generate hypotheses as to which patients has best predictive capability. Investigations into when the ACS-NSQIP calculator has utility and conversely where it falls short will assist in the application of this tool into situations in which it has clinical merit.

Section snippets

Methods

Patients who were admitted to the Division of Colorectal Surgery at this institution following colorectal resection between October 2015 and September 2016 were identified from a prospectively maintained institutional review board-approved outcomes database, Columbia Colorectal Surgical Outcomes Database (CCSOD). Complication data is entered into CCSOD 30 days post-operatively and charts were reanalysed for missed complication at time of enrolment into this study. Analysis was limited to

Results

Of 288 colorectal resections, 78% were laparoscopic and cancer was the most common indication for surgery (52.4%) (Table 1). The four surgeons represented in the sample differed with respect to operative indication, case-mix and operative approach (Table 2). The ACS-NSQIP calculator accurately predicted cohort rates for seven of the eight outcomes considered: any complication, surgical site infection, venous thromboembolism, readmission, reoperation, and death. Serious complications were under

Discussion and conclusion

In this study, the ACS-NSQIP calculator predicted cohort risk of most outcomes, with the notable exception of under predicting serious complication. However, for individual patients, the ACS NSQIP calculator did not perform better than the null calculator in predicting patient risk of major complication following colorectal resection as measured by both a comparative Brier score and receiver operator curves. Most prior studies validating the calculator looked at its predictive capability on a

Author contributions

Study design: all authors; Data Acquisition: Hyde, Valizadeh, Data analysis and interpretation: Hyde, Al-Mazrou, Kiran; Manuscript drafting and final approval: all authors.

Funding

No grant support to disclose.

References (34)

T. Asgeirsson et al.
Postoperative ileus: it costs more than you expect
J Am Coll Surg
(2010 Feb)
A.C. Kwok et al.
Are targeted preoperative risk prediction tools more powerful? A test of models for emergency colon surgery in the very elderly
J Am Coll Surg
(2011 Aug)
M.E. Cohen et al.
Development of an American college of surgeons national surgery quality improvement program: morbidity and mortality risk calculator for colorectal surgery
J Am Coll Surg
(2009 Jun)
T.O. Adegboyega et al.
Applying the National Surgical Quality Improvement Program risk calculator to patients undergoing colorectal surgery: theory vs reality
Am J Surg
(2017 Jan)
K.G. Cologne et al.
Use of the American college of surgeons NSQIP surgical risk calculator for laparoscopic colectomy: how good is it and how can we improve it?
J Am Coll Surg
(2015 Mar)
D.S. Keller et al.
Does one score fit all? Measuring risk in ulcerative colitis
Am J Surg
(2016 Sep)
S.K. Sherman et al.
Development of an improved risk calculator for complications in proctectomy
J Gastrointest Surg
(2014 May)
D.A. Redelmeier et al.
Assessing predictive accuracy: how to compare Brier scores
J Clin Epidemiol
(1991)
P. Renzulli et al.
The influence of the surgeon's and the hospital's caseload on survival and local recurrence after colorectal cancer surgery
Surgery
(2006 Mar)
L.M. Napolitano et al.
Risk-adjusted outcomes and perioperative care
Surg Clin North Am
(2005 Dec)

M.R. Decker et al.

Optimizing risk-adjusted outcome measures: a moving target

J Surg Res

(2014 Jan)

R.F. Goldberg et al.

Variability of NSQIP-assessed surgical quality based on age and disease process

J Surg Res

(2013 Jun 15)

J.H. Park et al.

The American Society of Anesthesiologists score influences on postoperative complications and total hospital charges after laparoscopic colorectal cancer surgery

Medicine (Baltim)

(2018 May)

O. Farges et al.

"Surgeons' intuition" versus "prognostic models": predicting the risk of liver resections

Ann Surg

(2014 Nov)

J. Daley et al.

Risk-adjusted surgical outcomes

Annu Rev Med

(2001)

J.B. Christianson et al.

Lessons from evaluations of purchaser pay-for-performance programs: a review of the evidence

Med Care Res Rev

(2008 Dec)

P.P. Tekkis et al.

Development of a dedicated risk-adjustment scoring system for colorectal surgery (colorectal POSSUM)

Br J Surg

(2004)

Cited by (24)

Association of Cumulative Colorectal Surgery Hospital Costs, Readmissions, and Emergency Department/Observation Stays with Insurance Type
2023, Journal of Gastrointestinal Surgery
Medicare’s Hospital Readmission Reduction Program disproportionately penalizes safety-net hospitals (SNH) caring for vulnerable populations. This study assessed the association of insurance type with 30-day emergency department visits/observation stays (EDOS), readmissions, and cumulative costs in colorectal surgery patients.
Retrospective inpatient cohort study using the National Surgical Quality Improvement Program (2013–2019) with cost data in a SNH. The odds of EDOS and readmissions and cumulative variable (index hospitalization and all 30-day EDOS and readmissions) costs were modeled adjusting for frailty, case status, presence of a stoma, and open versus laparoscopic surgery.
The cohort had 245 private, 195 Medicare, and 590 Medicaid/uninsured cases, with a mean age 55.0 years (SD = 13.3) and 52.9% of the cases were performed on male patients. Most cases were open surgeries (58.7%). Complication rates were 41.8%, EDOS 12.0%, and readmissions 20.1%. Medicaid/uninsured had increased odds of urgent/emergent surgeries (aOR = 2.15, CI = 1.56–2.98, p < 0.001) and complications (aOR = 1.43, CI = 1.02–2.03, p = 0.042) versus private patients. Medicaid/uninsured versus private patients had higher EDOS (16.6% versus 4.1%) and readmissions (22.9% versus 14.3%) rates and higher odds of EDOS (aOR = 4.81, CI = 2.57–10.06, p < 0.001), and readmissions (aOR = 1.62, CI = 1.07–2.50, p = 0.025), while Medicare patients had similar odds versus private. Cumulative variable cost %change was increased for Medicare and Medicaid/uninsured, but Medicaid/uninsured was similar to private after adjusting for urgent/emergent cases.
Increased urgent/emergent cases in Medicaid/uninsured populations drive increased complications odds and higher costs compared to private patients, suggesting lack of access to outpatient care. SNH care for higher cost populations, receive lower reimbursements, and are penalized by value-based programs. Increasing healthcare access for Medicaid/uninsured patients could reduce urgent/emergent surgeries, resulting in fewer complications, EDOS/readmissions, and costs.
Development and Validation of Machine Learning Models to Predict Readmission After Colorectal Surgery
2022, Journal of Gastrointestinal Surgery
Readmission after colorectal surgery is common and often implies complications for patients and costs for hospitals. Previous works have created predictive models using logistic regression for this outcome but have shown limited accuracy. Machine learning has shown promise in improving predictions by identifying non-linear patterns in data. We sought to create a more accurate predictive model for readmission after colorectal surgery using machine learning.
Patients who underwent colorectal surgery were identified in the National Quality Improvement Program (NSQIP) database including years 2012–2019 and split into training, validation, and test sets. The primary outcome was readmission within 30 days of surgery. Three types of machine learning models were created, including random forest (RF), gradient boosting (XGB), and neural network (NN). A logistic regression (LR) model was also created for comparison. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC).
The dataset included 213,827 patients after application of exclusion criteria. A total of 23,083 (10.8%) of patients experienced readmission. NN obtained an AUROC of 0.751 (95% CI 0.743–0.759), compared with 0.684 (95% CI 0.676–0.693) for LR. RF and XGB performed similarly with AUROCs of 0.749 (95% CI 0.741–0.757) and 0.745 (95% CI 0.737–0.753) respectively. Ileus, index admission length of stay, organ-space surgical site infection present at time of surgery, and ostomy placement were identified as the most contributory variables.
Machine learning approaches outperformed traditional statistical methods in the prediction of readmission after colorectal surgery. After external validation, this improved prediction model could be used to target interventions to reduce readmission rate.
Validation of the ACS NSQIP surgical risk calculator in older patients with colorectal cancer undergoing elective surgery
2022, Journal of Geriatric Oncology
Citation Excerpt :
Some studies found a good overall performance [25,26], others a poor performance [27–30]. For patients undergoing colorectal procedures, the ACS NSQIP calculator accurately predicted postoperative outcomes (e.g. postoperative complications, return to OR, or discharge to rehabilitation center) [31–34]. However, these study cohorts are not completely comparable to our study population, as we focused specifically on the older population (≥70 years) and colorectal cancer as indication for surgery.
For clinical decision making it is important to identify patients at risk for adverse outcomes after colorectal cancer (CRC) surgery, especially in the older population. Because the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) surgical risk calculator is potentially useful in clinical practice, we performed an external validation in a Dutch multicenter cohort of patients ≥70 years undergoing elective non-metastatic CRC surgery.
We compared the ACS NSQIP calculator mean predicted risk to the overall observed rate of anastomotic leakage, return to operation room, pneumonia, discharge not to home, and readmission in our cohort using a one-sample Z-test. Calibration plots and receiver operating characteristic (ROC) curves were used to determine the calculator's performance.
Six hundred eighty-two patients were included. Median age was 76.2 years. The ACS NSQIP calculator accurately predicted the overall readmission rate (predicted: 8.6% vs. observed: 7.8%, p = 0.456), overestimated the rate of discharge not to home (predicted:11.2% vs. observed: 7.0% p = 0.005) and underestimated the observed rate of all other outcomes. The calibration plots showed poor calibration for all outcomes. The ROC-curve showed an area under the curve (AUC) of 0.75 (95% confidence interval [CI] 0.67–0.83) for pneumonia and 0.70 (0.62–0.78) for discharge not to home. The AUC for all other outcomes was poor.
The ACS NSQIP surgical risk calculator had a poor individual risk prediction (calibration) for all outcomes and only a fair discriminative ability (discrimination) to predict pneumonia and discharge not to home. The calculator might be considered to identify patients at high risk of pneumonia and discharge not to home to initiate additional preoperative interventions.
Surgical risk calculators in veterans following lower extremity amputation
2022, American Journal of Surgery
To evaluate the accuracy of multiple risk calculators for 30-day mortality on patients undergoing major lower extremity amputation.
The actual 30-day mortality at a single Veterans Affairs institution was compared to the predicted outcome from the following risk calculators: ACS-NSQIP, VASQIP, amputation scoring tool (AST), and POTTER elective.
The overall calculated 30-day mortality was similar to the actual mortality with the VASQIP and POTTER elective risk calculators, while the NSQIP and AST over-estimated the 30-day mortality. The predictive accuracy of the POTTER and NSQIP risk calculators were moderate (AUC >0.7), and fair for the VASQIP and AST.
Risk assessment tools can provide adjunctive data on predicted 30-day mortality in patients undergoing major lower extremity amputation. In our study, there were differences in predictability of the risk calculators for lower extremity amputation that should be considered when utilizing a risk assessment tool to improve physician-patient shared decision-making.
Predictive Accuracy of the American College of Surgeons Risk Calculator in Patients Undergoing Major Lower Extremity Amputation
2022, Annals of Vascular Surgery
Citation Excerpt :
For example, the ACS-RC includes a Surgeon Adjusted Risk (SAR) parameter that allows ad hoc adjustment of calculated risk based on perceived relevant risk factors not included in the initial 21 variables.6 The accuracy of the ACS-RC has been explored in a number of surgical disciplines10–13; however, there is a paucity of knowledge regarding its accuracy in vascular surgery patients. Furthermore, there is limited data on how the accuracy of this calculator differs based on utilization of the SAR parameter.
The American College of Surgeons Risk Calculator (ACS-RC) provides an assessment of a patient's risk of 30-day postoperative complications. The Surgeon Adjusted Risk (SAR) parameter of the calculator allows for ad hoc adjustment of risk based on risk factors not considered by the model. This study aims to evaluate the predictive accuracy of the ACS-RC in vascular surgery patients undergoing major lower-extremity amputation (LEA) and identify additional risk factors that warrant use of the SAR parameter.
This is a retrospective study of 298 sequential amputations at a single institution. At the population level, the mean of predicted 30-day outcomes from the ACS-RC with a SAR score of 1 (no adjustment necessary) and 2 (risk somewhat higher than estimate) were compared to the rate of observed outcomes. Predictive accuracy at the individual level was completed using receiver operating curve area under the curve (AUC). Logistic regression with respect to mortality was performed over variables not considered by the ACS-RC. Efficacy of selectively utilizing the SAR parameter in predicting mortality was analyzed with a stratified analysis in which patients with risk factors significant for mortality were assigned increased risk.
At the population level, ACS-RC grossly underpredicted serious complications, SSI, VTE, and unplanned RTOR, while overpredicting mortality and cardiac complications. At the individual level, SAR1 was more predictive for serious complications (AUC = 0.624), SSI (AUC = 0.610), and unplanned RTOR (AUC = 0.541). Conversely, SAR2 was more predictive for mortality (AUC = 0.709), cardiac complications (AUC = 0.561), and VTE (AUC = 0.539). Logistic regression identified history of CVA with a residual deficit (OR = 4.61, P = 0.033) and ischemic rest pain without tissue loss (OR = 4.497, P = 0.047) as independent risk factors for postoperative mortality. Stratified analysis with utilization of the SAR2 based on the 2 independent risk factors improved AUC in predicting mortality (AUC 0.792 from 0.709).
Major LEAs are associated with high perioperative morbidity and mortality. In a veteran population, the ACS-RC showed mixed predictability at the population level and fair predictability at the individual level with regards to postoperative outcomes. Rest pain without tissue loss and history of CVA with residual deficit were identified as risk factors for postoperative mortality. Although ad hoc adjustment with the subjective SAR modifier based on the presence of these 2 risk factors increased the calculator's accuracy, this study highlights some potential limitations of the ACS-RC when applied to vascular surgery patients undergoing major LEA.
What is the Accuracy of the ACS-NSQIP Surgical Risk Calculator in Emergency Abdominal Surgery? A Meta-Analysis
2021, Journal of Surgical Research
Citation Excerpt :
The calculator remains fairly new to surgical practice. Currently there are few studies that have validated the calculator within an external cohort of patients undergoing emergency operations in the field of General Surgery and some studies have questioned it value in external settings.2,3,4 In this project, we sought to perform a meta-analysis assessing the performance of the ACS-NSQIP calculator in an external patient cohort for patients undergoing abdominal surgery in the emergency setting.
The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) surgical risk calculator provides an estimation of 30-d post-operative complications including mortality. This tool has the potential to both aid in decision-making for patients and their families and also in optimizing the clinical management of high-risk patients. However, it's utility in patients requiring emergency abdominal surgery has shown to be inconsistent outside of NSQIP participating institutions. This study undertook a meta-analysis to assess the calculator's accuracy in predicting mortality in these patients.
A literature search of PubMed, Medline and Cochrane databases was conducted between October 2019 to April 2020. The PubMed, Medline and Cochrane Databases were searched for relevant studies. The search strategy included studies from January 2013 to April 2020. Studies including elective surgery were excluded. A random effects model was used and fitted using restricted maximum likelihood estimation. The O:E ratio was used to validate the calculator's accuracy in predicting mortality.
Six studies were included in the meta-analysis, with a total of 1835 patients undergoing emergency intra-abdominal surgery. The summary estimate of the O:E ratio of the ACS-NSQIP surgical risk calculator in predicting 30-d post-operative mortality was 1.06 (95% CI 0.74-1.51). There was significant heterogeneity between studies with a Cochrane Q of 11.96 (P = 0.04) and I² = 57.5%.
The ACS-NSQIP surgical risk calculator is a reliable predictor of mortality in this external cohort and has potential to be utilised in the multi-disciplinary care of patients undergoing emergency abdominal surgery.

View all citing articles on Scopus

View full text

ACS-NSQIP risk calculator predicts cohort but not individual risk of complication following colorectal resection

Highlights

Abstract

Objective

Methods

Results

Patients

Limitations

Conclusions

Introduction

Section snippets

Methods

Results

Discussion and conclusion

Author contributions

Funding

J Am Coll Surg

J Am Coll Surg

J Am Coll Surg

Am J Surg

J Am Coll Surg

Am J Surg

J Gastrointest Surg

J Clin Epidemiol

Surgery

Surg Clin North Am

J Surg Res

J Surg Res

The American Society of Anesthesiologists score influences on postoperative complications and total hospital charges after laparoscopic colorectal cancer surgery

Medicine (Baltim)

"Surgeons' intuition" versus "prognostic models": predicting the risk of liver resections

Ann Surg

Risk-adjusted surgical outcomes

Annu Rev Med

Lessons from evaluations of purchaser pay-for-performance programs: a review of the evidence

Med Care Res Rev

Development of a dedicated risk-adjustment scoring system for colorectal surgery (colorectal POSSUM)

Br J Surg