Since 2012, US hospitals have publicly reported surgical-site infections (SSIs) following colon surgeries and abdominal hysterectomies. 1 Hospitals perform surveillance using standardized definitions from the Centers for Disease Control and Prevention (CDC) 2 and submit events to the CDC National Healthcare Safety Network (NHSN). 3 These data are available on the Medicare Care Compare website, 4 and they began affecting financial reimbursement in 2016. 5,6
Unfortunately, current surveillance strategies miss 25%–79% of SSIs following colon surgery Reference Haley, Van Antwerpen and Tserenpuntsag7–Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9 and 32%–61% of SSIs following abdominal hysterectomy. Reference Yokoe, Khan and Olsen8–Reference Reilly, Allardice, Bruce, Hill and McCoubrey11 These unreported infections are due to considerable variation in SSI surveillance practices across hospitals. Reference Reese, Knepper, Price and Young12–Reference Ming, Chen, Miller and Anderson14
We previously showed that claims data are more sensitive and efficient for detecting SSIs compared to routine surveillance following multiple surgical procedures. Reference Huang, Placzek and Livingston15–Reference Calderwood, Kleinman, Murphy, Platt and Huang18 We now report on the national validation of claims-based surveillance of SSIs following colon surgery and abdominal hysterectomy, using surveillance codes developed in a pilot study and previously shown to be effective in identifying unreported SSIs. Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19
Methods
Selection of national validation sample
From October 1, 2012, to September 30, 2013, the Centers for Medicare & Medicaid Services (CMS) selected 600 US hospitals for data validation as part of their Hospital Inpatient Quality Reporting program, including 550 hospitals that performed 22,992 colon surgeries and 458 hospitals that performed 5,265 abdominal hysterectomies on Medicare fee-for-service patients. We obtained records of 1,200 patients from this sample to test the performance of diagnosis and procedure codes for identifying SSIs Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19 and to evaluate the likelihood that the selected codes might miss SSIs.
We identified colon surgeries using ICD-9 codes 17.31–17.36, 17.39, 45.03, 45.26, 45.41, 45.49, 45.52, 45.71–45.76, 45.79, 45.81–45.83, 45.92-45.95, 46.03, 46.04, 46.10, 46.11, 46.13, 46.14, 46.43, 46.52, 46.75, 46.76, and 46.94. 20 We then screened Medicare Part A inpatient claims for International Classification of Disease, Ninth Revision (ICD-9) diagnosis and procedure codes suggestive of SSI during the surgical admission and/or readmission to the same hospital within 30 days following colon surgery: ICD-9 codes 567.21, 567.22, 567.29, 567.38, 569.5, 596.61, 596.81, 682.2, 879.9, 998.31, 998.32, 998.51, 998.59, 998.6, 54.0, 54.11, 54.19, 86.04, 86.22, and 86.28. Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19 To identify potential SSIs missed by these codes, we assessed readmissions to the same hospital within 30 days that did not include an SSI code.
We then randomly selected 150 patients per quarter for full-text medical record review. As decided a priori, this sample included 60% with ≥1 code suggestive of SSI during their surgical admission and/or readmission within 30 days and 40% who were readmitted to the same hospital within 30 days of their surgery without an SSI code. Based on the performance of claims-based surveillance in our prior work, we selected this distribution to pull more records with ≥1 code suggestive of SSI while allowing for review of sufficient records that might identify SSIs missed by our codes. Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19 For patients with an SSI code during their surgical admission, we requested that admission’s records as well as readmission records. For patients without an SSI code during their surgical admission, we only requested records of readmissions. We made this decision based on the number of records that we were contracted to request and the ability to abstract prior history from the readmission records.
For abdominal hysterectomy, we used ICD-9 codes 68.31, 68.39, 68.41, 68.49, 68.61, and 68.69. 20 We then screened Medicare Part A inpatient claims for ICD-9 diagnosis codes suggestive of SSI during the surgical admission and/or readmission to the same hospital within 30 days following abdominal hysterectomy: ICD-9 codes 567.22, 682.2, 998.31, 998.32, 998.51, and 998.59. Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19
We requested full-text medical records for all patients with an SSI code during their surgical admission, for all patients with a readmission (both with and without SSI codes), and for a random sample of patients with a prolonged surgical admission length-of-stay (>7 days) without an SSI code. Chart selection differed for abdominal hysterectomy compared to colon surgery because fewer patients had an SSI code.
Medical record review based on NHSN criteria
A CMS contractor, Edaptive Systems LLC, requested, scanned, and encrypted all records. Trained research assistants at the Harvard Pilgrim Health Care Institute performed initial chart abstractions for the presence or absence of SSI using CDC NHSN criteria. 2 An infectious diseases physician with expertise in hospital epidemiology verified the reviews. This process included weekly review meetings with the research assistants; these discussions identified cases for secondary chart review and verification of SSI classification.
We collected data on type of surgery, sex, age, reason for review, SSI codes during the surgical admission and/or 30-day readmission, infections present at the surgical site at the time of surgery, 2 reoperation within 30 days, and whether reoperation was for infection. For each chart-confirmed SSI, we collected SSI type (superficial, deep, or organ-space), NHSN SSI criteria met, and time from surgery to SSI. When SSI was not confirmed, we recorded whether this was an alternative infection, a noninfectious diagnosis, or cellulitis at the surgical site not meeting criteria for SSI.
The Harvard Pilgrim Health Care Institutional Review Board approved this study.
Analysis
We compared the frequency of superficial SSI, deep or organ-space SSI (“complex SSI”), cellulitis, alternative infection, and no infection identified by the mutually exclusive surveillance strategies using the Fisher exact test. We also calculated the positive predictive value (PPV) of each surveillance strategy.
Applying the PPV for each group to the CMS 2013 data validation cohort for the Inpatient Quality Reporting program, we extrapolated SSI counts for each surveillance strategy. CMS provided the total number of patients meeting each of the surveillance criteria within the 2013 data validation cohort. We used these data to estimate the sensitivity of our claims-based surveillance. The numerator in this sensitivity analysis was the extrapolated number of SSIs identified using claims-based surveillance alone, and the denominator in this sensitivity analysis was the extrapolated total number of SSIs identified using all of the surveillance strategies. For colon surgery, the denominator included SSIs identified based on claims-based surveillance plus SSIs identified on readmissions without an SSI code. For abdominal hysterectomy, the denominator included SSIs identified based on claims-based surveillance plus SSIs identified on readmissions without an SSI code and/or prolonged surgical admission length of stay (>7 days) without an SSI code.
For patients with a chart-confirmed SSI who were missed by claims-based surveillance, we evaluated the specific CDC NHSN SSI criteria met to assess the value of additional surveillance options.
Secondary analyses
We also analyzed the performance of individual ICD-9 codes. In prior work, ICD-9 procedure codes did not significantly add to SSI case capture following colon surgery. Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe9 We sought to confirm this finding in a national sample. SSI surveillance codes for abdominal hysterectomy do not include ICD-9 procedure codes. Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19
Finally, for patients confirmed to have SSIs solely based upon readmission within 30 days or prolonged hospitalization (hysterectomy only), we requested all ICD-9 codes submitted to CMS for reimbursement from the hospitalization when the infection was identified. This allowed us to screen for codes that might improve SSI surveillance.
Results
Study population
Figure 1 shows the national validation sample selected for chart review. Overall, we requested 660 records on 600 patients who underwent colon surgery at 289 hospitals and 540 records on 515 patients who underwent abdominal hysterectomy at 200 hospitals. We received sufficient records to determine a diagnosis in 550 colon-surgery patients (92%) and 466 abdominal hysterectomy patients (90%). The colon-surgery patients had a median age of 73 years (range, 22–98) and were 68% female. The abdominal-hysterectomy patients had a median age of 69 years (range, 27–94) and were all female.
Figure 1. National validation sample requested for chart review.
For colon surgery, we reviewed records of 329 patients with ICD-9 codes suggestive of SSI during their surgical admission and/or readmission (60%) and records of 221 patients readmitted without an SSI code (40%). For abdominal hysterectomy, we reviewed records of 169 patients with an SSI code during their surgical admission and/or readmission (36%), records of 178 patients readmitted without an SSI code (38%), and records of 119 patients with a surgical admission length of stay >7 days (26%).
SSI surveillance following colon surgery
On medical record review, we identified 142 patients with SSIs following colon surgery: 46 superficial SSIs and 96 deep or organ-space SSIs. Table 1 shows the breakdown by surveillance strategy.
Table 1. Performance of Different Surveillance Strategies for Identifying a Surgical-Site Infection Following Colon Surgery
Note. SSI, surgical-site infection; NHSN, National Healthcare Safety Network.
a Not meeting NHSN definition for superficial SSI.
b 134 of 148 due to infection present at the time of surgery. 2
a. Performance of claims-based surveillance
For those patients with an SSI code, 124 (38%) of 329 had chart-confirmed SSIs. For patients readmitted without an SSI code, 18 (8%) of 221 had chart-confirmed SSIs. This difference was significant (P < .001) for both superficial and deep or organ-space SSIs. An additional 6 patients had cellulitis not meeting the NHSN SSI criteria, with no difference in cellulitis based on presence of an SSI code (P < .69).
Although 154 (47%) of 329 patients with an SSI code had an alternative infection, 148 (96%) were flagged on the surgical admission only (Table 1), and 134 (91%) of these met the criteria for an infection present at the time of surgery (excluded from publicly reported SSI). 2
The PPVs for claims-based surveillance varied by whether the SSI code was in the surgical admission and/or readmission (Table 2). Overall, claims-based surveillance yielded an SSI for every 2.6 patients reviewed (124 chart-confirmed SSIs of 329 patients reviewed). Reviewing readmissions without an SSI code yielded an SSI for every 12.3 patients reviewed (18 chart-confirmed SSIs of 221 patients reviewed).
Table 2. Data Extrapolation for CMS Fiscal Year 2013 Data Validation Cohort for the Inpatient Quality Reporting Program—Colon Surgery
Note. CMS, Centers for Medicare and Medicaid Services; SSI, surgical-site infection.
b. Extrapolation to full CMS FY2013 data validation cohort
Extrapolating to all 22,992 colon surgeries included in the CMS 2013 data validation cohort for the IQR program, chart abstraction triggered by SSI codes would identify SSIs in an estimated 1,759 patients of 22,922 colon surgeries in the validation cohort, with an additional 139 identified by review of 30-day readmissions without an SSI code (Table 2). Overall, we estimate an SSI rate of 8.3% (1,898 of 22,922) with a 95% confidence interval of 7.9%–8.6%.
Assuming no additional SSIs in patients without an SSI code or a readmission, the estimated sensitivity for claims-based surveillance was 93% overall (1,759 out of 1,898) and was 92% for deep and organ-space SSI (1,178 out of 1,278).
c. Alternative surveillance options
Of the 18 patients with a chart-confirmed SSI missed by our SSI codes, 9 (50%) required reoperation, so surveillance based on reoperation could capture some of these cases. In addition, 5 (28%) of the 18 patients missed by our SSI codes had a positive culture from aseptically obtained surgical-site fluid or tissue, although 3 of these also would have been identified by reoperation.
SSI surveillance following abdominal hysterectomy
On medical record review, we identified 127 patients with SSIs following abdominal hysterectomy, 58 with superficial SSIs and 69 with deep or organ-space SSIs. Table 3 shows the breakdown by surveillance strategy.
Table 3. Performance of Different Surveillance Strategies for Identifying a Surgical-Site Infection Following Abdominal Hysterectomy
Note. SSI, surgical-site infection; NHSN, National Healthcare Safety Network.
a Not meeting NHSN definition for superficial SSI.
a. Performance of claims-based surveillance
For those patients with an SSI code, 103 (61%) of 169 had chart-confirmed SSIs. For patients readmitted without an SSI code, 20 (11%) of 178 had chart-confirmed SSIs, and for patients with a prolonged surgical length of stay without an SSI code, 4 (3%) of 119 had chart-confirmed SSIs. The difference among these 3 groups was significant (P < .001) for both superficial SSI and deep or organ-space SSI. An additional 25 patients had cellulitis not meeting NHSN SSI criteria, with 24 of these having an SSI code. Thus, there was a significant difference in cellulitis between patients with and without an SSI code (P < .001).
Compared to colon surgery, an infection present at the time of surgery was much less common in patients undergoing abdominal hysterectomy. Among the 89 patients identified as having an alternative infection on chart review, 16 (18%) met criteria for an infection present at the time of surgery.
The PPVs for claims-based surveillance varied by whether the SSI code was in the surgical admission and/or a readmission (Table 4). Overall, claims-based surveillance yielded an SSI for every 1.6 patients reviewed (103 chart-confirmed SSIs of 169 patients reviewed). Reviewing readmissions without an SSI code yielded an SSI for every 8.9 patients reviewed (20 chart-confirmed SSIs of 178 patients reviewed). Reviewing prolonged surgical admissions without an SSI code yielded an SSI for every 27.7 patients reviewed (4 chart-confirmed SSIs of 119 patients reviewed).
Table 4. Data Extrapolation for CMS Fiscal Year 2013 Data Validation Cohort for the Inpatient Quality Reporting (IQR) Program— Abdominal Hysterectomy
Note. CMS, Centers for Medicare and Medicaid Services; SSI, surgical-site infection; FY, fiscal year.
b. Extrapolation to full CMS FY2013 data validation cohort
Extrapolating to all 5,625 abdominal hysterectomies included in the CMS 2013 data validation cohort for the IQR program, chart abstraction triggered by SSI codes would identify SSIs in an estimated 122 of 5,625 abdominal hysterectomies in the data validation cohort, with an additional 24 identified by 30-day readmissions without an SSI code, and an additional 22 identified by a surgical admission >7 days without an SSI code. Overall, we estimate an SSI rate of 3.0% (168 of 5,625) with a 95% confidence interval of 2.5%–3.4%.
Assuming no additional SSIs in the patients without an SSI code, a readmission, or a prolonged surgical admission, the estimated sensitivity for claims-based surveillance was 73% overall (122 of 168) and 76% for deep and organ-space SSI (69 of 91).
c. Alternative surveillance options
Of the 24 patients with a chart-confirmed SSI missed by our SSI codes, only 2 (8%) of these 24 patients required reoperation in the 30 days following abdominal hysterectomy, and only 4 (17%) of the 24 patients had a positive culture from aseptically obtained surgical site fluid or tissue.
Individual code performance
Table 1 in the Supplementary Appendix (online) shows the breakdown of abstracted diagnoses in patients flagged by specific codes during their surgical admission and/or readmission following colon surgery. Five codes identified more alternative infections than SSIs: ICD-9 codes 567.21/567.29 for peritonitis; 567.38 for other retroperitoneal abscess; 596.5 for abscess of intestine; and 596.81 for fistula of intestine. We did not have data on which codes were present on admission to suggest infection present at the time of surgery. In terms of the most sensitive code for SSI, ICD-9 code 998.59 (other postoperative infection) identified 90 (63%) of 142 infections following colon surgery. Finally, dropping ICD-9 procedure codes from our claims-based surveillance strategy would only miss 2 of the 124 SSIs identified by our full set of codes. These 2 SSIs were both organ-space SSIs identified by ICD-9 procedure code 54.19 (other laparotomy) with no ICD-9 diagnosis codes suggestive of SSI.
Table 2 in the Supplementary Appendix (online) shows the breakdown of abstracted diagnoses in patients flagged by specific codes during their surgical admission and/or readmission following abdominal hysterectomy. None of the codes appeared to disproportionately identify non-SSI diagnoses. As with colon surgery, ICD-9 code 998.59 was the most sensitive code, identifying 86 of 127 SSIs (68%).
Additional surveillance codes
For the 18 patients with SSIs missed by our surveillance codes following colon surgery and the 24 patients with SSIs missed by our surveillance codes following abdominal hysterectomy, we reviewed all ICD-9 codes submitted to CMS for reimbursement from the hospitalization during which they met SSI criteria. The codes suggestive of a possible infection were nonspecific. These included ICD-9 code 288.60 for unspecified leukocytosis, which was used in 4 (22%) of the missed SSI cases following colon surgery and 4 (17%) of the missed SSI cases following abdominal hysterectomy; code 995.92 for severe sepsis, which was used in 4 (22%) of the missed SSI cases following colon surgery; code 785.52 for septic shock, which was used in 3 (17%) of the missed SSI cases following colon surgery; and code 780.60 for fever, which was used in 3 (13%) of the missed SSI cases following abdominal hysterectomy.
Discussion
We assessed the performance of claims-based surveillance used to identify records for review during validation site visits by CMS. 21 The benefit of claims-based surveillance is that it has a high sensitivity relative to other surveillance methods, as well as a high PPV, meaning that fewer records need to be reviewed for each identified SSI. Because these data are used publicly to rank hospitals on quality and to determine hospital reimbursement, data validation is key to ensuring fair comparisons.
During data validation, surveyors need an efficient means of identifying charts to review to determine whether an infection occurred and whether the infection was reported to NHSN. Our results show a methodology that is well suited for identifying patients for validation review. This does not mean that there is a perfect correlation between having an SSI code and having a chart-confirmed SSI. Instead, this is a population of patients with a higher likelihood of SSI on chart review.
This methodology can also be used by hospitals to improve SSI case finding, especially when looking retrospectively on a monthly or a quarterly basis. One study found that infections preventionists dedicated 24 hours of manual review time for every SSI identified following colon surgery. Reference Knepper, Young, Reese, Savitz and Price22 Our methodology is much more efficient and retained sensitivity higher than traditional hospital methods. Reference Haley, Van Antwerpen and Tserenpuntsag7–Reference Reilly, Allardice, Bruce, Hill and McCoubrey11,Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19 Infection prevention teams can collaborate with clinical documentation specialists to flag patient records containing specific codes suggestive of SSI.
In terms of supplemental strategies, such as reviewing readmissions or hospitalizations with a prolonged length of stay, our results showed only a modest increase in case detection for a significantly greater workload for those doing the surveillance. It is possible that reviewing patients who require reoperation or who have positive microbiology may efficiently identify cases missed by claims-based surveillance, but this requires additional study.
As for limitations, we identified patients with codes suggestive of SSI during the surgical admission and/or readmission to the same hospital within 30 days. We did this because the CMS validation strategy only targets SSIs that occur at the operative hospital. In our validation sample, most but not all patients returned to the surgical hospital for readmission. Prior research has revealed that limiting surveillance to the operative hospital can lead to inaccurate rankings disadvantaging some hospitals in terms of financial penalties. Reference Yokoe, Avery, Platt, Kleinman and Huang23 It is possible to link patients across hospitals using Medicare claims data, so this may be worth exploring to further improve validation.
We also limited our chart abstraction to inpatient records, which missed patients who were diagnosed and treated in the outpatient setting. We believe that the majority of these infections are superficial and not part of the determination of hospital reimbursement under federal programs. 5,6 Thus, we continue to advocate that data validation focus on infections identified during the surgical admission or on readmission. The proposal to focus on complex SSIs diagnosed in the inpatient setting has also been supported by others in the field. Reference Ming, Chen, Miller and Anderson14
Regarding the high percentage of patients who were flagged for review by an SSI code during their index surgical admission only and subsequently found to have an infection present at the time of surgery and not resultant from the surgery, it is possible that these could be filtered out by looking for a present-on-admission (POA) modifier in the claims. We did not have access to POA status at the time of our data collection and analysis.
We also acknowledge that we studied data from the CMS 2013 data validation cohort and not a more recent year. The reduction in SSIs over the past 10 years might affect the sensitivity of claims-based surveillance, 24 but we have no reason to believe that the reduction in SSIs nationally will differentially affect the performance of the different surveillance strategies. Instead, it is likely that fewer records will be flagged for review due to fewer SSIs.
Finally, it will be important to study the impact of transitioning from the ICD-9 to the ICD-10. One study showed no significant change in the performance of SSI surveillance codes for colon surgery and abdominal hysterectomy after the transition to ICD-10. Reference Bucher, Yang, Arndorfer, Frame, Orton, Samore and Dascomb25 We previously published a comparison of the ICD-9 diagnosis codes to ICD-10 diagnosis codes (Supplementary Appendix Table 3 online). Reference Calderwood, Huang, Keller, Bruce, Kazerouni and Janssen19
In summary, our findings provide strong support for the current methodology being used by CMS and some state health departments to identify records to review during validation site visits. US hospitals should consider adopting claims-based surveillance to retrospectively identify cases missed by traditional surveillance.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/ice.2023.193
Acknowledgments
Financial support
This research was supported by a grant from the Agency for Healthcare Research and Quality (grant no. R18 HS021424).
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
