Comparison of Belgian and US validation studies of ICU-acquired infection surveillance

Introduction

Many validation studies measured the effectiveness of infection surveillance in ICUs. Few, such as Belgium and USA, did so nationwide. The American National Nosocomial Infection Surveillance System (NNIS, CDC) is a worldwide benchmark, although its pilot study showed low sensitivity (SS) on detecting ICU-acquired pneumonia infections (PN). This paper compares the American and Belgian validation studies regarding methodology and likely impact of design differences on PN surveillance assessment.

Methods

The methodology and capacity to estimate PN surveillance's performance of the Belgian and American validation studies are assessed on epidemiological criteria.

Results

The NNIS system set up an accuracy pilot study in 1993, published in 1998. The sample included nine hospitals. All reported nosocomial infections (NI) and a selection of high-risk and low-risk PN-negative patients were examined. In a first phase, 32 external data collectors found over 2.5 times more PN than reported. The PN SS was 39% although hospitals with six consecutive month's surveillance participation and high NI incidence were chosen. In a second phase, two CDC epidemiologists re-examined a nonrepresentative sample of discrepant charts achieving 68% PN SS. Confidence intervals were not reported.

The Belgian validation study used a blind retrospective chart review as the NNIS. Investigators were members of the national surveillance program (NSIH). A two-step (patient and ICU) sampling was used. A sample for single proportion determined the number of PN patients to be reviewed. To get the requested PN cases, 45 ICUs were selected by systematic random sampling from all ICU participation-quarters. The required negative charts were a 20% random sample of all negative charts in these ICUs. The PN SS was 56.31% (95% CI = 47.92–65.21).

Discussion

The NSIH study re-examined a representative sample of 45 out of all participating ICUs (30%), while the NNIS did so in nine of them. At the patient level, the Belgian sample was representative of the ICU case mix since all reported NI infections and a random sample of reported negative charts were reviewed. This methodology allowed CI determination.

The investigators' experience is known to influence the accuracy of identifying NI. Regular NSIH employees collected data in Belgium while such personal was involved in the second phase of the NNIS study only.

Conclusion

The NSIH validation study tackles difficulties with statistical inference found in the NNIS 'accuracy of reporting pilot study'. Given the differences in validation methodology, the surveillance SS in both countries (NNIS: 39–68% in phase 2 vs NSIH: 56%) should be compared cautiously.

Being part of a nationwide surveillance routinely performed on continuous basis, the Belgian validation study design offers an alternative to validate national surveillance systems.