The confusion assessment method for the intensive care unit (CAM-ICU) and intensive care delirium screening checklist (ICDSC) for the diagnosis of delirium: a systematic review and meta-analysis of clinical studies

Delirium is a prevalent form of acute brain dysfunction that occurs in critically ill patients [1]. Despite its elevated frequency and association with increased morbidity and mortality [2], delirium remains an underdiagnosed condition in the intensive care unit (ICU), and a standard clinical evaluation does not have an adequate accuracy for the diagnosis [3]. Several methods have been developed and validated to diagnose delirium in ICU patients [4], but the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) and the Intensive Care Delirium Screening Checklist (ICDSC) are the most frequently employed tools for this purpose [5].

Since the validation of the CAM-ICU and the ICDSC, these tools have been translated into and validated in many languages [6–12] and have been widely employed in clinical practice [5, 13]. However, studies show different results regarding their accuracy for the diagnosis of delirium, possibly affecting the reported incidence of this clinical condition and the implementation of prompt preventive and therapeutic measures [14].

The aim of this study was to perform a systematic review and use a meta-analytic approach to pool previously published studies presenting data on the CAM-ICU and the ICDSC for the diagnosis of delirium in the critically ill.

A two-stage systematic review process was performed. First, we performed a systematic MEDLINE, SciELO, CINAHL and EMBASE databases search using the keywords "CAM-ICU" or "Confusion Assessment Methods for the Intensive Care Unit" (Figure 1) and "ICDSC" or "Intensive Care Delirium Screening Checklist" (Figure 2) from January 2001 through 18 November 2011. However, the automatic alert system of MEDLINE was used to identify studies published during the process of the analysis of the results.

The aim of the review was to identify full-text, English-language publications that evaluated the performance of the CAM-ICU and the ICDSC in critically ill patients. Only articles comparing these diagnostic tools with the gold standard, the DSM-IV criteria, were included.

Original peer-reviewed studies involving the adult population were selected and analyzed. We excluded case reports, review articles, studies that have used these tools to evaluate the correlation between delirium and morbidity or mortality, or compare it with other tools (those not based on DSM-IV criteria). All letters and comments were analyzed for information on validation or implementation of these tools. Studies that assessed children were initially excluded, but further analysis was performed and data including this distinct population were attached in the Additional file 1, Figures S1 and S2. In stage two, eligibility assessment (articles comparing the CAM-ICU or the ICDSC with DSM-IV criteria) and data abstraction were performed independently in an unblinded, standardized manner by two reviewers (DGF and RAC). Discrepancies in the search were resolved by consensus among the authors.

Subsequently, the identified articles were screened electronically. For each eligible article, using a predefined categorization system, information was extracted on the authors, journal, year of publication, study design, inclusion period, number of patients, number of observations, patient population, total number of patients diagnosed with delirium, APACHE II score and sensitivity and specificity of the CAM-ICU and the ICDSC. Moreover, when available, we extracted information about accuracy of these tools in different subgroups of patients analyzed in each paper.

In studies that involved more than one assessor (nurses and/or physicians) in the process of the validation of the tools, we selected the highest sensitivity and included it in the meta-analysis (data from all evaluators are attached in Additional file 2, Table S1). In addition, several studies evaluated the same patient at multiple time-points with different diagnostic tools. In this case, the accuracy of the tool was calculated based on the total number of assessments and not on the number of patients.

The QUADAS scale (first version) was employed to assess the quality of the studies [15]. This tool was developed specifically to assess the quality of studies of diagnostic accuracy included in systematic reviews. Fourteen items were evaluated and, accordingly, each included study was scored from 0 to 14, with a high value indicating a better quality of study. Finally, results were summarized by meta-analysis.

A total of 969 patients were included for the evaluation of the CAM-ICU in the nine studies identified, whereas 391 patients were evaluated in the four validation studies of the ICDSC. All of the studies were conducted in the ICU and, except for the study of van Eijk et al. [14], all of the studies used a methodology for the validation of diagnostic tools. The study by van Eijk et al. [14] evaluated the CAM-ICU in daily practice.

Meta-analysis of studies assessing the CAM-ICU

The pooled values of sensitivity and specificity for the CAM-ICU were 80.0% (95% confidence intervals (CI): 77.1 to 82.6%) and 95.9% (95% CI: 94.8 to 96.8%), respectively (Figure 3). The area under the SROC was 0.97 (Figure 4), suggesting excellent accuracy.

Meta-analysis of studies assessing the ICDSC

The pooled values of sensitivity and specificity for the ICDSC were 74% (95% CI: 65.3 to 81.5%) and 81.9% (95% CI: 76.7 to 86.4%), respectively (Figure 5). The area under the SROC was 0.89 (Figure 6), suggesting good accuracy.

This study represents the first attempt to synthesize the validity and added value of the CAM-ICU and ICDSC in ICU patients. Our results showed that the overall accuracy of the CAM-ICU is excellent, with pooled values for sensitivity and specificity of 80% and 95.9%, respectively. In addition, the pooled values for the sensitivity and specificity of the ICDSC were 74% and 81.9%, respectively. Thus, the currently available data support the use of the CAM-ICU or of the ICDSC as screening tools for delirium in critically ill patients. In addition, because of its high specificity, the CAM-ICU is an excellent diagnostic tool to delirium. This is relevant because a validated tool should be used routinely for monitoring critically ill patients and when delirium is present an algorithm to investigate its cause and a therapeutic strategy should be performed.

After the first validation study [19], the CAM-ICU was translated into and validated in many languages [6–9, 11, 22]. Although studies published in non-English languages have been excluded from this systematic review and meta-analysis, some have shown similar accuracy to the CAM-ICU. Tobar et al. evaluated 29 ventilated patients in the ICU and showed a sensitivity and specificity of 80% and 96%, respectively [6]. Additionally, Toro et al. evaluated 129 patients and observed a sensitivity of 79.4% and a specificity of 97.9% for the CAM-ICU [8]. These same authors performed a subgroup analysis with the ventilated patients (n = 29), and the results suggested better sensitivity (92.9% versus 79.4%) and worse specificity (86.7% versus 97.9%) in this subgroup of patients. Both studies were published in the Spanish language. Chuang et al. validated a Chinese version of the CAM-ICU and again reported high sensitivity (96%) when it was performed by a physician [12].

The present meta-analysis has shown that the pooled sensitivity of the CAM-ICU was 80%, which demonstrates that this tool has good performance for screening patients with delirium in ICU. However, it is also evident that no other validation study has found as high a sensitivity as was observed in the initial studies by Ely et al. [18, 19]. In addition, there was an even lower sensitivity when the CAM-ICU was used in daily practice, that is, outside of a methodology for validation [14].

Although there is no clear explanation for this loss of sensitivity in the most recent studies, it is possible that the evaluation in cohorts of patients that were less sedated, which is a current trend [23], contributes to decreases in the accuracy of the CAM-ICU. In this systematic review, a higher sensitivity of the CAM-ICU was observed in two studies in subgroups of patients with RASS < 0 [4, 10]. Also, a higher sensitivity seems to be present in sedated patients and it is suggested by the differences in accuracy of the CAM-ICU between ventilated and non-ventilated patients. Although no studies compared the accuracy in these subgroups of patients, the study by Toro et al. [8] (not included in this systematic review) is consistent with Ely's study [19] and indicates excellent sensitivity in the subgroup of patients undergoing mechanical ventilation. Again, perhaps the sedation effects can contribute to these findings. It is reasonable to hypothesize that feature 2 (inattention) or feature 3 (disorganized thinking) of the CAM-ICU is less likely to be detected when patients are less sedated. Recently, Vasilevskis et al. suggested a more intense approach to the detection of inattention when the CAM-ICU is used in daily practice [24]. In addition, feature 1 (an acute onset of mental status changes) may be most frequently considered to be positive in patients with sedation and thus increases the sensitivity of the tool. Of course, more studies are necessary to explain and prove this hypothesis.

The four features of the CAM-ICU - 1) acute onset of mental status changes or fluctuating course; 2) inattention; 3) disorganized thinking; and 4) altered level of consciousness - have objective definitions. This characteristic likely justifies the high inter-rater reliability reported in several studies [4, 10, 11, 18, 19].

Moreover, the specificity of the CAM-ICU is high. The pooled value for specificity was 96%, suggesting that when the CAM-ICU is positive, it is not necessary to confirm the diagnosis of delirium by the DSM-IV criteria, improving its feasibility in the ICU. In other words, the CAM-ICU is not only adequate for screening but also a good confirmatory diagnostic tool for delirium in critically ill patients.

Recently, Guenther et al. published a study of the accuracy of the CAM-ICU Flowsheet, comparing it with the DSM-IV criteria [25]. Interestingly, they found a sensitivity of 88% to 92% and an excellent specificity of 100%. Clearly, the CAM-ICU and the CAM-ICU Flowsheet are very similar tools. However, our previous study, despite an excellent correlation (kappa: 0.96) between these tools [9], showed that they were not identical, so we decided not to add the Guenther's study in this meta-analysis.

A Canadian group developed and validated the ICDSC [20] motivated by the same challenge: diagnosing delirium in critically ill and mechanically ventilated patients.

The ICDSC checklist is an eight-item screening tool (one point for each item) that is based on DSM criteria and applied to data that can be collected through medical records or to information obtained from the multidisciplinary team.

Bergeron et al. developed and validated the ICDSC in a mixed ICU [20]. All of the information used to complete the scale was collected from the patient, the primary nurses' evaluation and the chart in the previous 24 hours. With a cutoff score of four points, they showed a sensitivity of 99% and a specificity of 64%. Similar results were found by our group [9]. However, we observed that the sensitivity of the ICDSC was not consistently high in all studies, and that the pooled value for sensitivity in this meta-analysis was 74%. These results suggest that this tool does not appear to be as accurate as the CAM-ICU for screening purposes. George et al., using a different threshold for positivity (3 rather than 4), showed a higher sensitivity (from 75% to 90%) and, consequently, improved screening characteristics of this tool [21]. However, these changes in the cutoff decreased the specificity of the ICDSC, which was already lower than that observed for the CAM-ICU. The pooled value for specificity of the ICDSC in this meta-analysis was 82%.

Additionally, a recent study by Tomasi et al. suggested that the CAM-ICU is a better predictor of outcomes than the ICDSC, which is likely related to the high rate of false positives with the ICDSC [26]. At least two characteristics of the ICDSC might explain its lower sensitivity and specificity. First, the information is collected from the previous 24 hours. Delirium is characterized by its fluctuation, with the possibility of resolution over a long period of evaluation. Additionally, the evaluation of inattention ("easily distracted by external stimuli" [20]), for example, may hinder an effective response by the evaluator.

Despite the limitations described above, the inter-rater reliability of the ICDSC appears to be good. George et al. [21] reported an inter-rater agreement of 0.947 (95% confidence interval, 0.870 to 0.979), and in the study by Bergeron et al. [20], the calculated alpha value was between 0.71 and 0.79.

Interestingly, both tools have worse sensitivity when patients with hypoactive delirium are tested. This issue is relevant because this subtype of delirium is the most prevalent [27]. A lower prevalence of delusions and perceptual disturbances in hypoactive delirium does not appear to explain these findings [28].

Despite the observation that no studies compared the accuracy of both tools in ventilated versus non-ventilated patients, most studies included these two types of patients.

Both tools are important in the care of the critically ill patients, each one with features that allow its use at different times or together. The CAM-ICU, to be quite specific, seems to be the ideal tool for the diagnosis of delirium in critically ill patients. In turn, the ICDSC, by its features not dichotomous, allows the diagnosis of subsyndromal delirium, which has potential prognostic implications [29] and can identify patients with potential therapeutic benefit [30].

Our findings should be understood in the context of some limitations. First, studies published in non-English languages were excluded. Unfortunately, a substantial part of the core information was not available from these studies precluding its use in the meta-analysis. However, as described above, the accuracy of the CAM-ICU appears to be consistent with the results of some of these studies. Second, this study cannot explain the findings with different accuracies of these tools in subgroups of patients (ventilated and nonventilated, RASS < 0, subtypes of delirium), but likely, this is a limitation of the tools. Additionally, the use of the CAM-ICU in patients with non-invasive ventilation has an excellent accuracy; however, its data are limited to a single study involving a small number of observations. This reflects the need for studies to evaluate specific groups of patients.

The present meta-analysis demonstrates that the CAM-ICU is an excellent tool for the detection of delirium in critically ill ICU patients regardless of the subgroup of patients evaluated. Despite having a good performance, the ICDSC presents lower sensitivity and specificity as compared to CAM-ICU. The available data suggest that both CAM-ICU and the ICDSC can be used as a screening tool for the diagnosis of delirium in critically ill patients.