Skip to main content

Delirium prediction in the intensive care unit: comparison of two delirium prediction models

Abstract

Background

Accurate prediction of delirium in the intensive care unit (ICU) may facilitate efficient use of early preventive strategies and stratification of ICU patients by delirium risk in clinical research, but the optimal delirium prediction model to use is unclear. We compared the predictive performance and user convenience of the prediction  model for delirium (PRE-DELIRIC) and early prediction model for delirium (E-PRE-DELIRIC) in ICU patients and determined the value of a two-stage calculation.

Methods

This 7-country, 11-hospital, prospective cohort study evaluated consecutive adults admitted to the ICU who could be reliably assessed for delirium using the Confusion Assessment Method-ICU or the Intensive Care Delirium Screening Checklist. The predictive performance of the models was measured using the area under the receiver operating characteristic curve. Calibration was assessed graphically. A physician questionnaire evaluated user convenience. For the two-stage calculation we used E-PRE-DELIRIC immediately after ICU admission and updated the prediction using PRE-DELIRIC after 24 h.

Results

In total 2178 patients were included. The area under the receiver operating characteristic curve was significantly greater for PRE-DELIRIC (0.74 (95% confidence interval 0.71–0.76)) compared to E-PRE-DELIRIC (0.68 (95% confidence interval 0.66–0.71)) (z score of − 2.73 (p < 0.01)). Both models were well-calibrated. The sensitivity improved when using the two-stage calculation in low-risk patients. Compared to PRE-DELIRIC, ICU physicians (n = 68) rated the E-PRE-DELIRIC model more feasible.

Conclusions

While both ICU delirium prediction models have moderate-to-good performance, the PRE-DELIRIC model predicts delirium better. However, ICU physicians rated the user convenience of E-PRE-DELIRIC superior to PRE-DELIRIC. In low-risk patients the delirium prediction further improves after an update with the PRE-DELIRIC model after 24 h.

Trial registration

ClinicalTrials.gov, NCT02518646. Registered on 21 July 2015.

Background

Delirium, defined as acute brain dysfunction featured by disturbances of attention, awareness, and cognition with a fluctuating course caused by an underlying medical condition [1], occurs frequently in the intensive care unit (ICU), is associated with impaired patient outcome, and substantially increases healthcare costs [2, 3]. Given these deleterious consequences, delirium prevention is crucial.

Delirium preventive measures are important for all ICU patients. However, a delirium prediction model may facilitate early recognition of the patients who may benefit the most from delirium prevention [4]. In the case of limited resources, non-pharmacologic reduction strategies and medication-based strategies may be most relevant for patients who have an increased risk of developing delirium. Prediction models may aid clinical decision making and setting of priorities regarding the use of delirium preventive measures. For instance, when deciding which patient should be admitted to the available room with adequate natural daylight and which patient to the room without it, preferably the patient with the highest delirium risk should be admitted to the room with adequate natural daylight. Also, the use of a delirium prediction model facilitates patient selection for studies on delirium prevention, which is not only efficient in terms of reducing waste, but it may also increase the chance of finding an effect, which ultimately might improve ICU patients’ outcomes. Furthermore, family members can be informed about the patient’s risk of developing delirium and be engaged to help provide strategies to reduce delirium (e.g. cognitive activities) [5]. Involvement of family in patient care in the ICU is stimulated by many ICU societies worldwide [6, 7] and might even increase the prevalence of interventions for delirium prevention and treatment in the ICU [8].

Two delirium prediction models have been validated for use in critically ill adults admitted to the ICU [9,10,11]. The prediction model for delirium in ICU patients (PRE-DELIRIC model) was developed and validated in a large cohort of Dutch ICU patients [9]. This model, which was recently recalibrated in a multinational cohort [10], reliably predicts ICU patients’ risk of delirium using ten predictors obtained within the first 24 h of ICU admission [10]. However, given that up to 25% of critically ill adults develop delirium within the first 24 h of ICU admission [12, 13], and delirium prevention strategies should be deployed as early as possible, an early prediction model (E-PRE-DELIRIC) was developed to predict the risk of delirium the moment a patient is admitted to the ICU [11]. This E-PRE-DELIRIC model was developed and validated in a multinational cohort and uses nine predictors to predict ICU patients’ risk of delirium [11].

It remains unclear which ICU delirium prediction model might be recommended for daily clinical practice, because the comparative predictive performance of the PRE-DELIRIC and the E-PRE-DELIRIC models and clinicians’ preferences have not been assessed [14]. Therefore, the objective of this study was to compare the predictive performance and user convenience of the PRE-DELIRIC and E-PRE-DELRIC models. Second, we sought to determine the value of the use of both models in a two-stage calculation of patients’ risk of delirium in the ICU (i.e. the E-PRE-DELIRIC model immediately after ICU admission with an updated delirium risk score after 24 h of ICU admission using the PRE-DELIRIC model) to see if we could expand on current models, since it is well-known that dynamic variables assessed over time as opposed to variables assessed at admission only tend to perform better in prediction models.

Methods

Design and study population

The “Delirium prediction in the intensive care unit: comparison of two delirium prediction models” (DECISION) study was a multinational prospective cohort study conducted in 11 ICUs from seven different countries (Australia, Belgium, Canada, Denmark, Portugal, USA, and the Netherlands). Each study site had a well-established delirium screening protocol and similar delirium treatment practices. All consecutive, critically ill adults admitted to the ICU were enrolled. Patients were excluded if they had delirium at the time of ICU admission, were discharged from the ICU within 6 h, or were unable to be reliably assessed for delirium (e.g. sustained coma, inability to understand the predominant language spoken in the ICU, severe cognitive dysfunction, receptive aphasia, or serious auditory or visual disorders) [9,10,11]. Each institution enrolled patients for up to three months or until data on 300 patients were collected.

Data collection

Data were collected over the first 14 days of the ICU stay. Data for each delirium predictor (nine predictors for the E-PRE-DELIRIC model and ten for the PRE-DELIRIC model) were collected in consecutive patients immediately after ICU admission (E-PRE-DELIRIC) [11] and within 24 h of ICU admission (PRE-DELIRIC) [9, 10] and entered into a validated web-based, data management system, Castor [15]. Severity of illness was estimated at ICU admission using the Acute Physiology and Chronic Health Evaluation (APACHE) II score [16] and daily using the Sequential Organ Failure Assessment (SOFA) score [17].

The presence of delirium in the ICU was evaluated at least every 12 h by the trained bedside nurse using either the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) [18] or the Intensive Care Delirium Screening Checklist (ICDSC) [19]. Development of delirium in the ICU was defined as at least one positive assessment of delirium using the CAM-ICU or ICDSC. Patients were also deemed to have delirium whenever they were administered haloperidol or an atypical antipsychotic drug for treatment of delirium, to prevent false negative delirium screenings. To eliminate bias, nurses were kept unaware of the fact that their delirium assessments were used for a study [20].

Level of sedation (using either the Richmond Agitation-Sedation Scale (RASS) or the Riker Sedation-Agitation Scale (SAS) [21, 22], and current intravenous (IV) sedative therapy was documented at the time each delirium assessment was completed. Delirium was preferentially evaluated when patients were maximally awake (e.g. after a spontaneous awakening trial). When coma was present (i.e., RASS = − 4 or − 5 or Riker-SAS = 1 or 2) patients were designated as unable to be assessed for delirium.

To help ensure that the nurse delirium assessments were of high quality [23], a trained investigator (or research nurse) independently and sequentially evaluated patients for the presence of delirium using the same tool as the bedside nurse (i.e., CAM-ICU or ICDSC) during one daytime shift each month and nurse-expert inter-rater reliability (IRR) was calculated. A delirium assessment compliance rate (i.e. delirium assessments documented/delirium assessments that should have been completed) was calculated for one day monthly in each ICU. If Cohen’s kappa for the paired delirium assessments was ≥ 0.80 and delirium screening compliance was ≥ 80% then the delirium assessment was considered to be reliable in that ICU. Prior to the study it was determined that centres would be described separately if they did not meet these two reliability criteria and if they had outcomes that significantly affected the performance of the (E)-PRE-DELIRIC model in the primary analysis.

Evaluation of the user convenience of the delirium prediction model

To estimate delirium model user convenience, the preferences of ICU physicians regarding the two delirium prediction models was determined by electronically administering a short, optional, and anonymous web-based survey (comprising four 5-point, Likert-scale questions) to all physicians working in each study ICU (Additional file 1). A completed questionnaire indicated that a physician provided consent for their data to be used.

Statistical analysis

For each delirium prediction model at least 200 events, i.e. positive delirium assessments, were needed [24]. With an anticipated delirium incidence conservatively set at 20%, we aimed to enrol 2000 patients in total (400/0.20 = 2000 patients).

The discriminative power of both models was assessed using the area under the receiver-operating characteristic curve (AUROC) [20]. The database was divided into groups based on the quartiles of the predicted probabilities for delirium development: very low (0.00–0.10), low (0.10–0.20), moderate (0.20–0.30), and high risk of delirium (≥ 0.30).

Sensitivity, specificity, and likelihood ratios were calculated for these four groups. Calibration was assessed graphically by plotting the observed outcome frequencies against the mean predicted outcome probabilities or risks, within subgroups of patients that were ranked by increasing estimated probability [25]. The predictive performance of both models was compared using the Hanley and McNeil method [26]. It is estimated that approximately a third of patients will develop delirium in the ICU [3]. We therefore rated patients with a predicted probability of delirium <0.30 as low-risk patients and with probability ≥0.30 as high-risk patients. The additional value of a two-stage calculation in low-risk patients was determined using the E-PRE-DELIRIC model to calculate a patient’s risk of delirium immediately after ICU admission. Subsequently we used data from the first 24 h in ICU to update the prediction using the PRE-DELIRIC model to determine how many patients with a probability of delirium <0.30 predicted using the E-PRE-DELIRIC model would subsequently be labelled at high risk of developing delirium using the PRE-DELIRIC score. Both risk calculations were compared to the patients’ delirium outcome.

The questionnaires for ICU physicians were analyzed using the Wilcoxon signed ranks test for non-parametric statistical testing of two dependent samples. Statistical significance was defined as p < 0.05 and the null hypotheses were tested against two-sided alternatives. Data were analysed using SPSS® Statistics version 22 and R statistics R3.2.4 [27].

Results

A total of 2802 patients were screened for inclusion; 2178 patients (78%) were included. Among the 624 patients excluded, inability to reliably assess for delirium (46.3% (289/624)) and delirium at the time of ICU admission (25.9% (162/624)) were the most common reasons for exclusion (see study flowchart, Fig. 1). Among the 83 patients that were excluded for other reasons the most common reasons were severe neurological injury and confidential files. Patients were 62.1 ± 15.2 years old, 60.8% male, and had a baseline APACHE-II score of 17.4 ± 7.1. During their ICU stay 21.4% of patients (467/2178) developed delirium. Patient characteristics are presented in Table 1. For patient and hospital characteristics per participating ICU see Additional file 2: Table S1.

Fig. 1
figure 1

Study flowchart. E-PRE-DELIRIC, early prediction model for delirium in ICU patients

Table 1 Patient characteristics

Model performance

The AUROC of the PRE-DELIRIC model (0.74 (95% CI 0.71–0.76)) was significantly greater than that for the E-PRE-DELIRIC model (0.68 (95% CI 0.66–0.71)) (z score of − 2.73 (p < 0.01)) (Fig. 2). Both models were well-calibrated (Fig. 3).

Fig. 2
figure 2

AUROC for the early prediction model for delirium in ICU patients (E-PRE-DELIRIC) and the prediction model for delirium in ICU patients (PRE-DELIRIC)

Fig. 3
figure 3

Calibration plot for the early prediction model for delirium in ICU patients (E-PRE-DELIRIC) and the prediction model for delirium in ICU patients (PRE-DELIRIC)

At a cutoff of 0.214, the incidence of delirium was 21.4% in the total sample, and the sensitivity and specificity were 60 and 65%, respectively, for the E-PRE-DELIRIC model and 69 and 66% for the PRE-DELIRIC model.

Two-stage calculation

A total of 1586 patients had a predicted probability of delirium <0.30 with the E-PRE-DELIRIC model and therefore were not deemed to be at high risk of delirium based on the cutoff of 0.30. However, 262 of these patients eventually did experience delirium during their ICU stay, despite an initial low predicted risk of delirium calculated using the E-PRE-DELIRIC model. Using data from the first 24 h in the ICU, the PRE-DELIRIC model identified 64 of these 262 patients (24%) to be at high risk of delirium (i.e. predicted probability of delirium ≥0.30). The two-stage calculation improved the sensitivity of the prediction by 14% compared to the E-PRE-DELIRIC model alone and by 10% compared to the PRE-DELIRIC model alone (Fig. 2).

User convenience

In total, 68 (41%) ICU physicians completed the user convenience questionnaire. Of the ICU physicians who participated in this study 52 (76%) were intensivists, 11 (16%) were intensivist trainees, 4 (6%) were specialists other than intensivists, and 1 (2%) was a non-specialist physician. There were 56 ICU physicians (82%) with no prior experience of using a delirium prediction model; only 1 physician had experience of using the E-PRE-DELIRIC model and 11 of using the PRE-DELIRIC model: none of the physicians used a prediction model regularly, although 6 physicians (9%) stated that a prediction model had been implemented in the ICU where they work. Physicians perceived that the PRE-DELIRIC model (versus the E-PRE-DELIRIC model) took more “time and effort to collect data” (p < 0.05) and was a greater “burden for the physician to collect the model data” (p < 0.01). In contrast, for the E-PRE-DELIRIC model, physicians perceived that “the predictors were more available” (p < 0.05) and that they were more likely “to use this model (vs. the PRE-DELIRIC model) in daily practice” (p < 0.05). The “clearness of the definitions” and the “reliability of the outcome” of the E-PRE-DELIRIC and PRE-DELIRIC models were perceived to be similar (Table 2 and Additional file 3: Table S2).

Table 2 Outcome user convenience questionnaire

Quality check of the delirium assessment

The overall quality of delirium assessment and screening compliance were strong (see Additional file 4: Table S3 for all IRR and compliance rates for each participating ICU). When the ICUs that did not fully meet all delirium assessment reliability criteria were removed from the analysis, as was determined a priori, neither the performance of the E-PRE-DELIRIC model nor the PRE-DELIRIC model was significantly affected. Consequently all centres were included in the primary analysis.

Discussion

This large, multinational prospective cohort study provides insight into the comparative performance of two available ICU delirium prediction models (i.e. the E-PRE-DELIRIC model that estimates the risk of delirium at the time of ICU admission and the PRE-DELIRIC model used to estimate the risk of delirium 24 h later) [9,10,11]. Both models had a moderate-to-good statistical performance. Although the predictive accuracy of the E-PRE-DELIRIC model was somewhat lower, its user convenience appeared to be better compared to the PRE-DELIRIC model. To allow for optimal implementation of a delirium prediction model in daily practice, involvement and the opinion on user convenience of the target group, i.e. the ICU physicians, is very important [28, 29]. Based on these results, the E-PRE-DELIRIC model is likely the model that can be implemented most successfully in daily ICU practice. Moreover, our analysis indicates that when the E-PRE-DELIRIC model predicts a low risk of delirium, an additional calculation using the PRE-DELIRIC model after 24 h in the ICU increases the model’s sensitivity to detect patients that will develop delirium who are incorrectly identified as low-risk patients. Thus, this method will prevent deprivation of delirium-preventive measures in patients with a false negative rating (i.e. the patients with predicted probability <0.30 who develop delirium during ICU admission).

The routine use of delirium preventive measures in the ICU is widely endorsed given the high prevalence of delirium, its deleterious effects on patient outcome [30, 31], and the high costs related to these effects [32]. The routine use of a delirium prediction model may facilitate early recognition of those patients at greatest risk of delirium who may benefit the most from delirium preventive measures [32]. Importantly, when resources are limited, the use of delirium risk stratification to target high-risk patients makes wider implementation of multicomponent non-pharmacological interventions aimed at preventing delirium more feasible [32]. Preventive measures should be initiated as soon as possible after ICU admission; therefore an early ICU delirium prediction model is preferred.

The performance of a prediction model outside the development sample determines its generalisability in clinical practice [33]. External validation of many other clinical prediction models is lacking [34]. Of interest, both the E-PRE-DELIRIC and recalibrated PRE-DELIRIC models are validated externally and had moderate-to-good statistical performance in independent data sets, allowing for generalisation to non-study ICUs around the world [[35], Wassenaar et al. 2017]. “External validation of two models to predict delirium in intensive care unit patients. Unpublished data.” The delirium incidence, estimated based on new positive delirium assessments after ICU admission, is important for the performance and thus the generalisation of both delirium prediction models. One might argue that the delirium incidence in our study cohort was relatively low. However, multiple previous studies have reported comparable delirium incidence rates [36,37,38].

Our study has important strengths. The use of a cohort design without strict eligibility criteria helped boost its generalisability [14], and its prospective nature allowed us to carefully measure and document the predictors and outcomes, thereby improving its applicability and reproducibility in non-study ICUs [14, 20, 39]. The large number of patients enrolled, their mixed characteristics, and the multinational character of our study allows the results to be applied in the vast majority of ICUs in the developed world. Of note, when generalising to high-intensity ICUs it should be taken into account that their patient group will probably have more severe illness in comparison with our study cohort. For future research it might be of interest to study the performance of both models in patients with APACHE and SOFA scores in the higher ranges. The proportion of ICU physicians responding was better than response rates shown in other physician surveys [40]. No imputation techniques were used to handle missing data, as we wanted to determine the clinical performance of both delirium prediction models and the use of a prediction model in daily clinical practice does not allow for imputation. Our efforts to provide clear definitions and instruction manuals to all study sites resulted in the exclusion of only fourteen patients due to missing values for the predictors.

Several limitations are also present. It might be possible that the two delirium prediction models evaluated might need to be updated in the future as new risk factors for delirium in the ICU may emerge. Of course, this also offers an opportunity to further improve the discriminative performance of each model, which in particular could benefit the E-PRE-DELIRIC. For an update, referred to as model revision, one needs to have insight into new risk factors for delirium, both available at ICU admission or within 24 h of ICU admission. Subsequently, a new prediction study is needed to determine which of the new risk factors improves the performance of the models and should be used for model revision [41]. It is important to realize that when a model is used to predict a patient’s risk of an event, it should always be considered an approximation no matter how strong the documented predictive accuracy. This is particularly important in the case of medical decision making. Two well-validated delirium screening-instruments (i.e. the CAM-ICU and the ICDSC) were used in this study. Naturally, the sensitivity and specificity of each instrument differs [42]. Realizing that the sensitivity of either screening tool is not 100%, we also defined delirium to be present when haloperidol or an atypical antipsychotic was administered for the treatment of delirium. While each ICU had similar delirium treatment protocols, we cannot exclude that antipsychotic therapy may have been initiated in patients who did not have delirium.

It is shown that ICU clinicians’ predictions are less accurate than those of an ICU delirium prediction model [9]. We believe that routine prognostic delirium evaluation in the ICU is important in the clinical setting to identify those patients who may benefit the most from early preventive measures and in the research setting to ensure that delirium risk is well-characterized and stratified in controlled studies. We want to emphasize that the predicted risk score is an estimation of the chance of developing delirium during ICU admission that may facilitate early clinical decisions on delirium prevention and personalized care to ICU patients and their family members. However, the rationing of critical care resources should not be based on the predicted risk score for delirium over the first 24 h of ICU admission alone as the predicted risk score does not take into account changes in the health status of ICU patients. For future controlled studies on the effect of both non-pharmacological and pharmacological interventions on ICU delirium, we suggest stratifying patients based on their risk of delirium and restricting delirium prevention to those patients at high risk of delirium.

To achieve the best predictive performance and user convenience currently possible, we suggest a two-stage calculation using the E-PRE-DELIRIC model in all patients admitted to the ICU to predict patients’ risk of delirium immediately after ICU admission and to update the risk scores of the patients at low risk of delirium after 24 h using the PRE-DELIRIC model. This way, the chance of missing a patient that will develop delirium during ICU admission is further attenuated. Still, a substantial minority of the patients that develop delirium during ICU admission will score a low predicted risk for delirium using both delirium prediction models in a two-stage calculation. Of interest, in this study cohort the delirium incidence was 21%. Based on the fact that the sensitivity and specificity of a prediction model are most optimal at the cutoff of the incidence level of the outcome of interest, in this case delirium, we expect that the sensitivity in a population with around 30% incidence of delirium will be better compared to the sensitivity shown in our study. Of importance, the acceptability of the suggested two-stage calculation by ICU physicians is not yet assessed. Future research should focus on the usefulness of both delirium prediction models in clinical practice and their impact on clinical outcomes, since this is the only way to determine whether their use improves usual care [41]. In addition, such an impact analysis also provides the opportunity to study the acceptance of the models in daily practice [41], in which it is interesting to also take the experiences of ICU patients and their families into account.

Conclusions

This study shows that statistically both ICU delirium prediction models have moderate-to-good performance. Although the predictive accuracy of the PRE-DELIRIC is greater, the E-PRE-DELIRIC model scores significantly better on user convenience. Moreover, the PRE-DELIRIC model needs data obtained during a period of 24 h, while the E-PRE-DELIRIC can be obtained at ICU admission, allowing direct preventive measures and stratified randomization in studies at the time of ICU admission. In patients who appear to be at low risk of delirium at ICU admission, it is of additional advantage to update their predicted risk scores using the PRE-DELIRIC model after 24 h in the ICU.

Abbreviations

AUROC:

Area under the receiver operating characteristic curve

CAM-ICU:

Confusion Assessment Method for the ICU

E-PRE-DELIRIC model:

Early prediction model for delirium in ICU patients

ICDSC:

Intensive Care Delirium Screening Checklist

ICU:

Intensive care unit

IRR:

Inter-rater reliability

PRE-DELIRIC model:

Prediction model for delirium in ICU patients

References

  1. APA. Diagnostic and statistical manual of mental disorders, fifth edition. 5th ed. Washington, D.C.: American Psychiatric Association (APA); 2013.

    Google Scholar 

  2. Milbrandt EB, Deppen S, Harrison PL, et al. Costs associated with delirium in mechanically ventilated patients. Crit Care Med. 2004;32(4):955–62.

    Article  PubMed  Google Scholar 

  3. Salluh JI, Wang H, Schneider EB, et al. Outcome of delirium in critically ill patients: systematic review and meta-analysis. BMJ. 2015;350:h2538.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Mistraletti G, Pelosi P, Mantovani ES, Berardino M, Gregoretti C. Delirium: clinical approach and prevention. Best Pract Res Clin Anaesthesiol. 2012;26(3):311–26.

    Article  PubMed  Google Scholar 

  5. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453–73.

    Article  CAS  PubMed  Google Scholar 

  6. Giannini A, Garrouste-Orgeas M, Latour JM. What’s new in ICU visiting policies: can we continue to keep the doors closed? Intensive Care Med. 2014;40(5):730–3.

    Article  PubMed  Google Scholar 

  7. Ely EW. The ABCDEF bundle: science and philosophy of how ICU liberation serves patients and families. Crit Care Med. 2017;45(2):321–30.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Morandi A, Piva S, Ely EW, et al. Worldwide survey of the “Assessing Pain, Both Spontaneous Awakening and Breathing Trials, Choice of Drugs, Delirium Monitoring/Management, Early Exercise/Mobility, and Family Empowerment” (ABCDEF) bundle. Crit Care Med. 2017;45(11):e1111–22.

    Article  PubMed  Google Scholar 

  9. van den Boogaard M, Pickkers P, Slooter AJ, et al. Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study. BMJ. 2012;344:e420.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. van den Boogaard M, Schoonhoven L, Maseda E, et al. Recalibration of the delirium prediction model for ICU patients (PRE-DELIRIC): a multinational observational study. Intensive Care Med. 2014;40(3):361–9.

    Article  CAS  PubMed  Google Scholar 

  11. Wassenaar A, van den Boogaard M, van Achterberg T, et al. Multinational development and validation of an early prediction model for delirium in ICU patients. Intensive Care Med. 2015;41(6):1048–56.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Ely EW, Shintani A, Truman B, et al. Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA. 2004;291(14):1753–62.

    Article  CAS  PubMed  Google Scholar 

  13. Serafim RB, Dutra MF, Saddy F, et al. Delirium in postoperative nonventilated intensive care patients: risk factors and outcomes. Ann Intensive Care. 2012;2(1):51.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375.

    Article  PubMed  Google Scholar 

  15. Castor Electronic Data Capture. Amsterdam, the Netherlands: Ciwit BV; 2017.

  16. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818–29.

    Article  CAS  PubMed  Google Scholar 

  17. Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–10.

    Article  CAS  PubMed  Google Scholar 

  18. Ely EW, Inouye SK, Bernard GR, et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA. 2001;286(21):2703–10.

    Article  CAS  PubMed  Google Scholar 

  19. Bergeron N, Dubois MJ, Dumont M, Dial S, Skrobik Y. Intensive Care Delirium Screening Checklist: evaluation of a new screening tool. Intensive Care Med. 2001;27(5):859–64.

    Article  CAS  PubMed  Google Scholar 

  20. Moons KG, Kengne AP, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683–90.

    Article  PubMed  Google Scholar 

  21. Riker RR, Picard JT, Fraser GL. Prospective evaluation of the Sedation-Agitation Scale for adult critically ill patients. Crit Care Med. 1999;27(7):1325–9.

    Article  CAS  PubMed  Google Scholar 

  22. Sessler CN, Gosnell MS, Grap MJ, et al. The Richmond Agitation-Sedation Scale: validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med. 2002;166(10):1338–44.

    Article  PubMed  Google Scholar 

  23. van Eijk MM, van den Boogaard M, van Marum RJ, et al. Routine use of the confusion assessment method for the intensive care unit: a multicenter study. Am J Respir Crit Care Med. 2011;184(3):340–4.

    Article  PubMed  Google Scholar 

  24. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–26.

    Article  PubMed  Google Scholar 

  25. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer Science+Business Media; 2009.

    Book  Google Scholar 

  26. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839–43.

    Article  CAS  PubMed  Google Scholar 

  27. R Core Team. R: A language and environment for statistical computing. Vienna: 2017. http://www.R-project.org/

  28. Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust. 2004;180(6 Suppl):S57–60.

    PubMed  Google Scholar 

  29. Grol R, Wensing M: Implementation; Effective improvement of patient care. (In Dutch: Implementatie; Effectieve Verbeteringen Van Patiëntenzorg. Amsterdam: Bohn Stafleu en van Loghum; 2016.

  30. Barr J, Fraser GL, Puntillo K, et al. Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med. 2013;41(1):263–306.

    Article  PubMed  Google Scholar 

  31. NICE. DELIRIUM: diagnosis, prevention and management. London: The National Institute for Health and Clinical Excellence; 2010. https://www.nice.org.uk/guidance/cg103/evidence/cg103-delirium-full-guideline3

    Google Scholar 

  32. Siddiqi N. Predicting delirium: time to use delirium risk scores in routine practice? Age Ageing. 2016;45(1):9–10.

    Article  PubMed  Google Scholar 

  33. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605.

    Article  PubMed  Google Scholar 

  34. Siontis GC, Tzoulaki I, Castaldi PJ, Ioannidis JP. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol. 2015;68(1):25–34.

    Article  PubMed  Google Scholar 

  35. Lee A, Mu JL, Joynt GM, et al. Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation. Br J Anaesth. 2017;118(3):391–9.

    Article  CAS  PubMed  Google Scholar 

  36. van den Boogaard M, Schoonhoven L, van der Hoeven JG, van Achterberg T, Pickkers P. Incidence and short-term consequences of delirium in critically ill patients: a prospective observational cohort study. Int J Nurs Stud. 2012;49(7):775–83.

    Article  PubMed  Google Scholar 

  37. Peterson JF, Pun BT, Dittus RS, et al. Delirium and its motoric subtypes: a study of 614 critically ill patients. J Am Geriatr Soc. 2006;54(3):479–84.

    Article  PubMed  Google Scholar 

  38. Woien H, Balsliemke S, Stubhaug A. The incidence of delirium in Norwegian intensive care units; deep sedation makes assessment difficult. Acta Anaesthesiol Scand. 2013;57(3):294–302.

    Article  CAS  PubMed  Google Scholar 

  39. Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology. Br J Cancer. 1994;69(6):979–85.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Dykema J, Jones NR, Piche T, Stevenson J. Surveying clinicians by web: current issues in design and administration. Eval Health Prof. 2013;36(3):352–81.

    Article  PubMed  Google Scholar 

  41. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606.

    Article  PubMed  Google Scholar 

  42. Gusmao-Flores D, Salluh JI, Chalhub RA, Quarantini LC. The Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) and Intensive Care Delirium Screening Checklist (ICDSC) for the diagnosis of delirium: a systematic review and meta-analysis of clinical studies. Crit Care. 2012;16(4):R115.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Alexandra Cheng (ICU pharmacist, Mount Sinai Hospital) for her help in collecting the patient data and all involved research nurses and the intensive care teams of each of the participating sites for their help in collecting the patient data.

Funding

The authors received no external funding for this study.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available but are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

Authors

Contributions

Study concept and design: AW, LS, RD, PP, MvdB. Acquisition of data: AW, JD, FvH, AS, PJ, MvdJ, KS, IE, LB, AB, JM. Statistical analysis: AW, RD, MvdB. Analysis and interpretation of data: AW, LS, RD, PP, MvdB. Drafting of the manuscript: AW. Critical revision of the manuscript for important intellectual content: LS, JD, FvH, AS, PJ, MvdJ, KS, IE, LB, AB, JM, RD, PP, MvdB. Study supervision: PP, MvdB, LS. All authors read and approved the final manuscript and take responsibility for the integrity of the data and the accuracy of the data analysis.

Corresponding author

Correspondence to Mark van den Boogaard.

Ethics declarations

Ethics approval and consent to participate

The study was reviewed by the Medical Research Ethics Committee (MREC) Arnhem-Nijmegen region, The Netherlands (CMO Region Arnhem-Nijmegen, no. 2015–1782) and the local MRECs/ Institutional Review Boards (IRBs)/ Research Ethics Boards (REBs) of the participating ICUs. Each institutional MREC/IRB/REB waived the need for informed consent. Only de-identified data were entered into the study database and used for analysis.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Physician questionnaire DECISION study. (DOCX 27 kb)

Additional file 2:

Table S1. Patient and hospital characteristics. (DOCX 22 kb)

Additional file 3:

Table S2. Outcome user convenience questionnaire for ICU physicians. (DOCX 19 kb)

Additional file 4:

Table S3. Quality check delirium assessment. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wassenaar, A., Schoonhoven, L., Devlin, J.W. et al. Delirium prediction in the intensive care unit: comparison of two delirium prediction models. Crit Care 22, 114 (2018). https://doi.org/10.1186/s13054-018-2037-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13054-018-2037-6

Keywords