- Research Letter
- Open Access
Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU
Critical Care volume 24, Article number: 123 (2020)
E-PRE-DELIRIC is a point-of-admission ICU delirium risk prediction tool , with reported good or moderate performance [2,3,4]. In this study, we assessed its performance in a large UK teaching hospital general ICU using routinely collected data, as approved by the local Research Data Governance Committee.
We retrospectively analysed data for 2445 consecutive ICU admissions (November 2014 to June 2017). Patients were routinely assessed for delirium, using twice daily Confusion Assessment Method for the ICU (CAM-ICU) assessment . As in previous E-PRE-DELIRIC studies [1,2,3,4], delirium was defined as any positive CAM-ICU assessment or antipsychotic initiation while on ICU.
We adopted the original E-PRE-DELIRIC exclusion criteria , excluding 683 ICU admissions for ICU stay < 24 h (425 admissions), incomplete CAM-ICU data (152), delirium on admission (50), comatose throughout entire ICU stay (47), and age under 18 (9). Sixteen admissions were excluded due to missing E-PRE-DELIRIC components; 1746 admissions (1569 unique patients) remained for analysis; this 71.4% inclusion rate is consistent with previous studies (Table 1).
Results and discussion
Seven hundred sixty-three delirium cases were identified (43.7% of ICU admissions), a higher incidence than reported previously (Table 1). This is likely due to differences in the study population compared to previous studies: more patients were classified as urgent, the mean APACHE II score was higher, and median length of stay (LoS) was longer (Table 1).
The mean E-PRE-DELIRIC score was 0.269 (Q1–Q3; 0.154–0.371). The histogram of E-PRE-DELIRIC scores shows extensive overlap between patients who did and did not develop delirium (Fig. 1a). The receiver operator characteristic (ROC) curve (Fig. 1b) and the precision-recall (PR) curve (Fig. 1c), showing precision (positive predictive value (PPV)) against recall (sensitivity), both indicate moderate-to-poor discriminative performance. The area under the ROC (AUROC) was 0.628 (95% CI 0.602–0.653). The area under the PR curve (AUPRC) was 0.534. For sensitivity > 0.1, PPV was between 0.437 and 0.585, indicating only around half of the patients predicted to develop delirium actually did, in a population with 43.7% incidence. Refitting the E-PRE-DELIRIC logistic regression model to our data hardly improved discrimination: AUROC was 0.648 (95% CI 0.622–0.673) and AUPRC was 0.566.
The calibration plot, of predicted risk against observed delirium rate, shows the risk of delirium is considerably underestimated, especially in patients with predicted risk of delirium less than 0.5 (Fig. 1d). Poor calibration is corroborated by the calibration slope model logit(probability of delirium) = alpha + beta ×logit(p), where p is the E-PRE-DELIRIC score . The estimated slope beta = 0.58 (95% CI 0.46–0.71) is significantly below 1, indicating the predicted probabilities are overly variable; and the estimated intercept alpha = 0.84 (95% CI 0.74–0.95) is significantly above 0 when fixing beta = 1, indicating the predicted probabilities are predominantly too low. E-PRE-DELIRIC is particularly poorly calibrated for the surgical patients in the study, many of whom have major intraabdominal pathology: those with predicted risk < 10% had an observed incidence of 26%.
Of 763 delirium cases, 563 were CAM-ICU positive and 200 were included due to antipsychotic initiation. When including only CAM-ICU-positive delirium, calibration was improved (alpha = 0.29) but remained overly variable (beta = 0.52), while discrimination was similar (AUROC 0.615; AUPRC 0.396, with 32.2% observed incidence).
While E-PRE-DELIRIC is intended as a point-of-admission score, some of its exclusion criteria are retrospective (LoS; CAM-ICU completeness; comatose throughout). To assess real-world performance, we repeated our analysis without these criteria. The AUROC (0.615) and AUPRC (0.423, with 35.0% observed incidence) remained similar.
In this population, the E-PRE-DELIRIC score is not as discriminative or as well calibrated as previously reported. PPV was only slightly higher than delirium incidence, meaning the utility of E-PRE-DELIRIC for guiding clinical decision-making in this population is limited.
Availability of data and materials
The data that support the findings of this study are available from Cambridge Clinical Informatics, but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. The data are anonymised, but to preserve patient confidentiality and privacy, the Data Use Agreement states that the data cannot be deposited into open access repositories of any kind. Anyone wishing to access data must submit and receive approval for access to these data from the Cambridge Clinical Informatics Research Data Governance Committee.
Area under the ROC curve
Area under the PR curve
Confusion Assessment Method for the ICU
Early prediction model for delirium in ICU patients
Intensive care unit
Length of stay
Positive predictive value
Receiver operator characteristic
Wassenaar A, van den Boogaard M, van Achterberg T, Slooter AJC, Kuiper MA, Hoogendoorn ME, et al. Multinational development and validation of an early prediction model for delirium in ICU patients. Intensive Care Med. 2015;41:1048–56.
Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC, Jorens PG, et al. Delirium prediction in the intensive care unit: comparison of two delirium prediction models. Crit Care. 2018;22:114.
Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC, Jorens PG, et al. External validation of two models to predict delirium in critically ill adults using either the Confusion Assessment Method-ICU or the Intensive Care Delirium Screening Checklist for delirium assessment. Crit Care Med. 2019;47:e827–35.
Green C, Bonavia W, Toh C, Tiruvoipati R. Prediction of ICU delirium: Validation of current delirium predictive models in routine clinical practice. Crit Care Med. 2019;47:428–35.
Ely EW, Margolin R, Francis J, May L, Truman B, Dittus R, et al. Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit Care Med. 2001;29:1370–9.
Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–W73.
This study was supported by the UK National Institute for Health Research (NIHR) through the Cambridge Biomedical Research Centre (BRC), with data provided by Cambridge Clinical Informatics (Led by Drs. Afzal Chaudhry and Lydia Drumright). RJBG was supported by the UK Medical Research Council [programme code MC_UU_00002/2]. We are grateful to Vince Taylor for his careful work extracting data for this study.
This study was supported by the UK National Institute for Health Research (NIHR) through the Cambridge Biomedical Research Centre (BRC), with data provided by Cambridge Clinical Informatics (led by Drs. Afzal Chaudhry and Lydia Drumright). RJBG was supported by the UK Medical Research Council [programme code MC_UU_00002/2]. The funders had no role in the design, collection, analysis, interpretation, or writing of the manuscript.
Ethics approval and consent to participate
The use of the anonymous data used in this study was approved by the Cambridge Clinical Informatics Research Data Governance Committee.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cowan, S.L., Preller, J. & Goudie, R.J.B. Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU. Crit Care 24, 123 (2020). https://doi.org/10.1186/s13054-020-2838-2
- Clinical scoring systems
- Critical care
- Intensive care
- Prediction model