Skip to main content

Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU

Introduction

E-PRE-DELIRIC is a point-of-admission ICU delirium risk prediction tool [1], with reported good or moderate performance [2,3,4]. In this study, we assessed its performance in a large UK teaching hospital general ICU using routinely collected data, as approved by the local Research Data Governance Committee.

Methods

We retrospectively analysed data for 2445 consecutive ICU admissions (November 2014 to June 2017). Patients were routinely assessed for delirium, using twice daily Confusion Assessment Method for the ICU (CAM-ICU) assessment [5]. As in previous E-PRE-DELIRIC studies [1,2,3,4], delirium was defined as any positive CAM-ICU assessment or antipsychotic initiation while on ICU.

We adopted the original E-PRE-DELIRIC exclusion criteria [1], excluding 683 ICU admissions for ICU stay < 24 h (425 admissions), incomplete CAM-ICU data (152), delirium on admission (50), comatose throughout entire ICU stay (47), and age under 18 (9). Sixteen admissions were excluded due to missing E-PRE-DELIRIC components; 1746 admissions (1569 unique patients) remained for analysis; this 71.4% inclusion rate is consistent with previous studies (Table 1).

Table 1 Patient characteristics in this study, the E-PRE-DELIRIC development dataset [1] and other validation studies [2,3,4]

Results and discussion

Seven hundred sixty-three delirium cases were identified (43.7% of ICU admissions), a higher incidence than reported previously (Table 1). This is likely due to differences in the study population compared to previous studies: more patients were classified as urgent, the mean APACHE II score was higher, and median length of stay (LoS) was longer (Table 1).

The mean E-PRE-DELIRIC score was 0.269 (Q1–Q3; 0.154–0.371). The histogram of E-PRE-DELIRIC scores shows extensive overlap between patients who did and did not develop delirium (Fig. 1a). The receiver operator characteristic (ROC) curve (Fig. 1b) and the precision-recall (PR) curve (Fig. 1c), showing precision (positive predictive value (PPV)) against recall (sensitivity), both indicate moderate-to-poor discriminative performance. The area under the ROC (AUROC) was 0.628 (95% CI 0.602–0.653). The area under the PR curve (AUPRC) was 0.534. For sensitivity > 0.1, PPV was between 0.437 and 0.585, indicating only around half of the patients predicted to develop delirium actually did, in a population with 43.7% incidence. Refitting the E-PRE-DELIRIC logistic regression model to our data hardly improved discrimination: AUROC was 0.648 (95% CI 0.622–0.673) and AUPRC was 0.566.

Fig. 1
figure1

a Histogram of predicted risk of delirium by delirium status. b Receiver operator characteristic plot, with labels indicating the corresponding threshold and the dashed line indicating the line of no discrimination. c Precision-recall plot, with the 43.7% observed incidence indicated by the dashed line. d Calibration plot (with 95% CI), by tenths of predicted risk, with the dashed line indicating perfect calibration

The calibration plot, of predicted risk against observed delirium rate, shows the risk of delirium is considerably underestimated, especially in patients with predicted risk of delirium less than 0.5 (Fig. 1d). Poor calibration is corroborated by the calibration slope model logit(probability of delirium) = alpha + beta ×logit(p), where p is the E-PRE-DELIRIC score [6]. The estimated slope beta = 0.58 (95% CI 0.46–0.71) is significantly below 1, indicating the predicted probabilities are overly variable; and the estimated intercept alpha = 0.84 (95% CI 0.74–0.95) is significantly above 0 when fixing beta = 1, indicating the predicted probabilities are predominantly too low. E-PRE-DELIRIC is particularly poorly calibrated for the surgical patients in the study, many of whom have major intraabdominal pathology: those with predicted risk < 10% had an observed incidence of 26%.

Of 763 delirium cases, 563 were CAM-ICU positive and 200 were included due to antipsychotic initiation. When including only CAM-ICU-positive delirium, calibration was improved (alpha = 0.29) but remained overly variable (beta = 0.52), while discrimination was similar (AUROC 0.615; AUPRC 0.396, with 32.2% observed incidence).

While E-PRE-DELIRIC is intended as a point-of-admission score, some of its exclusion criteria are retrospective (LoS; CAM-ICU completeness; comatose throughout). To assess real-world performance, we repeated our analysis without these criteria. The AUROC (0.615) and AUPRC (0.423, with 35.0% observed incidence) remained similar.

Conclusion

In this population, the E-PRE-DELIRIC score is not as discriminative or as well calibrated as previously reported. PPV was only slightly higher than delirium incidence, meaning the utility of E-PRE-DELIRIC for guiding clinical decision-making in this population is limited.

Availability of data and materials

The data that support the findings of this study are available from Cambridge Clinical Informatics, but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. The data are anonymised, but to preserve patient confidentiality and privacy, the Data Use Agreement states that the data cannot be deposited into open access repositories of any kind. Anyone wishing to access data must submit and receive approval for access to these data from the Cambridge Clinical Informatics Research Data Governance Committee.

Abbreviations

AUROC:

Area under the ROC curve

AUPRC:

Area under the PR curve

CAM-ICU:

Confusion Assessment Method for the ICU

E-PRE-DELIRIC:

Early prediction model for delirium in ICU patients

ICU:

Intensive care unit

LoS:

Length of stay

PPV:

Positive predictive value

PR:

Precision-recall

ROC:

Receiver operator characteristic

References

  1. 1.

    Wassenaar A, van den Boogaard M, van Achterberg T, Slooter AJC, Kuiper MA, Hoogendoorn ME, et al. Multinational development and validation of an early prediction model for delirium in ICU patients. Intensive Care Med. 2015;41:1048–56.

    CAS  Article  Google Scholar 

  2. 2.

    Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC, Jorens PG, et al. Delirium prediction in the intensive care unit: comparison of two delirium prediction models. Crit Care. 2018;22:114.

    Article  Google Scholar 

  3. 3.

    Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC, Jorens PG, et al. External validation of two models to predict delirium in critically ill adults using either the Confusion Assessment Method-ICU or the Intensive Care Delirium Screening Checklist for delirium assessment. Crit Care Med. 2019;47:e827–35.

    Article  Google Scholar 

  4. 4.

    Green C, Bonavia W, Toh C, Tiruvoipati R. Prediction of ICU delirium: Validation of current delirium predictive models in routine clinical practice. Crit Care Med. 2019;47:428–35.

  5. 5.

    Ely EW, Margolin R, Francis J, May L, Truman B, Dittus R, et al. Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit Care Med. 2001;29:1370–9.

    CAS  Article  Google Scholar 

  6. 6.

    Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–W73.

    Article  Google Scholar 

Download references

Acknowledgements

This study was supported by the UK National Institute for Health Research (NIHR) through the Cambridge Biomedical Research Centre (BRC), with data provided by Cambridge Clinical Informatics (Led by Drs. Afzal Chaudhry and Lydia Drumright). RJBG was supported by the UK Medical Research Council [programme code MC_UU_00002/2]. We are grateful to Vince Taylor for his careful work extracting data for this study.

Funding

This study was supported by the UK National Institute for Health Research (NIHR) through the Cambridge Biomedical Research Centre (BRC), with data provided by Cambridge Clinical Informatics (led by Drs. Afzal Chaudhry and Lydia Drumright). RJBG was supported by the UK Medical Research Council [programme code MC_UU_00002/2]. The funders had no role in the design, collection, analysis, interpretation, or writing of the manuscript.

Author information

Affiliations

Authors

Contributions

SLC interpreted and guided the statistical analyses and drafted the manuscript. JP conceived the study, interpreted the study results, and substantively revised the manuscript. RJBG designed the study, conducted and interpreted the statistical analyses, and substantively revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Robert J. B. Goudie.

Ethics declarations

Ethics approval and consent to participate

The use of the anonymous data used in this study was approved by the Cambridge Clinical Informatics Research Data Governance Committee.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cowan, S.L., Preller, J. & Goudie, R.J.B. Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU. Crit Care 24, 123 (2020). https://doi.org/10.1186/s13054-020-2838-2

Download citation

Keywords

  • Clinical scoring systems
  • Critical care
  • Delirium
  • Intensive care
  • Prediction model
  • Validation