Skip to main content

Machine learning identifies ICU outcome predictors in a multicenter COVID-19 cohort

Abstract

Background

Intensive Care Resources are heavily utilized during the COVID-19 pandemic. However, risk stratification and prediction of SARS-CoV-2 patient clinical outcomes upon ICU admission remain inadequate. This study aimed to develop a machine learning model, based on retrospective & prospective clinical data, to stratify patient risk and predict ICU survival and outcomes.

Methods

A Germany-wide electronic registry was established to pseudonymously collect admission, therapeutic and discharge information of SARS-CoV-2 ICU patients retrospectively and prospectively. Machine learning approaches were evaluated for the accuracy and interpretability of predictions. The Explainable Boosting Machine approach was selected as the most suitable method. Individual, non-linear shape functions for predictive parameters and parameter interactions are reported.

Results

1039 patients were included in the Explainable Boosting Machine model, 596 patients retrospectively collected, and 443 patients prospectively collected. The model for prediction of general ICU outcome was shown to be more reliable to predict “survival”. Age, inflammatory and thrombotic activity, and severity of ARDS at ICU admission were shown to be predictive of ICU survival. Patients’ age, pulmonary dysfunction and transfer from an external institution were predictors for ECMO therapy. The interaction of patient age with D-dimer levels on admission and creatinine levels with SOFA score without GCS were predictors for renal replacement therapy.

Conclusions

Using Explainable Boosting Machine analysis, we confirmed and weighed previously reported and identified novel predictors for outcome in critically ill COVID-19 patients. Using this strategy, predictive modeling of COVID-19 ICU patient outcomes can be performed overcoming the limitations of linear regression models.

Trial registration “ClinicalTrials” (clinicaltrials.gov) under NCT04455451.

Background

The COVID-19 pandemic hit Germany in spring 2020 and since then intensive care resources were heavily utilized up to now [1]. Although large numbers of SARS-CoV-2 patients required intensive care unit (ICU) admission, ICU capacity in Germany was not exceeded. However, risk stratification and prediction of outcomes continues to be challenging. Several investigators have reported their ICU COVID-19 experience during this time period, yet these data show great variability in the number of cases and outcomes reported [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16].

Few of these reports attempted to identify risk factors predicting morbidity, mortality and overall clinical outcome. This may be the result of the reporting of (1) incomplete data sets earlier in the pandemic as many patient were still undergoing ICU care for SARS-CoV-2 infection [10, 13, 15, 16], and/or (2) data sets biased by the need to triage ICU care to patients in the face of the exhaustion of local/regional ICU capacity [7, 10, 14, 15]. Nonetheless, there was consensus that SARS-CoV-2 ICU patients experienced lengthy ICU stays with ICU mortality in the range of 25 to 41% [14, 17]. Classical statistical analysis identified risk factors in these patient populations including age, renal function, the degree of pulmonary compromise and severity of acute respiratory distress syndrome (ARDS). But standard statistical techniques are limited in their ability to integrate diverse data types such as past medical history, therapeutic ICU interventions and many more in relation to clinical outcome variables [18].

To overcome these limitations, we employed machine learning methods to optimize risk stratification and prediction of overall outcomes for individual COVID-19 ICU patients. It has been recently shown that machine learning (ML) algorithms in combination with numerous, multidimensional variables with non-linear relationships may have advantages in clinical outcome prediction. Machine learning strategies were found to be superior to classical methods of outcome prediction typically used in cardiovascular pathologies [18, 19]. To take advantage of this superior technique for outcome prediction, we investigated 1186 PCR-confirmed COVID-19 patients receiving ICU care at 27 German hospitals that were enrolled retrospectively and prospectively. The aim of this study is to investigate whether ML can provide additional and interpretable insights for outcome prediction and weigh the identified outcome factors in COVID-19 ICU patients.

Methods

Study design, setting and participants

This multi-center retrospective—prospective cohort study was performed with 27 participating German hospitals (Additional file 1: Table E1 and Fig. 1). An ethics approval was obtained from the participating hospitals’ Institutional Review Boards. The study was registered in “ClinicalTrials” (clinicaltrials.gov) under NCT04455451. COVID-19 patients 18 years and older requiring ICU admission between 1st January 2020 and 4th May 2021 at a participating center were recruited for this study. Patients were recruited either retrospectively (1st January 2020 to 31st July 2020) or prospectively (29th September 2020 to 4th May 2021). Inclusion criteria were the requirement for ICU treatment due to COVID-19 confirmed by a positive SARS-CoV-2 PCR test. The local investigator confirmed the accuracy and completeness of all entered data. A secure electronic research data capture system (REDCap) was used to collect and manage study data in a pseudonymous fashion [20, 21].

Table 1 Clinical characteristics of N = 1186 patients included in the study; clinical and laboratory parameters
Fig. 1
figure 1

Descriptive data of patients included into the study population. (n = 596 retrospective cohort and n = 443 prospective cohort). A Distribution Age B Horovitz quotient at admission C Murray lung injury score and SOFA score without GCS at admission D Survival rates E Interaction of Murray long injury score and admission status F Laboratory values. Grey indicates patients that did not survive ICU therapy, orange indicates patients that did survive ICU therapy

Variables and measurements

During the data collection process demographic data, past medical history, previous medications, current illness data, laboratory values as well as outcome data were collected. A total of 49 variables were used for the ML models (Additional file 1: Table E8).

To allow comparability of intubated and spontaneously breathing patients the Sequential Organ Failure Assessment (SOFA) score was calculated without the Glasgow Coma Scale (GCS) [22]. Murray Lung injury score was calculated as previously published [23]. Static compliance and driving pressure were calculated as previously described [24]. Laboratory values were converted to a common unit to permit analysis. Oxygen supply in spontaneously breathing patients was converted to an estimated FiO2 (Additional file 1: Table E2).

Table 2 Overall performance of the machine learning models for ICU outcome prediction

Bias management

Discontinuation of ICU care

107 (8.3%) patients, or their legal representative requested that ICU level care be discontinued during the ICU stay. The majority of these patients died during the ICU stay (n = 95, 88.8%). To avoid bias in predictor analyses this patient group was excluded from further analyses (for patient characteristics please see Additional file 1: Table E5). For three patients these data were not available, they were excluded from the analyses.

Dataset used for ML

Out of the 1186 patients included in the final study (Table 1), 147 were transferred to another ICU. Due to study design the ultimate ICU outcome of this subset of patients is unknown. To avoid bias in survival prediction these patients were excluded, thus ML models were trained on 1039 (complete cohort), 596 (retrospective cohort) and 443 (prospective cohort) patients.

Statistical analyses

Observed parameters were assessed for their distribution. Outliers were excluded by visual assessment of clinical validity based on the distribution plots (excluded data points are provided in Additional file 1: Table E3). Baseline characteristics of all patients were evaluated. Continuous variables are reported as either means and standard deviation (SD) if normally distributed or as median and interquartile ranges (IQR) if not normally distributed. Shapiro–Wilk-Test was used prior to Student’s t test or Wilcoxon rank sum test. Kaplan–Meier estimators were compared using Log-Rank-Test. Categorical variables were compared using the Fisher’s Exact Test. A sample size calculation was not performed. Study size is defined by the available datasets in the recruitment period. All statistical analyses were performed in R (version 4.0.3) and JMP (version 15.2.0, SAS Institute, Cary, USA).

Description of machine learning process

Variables are referred to as features in machine learning (ML) but for consistency we will refer to them as variables. For a detailed description of the machine learning process please see Additional file 1: Table E7. We trained Support Vector Classifier (SVC), Random Forest Classifier (RF), and EBM with a fivefold stratified Cross Validation (CV) by using 80% of the data for training and 20% of the data for testing. We excluded variables with more than 30% of data missing (see Additional file 1: Table E7). For all ML-methods, we applied one-hot encoding for categorical data, i.e. creating indicator columns for each category (including missing values). We converted Boolean data to numerical values zero and one. We performed a hyper-parameter optimization across all ML-algorithms with nested CV techniques [25]. Performance of the models was evaluated as the average of balanced accuracy and the area under precision-recall curve (PR-AUC) per fold of CV. A regular accuracy or AUC would be biased towards the overrepresented class (“survival”). In order to verify the robustness of our results in light of the imbalanced outcome variable, we used both over-sampling and under-sampling for the outcome “survival”. For over-sampling, the observations from the under-represented class (here: “non-survival”) were added at random to the data set. For under-sampling, the over-represented class (here”survival”) was reduced at random to the same size as the underrepresented class. We compared the ranking of variable importance and the shape function with the results from each of the fivefold stratified CV runs on the retrospective dataset. The results of each run were the same (data not shown). We further validated the results by training the ML-models with a fivefold CV for hyper-parameter optimization (RF and SVC) on the retrospective data and predicting the outcome on the prospective data (see Table 2).

For the results presented in this paper, we trained the EBM on the entire dataset (retrospective and prospective).

Rationale for the use of the explainable boosting machines model

EBMs are built on a generalized additive model (GAM) of the form

$$g\left( y \right) = \varpi_{1} f_{1} \left( {x_{1} } \right) + \varpi_{2} f_{2} \left( {x_{2} } \right) + ::: + \varpi_{p} f_{p} \left( {x_{p} } \right),$$

where \(g\) is the link function and \(f_{i} \left( {x_{i} } \right)\) the shape function for variable \(x_{i}\) and \(\varpi\)i is the weight for variable \(x_{1}\), with which each variable influences the model. In a classification problem, the link function \(g\) is a logistic function [26]. As the model is additive, each variable contributes in a modular way. This allows for an easy interpretation about the influence of a variable to the prediction (see Fig. 2A). The idea of using shape functions for each variable allows for complex relationships (even non-linear) between the variable and the outcome prediction (see Fig. 2B). Therefore, GAMs can be significantly more accurate than simple linear models [27]. We use EBMs as they additionally employ modern machine learning techniques such as bagging and boosting and have a comparable performance to state-of-the art ML techniques such as RF [27, 28]. Overall performance of the ML models was assessed by balanced accuracy and PR-AUC (Table 2).

Fig. 2
figure 2

EBM prediction model showing importance of risk factors predicting “survival” in COVID-19 ICU patients including admission data. Top A significant risk factors for outcome after analysis of admission data and weighed according to their importance for outcome. bottom) B importance of age for outcome and distribution of age data C platelet/neutrophil ratio and distribution of data on admission D initial D-dimer serum values and distribution of data determined on admission E importance of Horovitz quotient (PaO2/FiO2) for outcome and distribution of data on admission F initial hemoglobin values and distribution of data on admission G initial procalcitonin (PCT) serum values and distribution of data on admission. Grey indicates patients that did not survive ICU therapy, orange indicates patients that did survive ICU therapy

Results

Participating centers and level of care

27 ICUs participated in this observational study including 24 ICUs from university hospitals and three ICUs from regional primary and secondary care hospitals (Additional file 1: Table E1, Figure E1). All patients requiring ICU treatment could receive the full treatment possibilities including ventilation, renal replacement therapy (RRT), and extracorporeal membrane oxygenation (ECMO).

Patient characteristics and status at ICU admission

1186 patients were recruited into the study (patient selection chart, Additional file 1: Figure E2) with 713 patients in the retrospective and 473 patients in the prospective cohort. Overall patient characteristics, severity of the disease, and organ failure are given in Table 1 and Additional file 1: Table E4. Twice as many males (71.9%) than females (28.1%) were treated at the participating ICUs. The median age was 63 (IQR 54 to 73), 180 patients (15.2%) had an age below 50 years, and 6 patients (0.5%) had an age above 90 years. For age distribution and baseline parameters please see Fig. 1. Kaplan Meier Curves for probability of ICU survival according to patient age are provided in Additional file 1: Figure E3a. At ICU admission spontaneous breathing via oxygen mask, non-invasive assisted ventilation or invasive ventilation were present in 47.2%, 11%, 41.7% patients, respectively. Data for the grading of the ARDS severity were available for 1154 patients (97.3%). According to the Berlin definition ARDS was graded using the PaO2/FiO2 index as mild (16.6%), moderate (47.3%), or severe (28.4%) [29]. Additional file 1: Figure E3b provides the Kaplan Meier Curves for probability of ICU survival according to ARDS severity.

Fig. 3
figure 3

EBM prediction model showing importance of risk factors predicting need for ECMO or RRT in COVID-19 ICU patients including admission data. (a) ECMO therapy left) A significant risk factors for outcome after analysis of admission data and weighed according to their importance for outcome. Right B importance of age for outcome and distribution of age data C importance of status “intubated” on ICU admission and distribution of status D) importance of status “external transfer” on ICU admission and distribution of status E importance of Murray lung injury score and distribution of MLIS data. Green indicates patients that did not receive ECMO therapy, orange indicates patients that did receive ECMO therapy. (b) Renal Replacement Therapy (RRT). Left A significant risk factors for outcome after analysis of admission data and weighed according to their importance for outcome. Right B importance of the interaction of age and D-dimer level for outcome and distribution of data C initial creatinine values and distribution of data determined on admission D initial SOFA score w/o GCS and distribution of data determined on ICU admission. Blue indicates patients that did not receive RRT, red indicates patients that did receive RRT

Patient outcome

Overall ICU mortality was 34% for all recruited patients. Median length of ICU stay was 15 days (IQR 7 to 30 days). Mortality was significantly lower in female patients (27.6%) than in male patients (36.5%) (p = 0.0041). Mortality was highest in octogenarians with an observed mortality of 45.7% (Additional file 1: Figure E3a). 22% patients received ECMO therapy (21% in the retrospective cohort and 23.5% in the prospective cohort) with a median duration of 16 days (IQR 9 to 26). 95% of patients received veno-venous ECMO, 2% of patients received a veno-arterial ECMO and 3% received a transition from veno-venous to veno-arterial ECMO. Patients receiving ECMO therapy were significantly younger than those not receiving ECMO (57 (IQR 49 to 65) years vs. 66 (IQR 56 to 76) years; p < 0.0001). 39.3% patients, not receiving chronic dialysis prior to ICU admission, received RRT/dialysis therapy during their ICU stay (41.7% in the retrospective cohort and 35.8% in the prospective cohort).

Prediction of ICU survival by EBM models

Overall performance of the different ML models including results for balanced accuracies and precision recall area under the curve (PR-AUC) are given in Table 2. The EBM model based on variables reflecting status at ICU admission (Additional file 1: Table E8), resulted in a high precision recall area under the curve (PR-AUC) of 0.81 and a moderate balanced accuracy of 0.64 (Additional file 1: Figure E4a). The ten most important predictive variables in the admission model were according to their predictive importance: age, platelet/neutrophil ratio, D-dimer, Horowitz quotient, hemoglobin, procalcitonin, Murray lung injury score, platelet count, interaction of c-reactive protein and interleukin-6 and absolute lymphocyte count (Fig. 2). Patients’ comorbidities were not under the fifteen most important variables. As shown in the shape function for the variable age, there is a transition from improved survival to worsened survival at the age of 61 years (confidence interval (CI) 60 to 62) with a first worsening at the age of 34.7 (CI 31 to 35) years. The platelet/neutrophil ratio was the second most important parameter showing a worsened outcome above a ratio of 43.7 (CI 19.6 to 44.1). Elevated D-Dimers, for instance, affect ICU survival negatively at levels above 4.06 µg/ml (CI 3.78 to 4.07). Low Horovitz quotients demonstrated a negative impact on ICU survival with transitions for the worst impact at PaO2/FiO2 quotients below 85 (CI 84 to 86) and improved survival above 163 to 172. Overall performance and results of the EBM model was similar for the different datasets (complete, prospective and retrospective) (Additional file 1: Table E9, Figure E5a).

Predicting the need for ECMO therapy by EBM models

EBM models for the prediction of ECMO therapy resulted in a good PR-AUC of 0.69 and a good balanced accuracy of 0.73. The five most important parameters associated with ECMO therapy according to their predictive importance were: age, ventilatory status “intubated” at ICU admission, admission by external transfer, Murray lung injury score, and admission by internal transfer (reduced risk) (Fig. 3a). The shape function for the factor age showed a higher risk for ECMO therapy below the age of 70 (CI 69 to 75) years. A Murray Lung injury score above a level of 2.8 (no CI) resulted in a higher risk for ECMO therapy. Patients admitted by external transfer had a higher risk to receive ECMO therapy. Comparison of the EBM models and selected shape functions of important variables revealed similar results (Additional file 1: Table E9 and Figure E5b).

Prediction of renal replacement therapy by EBM models

Patients on chronic dialysis were excluded prior to EBM model generation. The EBM model on the complete dataset resulted in a good PR-AUC (Additional file 1: Figure E4c). The five most important parameters according to their predictive importance were: interaction of age with D-dimer level, creatinine level, SOFA score w/o GCS, interaction of BMI with creatinine, and platelet/neutrophile ratio (Fig. 3b). Patients with an age below approximately 65 years combined with elevated D-dimers had a higher risk for the need of RRT (see heatmap of interaction of age and D-dimers in Fig. 3b). An elevated creatinine level above 1.3 mg/dl (no CI) at ICU admission, as well as a SOFA score w/o GCS above 5 (no CI) resulted in a higher risk to receive RRT during ICU stay. Throughout all EBM models, creatinine and bilirubin levels showed a reverse correlation relationship.

Discussion

In this multi-center retrospective—prospective cohort study we identified and weighed possible predictive factors on COVID-19 outcome using a machine learning approach on 49 variables. Using the present ML approach, we confirmed previously reported factors and extend knowledge to novel factors and factor combinations likely predicting outcome in COVID-19 patients. Shape functions for each of these variables show the individual influence of the variable for the prediction of the outcome. For ICU survival these include age, platelet/neutrophil ratio, D-dimers, and ARDS severity. The most important factors for the prediction of RRT need include the combination of Age and D-Dimers, Creatinine levels and SOFA score without GCS.

Previous studies have shown that older age, obesity, diabetes, being immunocompromised, lower PaO2/FiO2, higher hemodynamic and renal SOFA score at ICU admission were independently associated with 90-day mortality in COVID-19 [14]. This has also been reported by other investigators, yet they did not show individual cutoff values nor weigh the individual importance for the identified factors [30, 31]. To exclude an early effect or a late effect as seen when logistic regression is performed, we included almost all admission variables collected for our cohort. Variable selection influencing outcome can be performed in ML models but is less crucial than for logistic regression. We refrained from such a variable selection in our EBM model’s decision process. In our analysis we were able to confirm that age and pulmonary function on admission are important predictors in COVID-19 ICU patients. The present shape functions clearly show a non-linear association between the predictive factors and the outcome variable. Patient’s age, for instance, as the most important predictive factor, shows a higher chance for ICU survival below 61 years. Additionally, the ML approach identified the D-dimer level and platelet/neutrophil ratio at ICU admission as important factors. This is especially interesting in the context of reported thrombotic complications of COVID-19 patients [32, 33]. When activated, neutrophils complex with platelets to form platelet-neutrophil complexes (PNCs) activating both cell types. These PNCs enhance inflammation, increases neutrophil extracellular trap formation, and result in micro-thrombosis [34, 35]. The same is applicable when looking at D-dimer levels. High D-dimer levels reflect an activation of inflammation and the formation of micro-thrombi with neutrophil extracellular trap formation. We can therefore say that our data reflects the inflammatory markers known from translational science and confirm their relevance to outcome [35].

In everyday clinical practice, it is of great interest to assess the further course of patients in intensive care, such as a necessity for renal replacement or ECMO therapy. The present ML model predicting the need for ECMO therapy identified age and pulmonary compromise (Murray lung injury score) as important factors. Admission both from an external hospital and already in an intubated state are associated with the need for ECMO therapy. This result is not surprising, as both younger and more severely pulmonary compromised patients were typically transferred for ECMO therapy to our participating centers [36]. Our ML models assessing the need for RRT include age as an important factor as well as variables quantifying disease severity (SOFA score) or inflammatory and thrombotic activity (D-dimers and Platelet/neutrophil ratio). Our models do not only permit the identification of risk factors in COVID-19 patients, they also provide insights to the weight of each individual variable for the selected ICU outcome of the individual patient [18, 37]. The ML models chosen allow for transparent assessment of various variables in a non-linear fashion which overcomes limitations of currently employed regression models. The use of shape functions in GAMs for each variable allows for complex relationships (even non-linear) between the variable and the outcome prediction. Therefore, EBMs can be significantly more accurate than simple linear models [27]. Interactions of different variables extend the analyzing capabilities of the ML approach. Overall, the results from the EBM offer a greater degree of interpretability than a p-value of a linear regression, or an odds ratio analysis. As shown in Figs. 2 and 3 the visualizations offer insight into transition values from positive to negative impact, plateaus, as well as confidence intervals as a certainty measure.

A limitation of the present study is that we were not able to include even more patients into the analysis. This is of course a valid point of criticism, yet the data used for our analyses were manually collected and curated. The data was not simply exported from an electronic medical record where missing data are prevalent and validity of the information has not been confirmed. Missing data often needs to be imputed prior to analysis. As a result of the design of our study, we were largely able to reduce imputation of missing data, again adding to the significance of our findings. The predictiveness of the models presented here differed for the three outcomes (survival, ECMO, RRT). This is likely due to the underlying dataset containing more information for predicting e.g. survival compared to ECMO. Since the study was designed with a focus on predicting survival, some variables which might better predict ECMO or RRT might not have been included in this study (for details see Additional file 1: Table E9). Furthermore, whereas the validation of survival prediction was largely consistent between the retrospective and prospective datasets, there was more variability with regard to ECMO and RRT. A possible reason for this might be structural differences between the retro- and prospective datasets, e.g. changes in treatment or age cohort over time. However, the moderate predictive capabilities of the variables used in these ML models leave open the opportunity to add further, even translational technologies for risk prediction in future. A strength of our approach is the ability to determine a weight for individual patient factors with respect to an individual prediction. Additionally, risk factors are presented with a shape function. This allows for a more detailed interpretation and segmentation of risk factors than a simple linear incrementation, as it is the case for the linear regression. Finally, due to the imbalanced dataset (more patients survived ICU therapy, more patients did not need ECMO or RRT), our model is more reliable for predicting “survival” than “mortality”. Nonetheless, the strength of these clinical data is the generalizability across institutions and even other similarly resourced countries.

Conclusions

Yet, we present individual risk factors that can be combined for a prediction of “survival” during COVID-19 treatment and ICU course and these factors are weighed for importance. This has been done for the first time and will allow clinicians to weigh clinical criteria for outcome prediction in the patients treated.

Availability of data and materials

Data can be obtained from the authors upon reasonable request.

Abbreviations

CI:

Confidence interval

CV:

Cross-validation

DIVI:

German interdisciplinary association for intensive care and emergency medicine

EBM:

Explainable boosting machine

ECMO:

Extracorporeal membrane oxygenation

GAM:

Generalized additive model

GCS:

Glasgow Coma Scale

ICU:

Intensive Care Unit

IQR:

Inter-quartile range

PCR:

Polymerase chain reaction

PEEP:

Positive end expiratory pressure

RF:

Random forest classifier

RRT:

Renal replacement therapy

SD:

Standard deviation

SOFA:

Sequential organ failure assessment

SVC:

Support vector classifier

References

  1. Coronavirus Disease 2019 (COVID-19) Daily Situation Report of the Robert Koch Institute at 31/07/2020 https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Situationsberichte/2020-07-31-en.pdf?__blob=publicationFile.

  2. Zheng Y, Sun LJ, Xu M, Pan J, Zhang YT, Fang XL, Fang Q, Cai HL. Clinical characteristics of 34 COVID-19 patients admitted to intensive care unit in Hangzhou, China. J Zhejiang Univ Sci B. 2020;21(5):378–87.

    Article  CAS  Google Scholar 

  3. Zhang J, Liu P, Wang M, Wang J, Chen J, Yuan W, Li M, Xie Z, Dong W, Li H, et al. The clinical data from 19 critically ill patients with coronavirus disease 2019: a single-centered, retrospective, observational study. Z Gesundh Wiss. 2020;1–4.

  4. Zangrillo A, Beretta L, Scandroglio AM, Monti G, Fominskiy E, Colombo S, Morselli F, Belletti A, Silvani P, Crivellari M, et al. Characteristics, treatment, outcomes and cause of death of invasively ventilated patients with COVID-19 ARDS in Milan, Italy. Crit Care Resuscitation J Aust Acad Crit Care Med. 2020.

  5. Yang X, Yu Y, Xu J, Shu H, Xia J, Liu H, Wu Y, Zhang L, Yu Z, Fang M, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: A single-centered, retrospective, observational study. Lancet Respir Med. 2020;8(5):475–81.

    Article  CAS  Google Scholar 

  6. Wang Y, Lu X, Chen H, Chen T, Su N, Huang F, Zhou J, Zhang B, Li Y, Yan F, et al. Clinical course and outcomes of 344 intensive care patients with COVID-19. Am J Respir Crit Care Med. 2020.

  7. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O’Donnell L, Chernyak Y, Tobin KA, Cerfolio RJ, Francois F, Horwitz LI. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: Prospective cohort study. BMJ. 1966;2020:369.

    Google Scholar 

  8. Pedersen HP, Hildebrandt T, Poulsen A, Uslu B, Knudsen HH, Roed J, Poulsen TD, Nielsen HB: Initial experiences from patients with COVID-19 on ventilatory support in Denmark. Dan Med J. 2020;67(5).

  9. Ling L, So C, Shum HP, Chan PKS, Lai CKC, Kandamby DH, Ho E, So D, Yan WW, Lui G, et al. Critically ill patients with COVID-19 in Hong Kong: A multicentre retrospective observational cohort study. Crit Care Resuscit J Aust Acad Crit Care Med. 2020.

  10. Grasselli G, Zangrillo A, Zanella A, Antonelli M, Cabrini L, Castelli A, Cereda D, Coluccello A, Foti G, Fumagalli R, et al. Baseline characteristics and outcomes of 1591 patients infected with SARS-CoV-2 admitted to ICUs of the Lombardy Region, Italy. JAMA. 2020;323(16):1574–81.

    Article  CAS  Google Scholar 

  11. Goyal P, Choi JJ, Pinheiro LC, Schenck EJ, Chen R, Jabri A, Satlin MJ, Campion TR, Jr., Nahid M, Ringel JB, et al. Clinical Characteristics of Covid-19 in New York City. N Engl J Med. 2020.

  12. Cummings MJ, Baldwin MR, Abrams D, Jacobson SD, Meyer BJ, Balough EM, Aaron JG, Claassen J, Rabbani LE, Hastie J, et al. Epidemiology, clinical course, and outcomes of critically ill adults with COVID-19 in New York City: a prospective cohort study. Lancet. 2020.

  13. Bhatraju PK, Ghassemieh BJ, Nichols M, Kim R, Jerome KR, Nalla AK, Greninger AL, Pipavath S, Wurfel MM, Evans L, et al. Covid-19 in critically Ill patients in the Seattle Region—case series. N Engl J Med. 2020;382(21):2012–22.

    Article  CAS  Google Scholar 

  14. Network C-IGobotR, the C-ICUI: Clinical characteristics and day-90 outcomes of 4244 critically ill adults with COVID-19: a prospective cohort study. Intensive Care Med. 2020.

  15. Barrasa H, Rello J, Tejada S, Martin A, Balziskueta G, Vinuesa C, Fernandez-Miret B, Villagra A, Vallejo A, San Sebastian A, et al. SARS-CoV-2 in Spanish Intensive Care Units: early experience with 15-day survival in Vitoria. Anaesth Crit Care Pain Med. 2020.

  16. Cummings MJ, Baldwin MR, Abrams D, Jacobson SD, Meyer BJ, Balough EM, Aaron JG, Claassen J, Rabbani LE, Hastie J, et al. Epidemiology, clinical course, and outcomes of critically ill adults with COVID-19 in New York City: a prospective cohort study. Lancet. 2020;395(10239):1763–70.

    Article  CAS  Google Scholar 

  17. Group RC, Horby P, Lim WS, Emberson JR, Mafham M, Bell JL, Linsell L, Staplin N, Brightling C, Ustianowski A, et al. Dexamethasone in Hospitalized Patients with Covid-19 - Preliminary Report. N Engl J Med. 2020.

  18. Schwalbe N, Wahl B. Artificial intelligence and the future of global health. Lancet. 2020;395(10236):1579–86.

    Article  CAS  Google Scholar 

  19. Motwani M, Dey D, Berman DS, Germano G, Achenbach S, Al-Mallah MH, Andreini D, Budoff MJ, Cademartiri F, Callister TQ, et al. Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis. Eur Heart J. 2017;38(7):500–7.

    PubMed  Google Scholar 

  20. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, McLeod L, Delacqua G, Delacqua F, Kirby J, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208.

    Article  Google Scholar 

  21. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  Google Scholar 

  22. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, Reinhart CK, Suter PM, Thijs LG. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996; 22(7):707–710.

  23. Murray JF, Matthay MA, Luce JM, Flick MR. An expanded definition of the adult respiratory distress syndrome. Am Rev Respir Dis. 1988;138(3):720–3.

    Article  CAS  Google Scholar 

  24. Guo L, Wang W, Zhao N, Guo L, Chi C, Hou W, Wu A, Tong H, Wang Y, Wang C, et al. Mechanical ventilation strategies for intensive care unit patients without acute lung injury or acute respiratory distress syndrome: a systematic review and network meta-analysis. Crit Care. 2016;20(1):226.

    Article  Google Scholar 

  25. Cawley GC, Talbot NLC. On over-fitting in model selection and subsequent selection bias in performance evaluation. J Mach Learn Res. 2010;11:2079–107.

    Google Scholar 

  26. Hastie T, Tibshirani R. Generalized additive models. Stat Sci. 1986;1(3):297–318.

    Google Scholar 

  27. Lou Y, Caruana R, Gehrke J. Intelligible models for classification and regression. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. Association for Computing Machinery; 2012. p. 150–158.

  28. Lou Y, Caruana R, Hooker G, Gehrke J. Accurate intelligible models with pairwise interactions. 2013.

  29. Force ADT, Ranieri VM, Rubenfeld GD, Thompson BT, Ferguson ND, Caldwell E, Fan E, Camporota L, Slutsky AS. Acute respiratory distress syndrome: the Berlin definition. JAMA. 2012;307(23):2526–33.

    Google Scholar 

  30. Holler JG, Eriksson R, Jensen TO, van Wijhe M, Fischer TK, Sogaard OS, Israelsen SB, Mohey R, Fabricius T, Johnk F, et al. First wave of COVID-19 hospital admissions in Denmark: a Nationwide population-based cohort study. BMC Infect Dis. 2021;21(1):39.

    Article  CAS  Google Scholar 

  31. Becerra-Munoz VM, Nunez-Gil IJ, Eid CM, Aguado MG, Romero R, Huang J, Mulet A, Ugo F, Rametta F, Liebetrau C, et al. Clinical profile and predictors of in-hospital mortality among older patients admitted for COVID-19. Age Ageing. 2020.

  32. Greinacher A, Thiele T, Warkentin TE, Weisser K, Kyrle PA, Eichinger S. Thrombotic Thrombocytopenia after ChAdOx1 nCov-19 Vaccination. N Engl J Med. 2021.

  33. Althaus K, Moller P, Uzun G, Singh A, Beck A, Bettag M, Bosmuller H, Guthoff M, Dorn F, Petzold GC, et al. Antibody-mediated procoagulant platelets in SARS-CoV-2- vaccination associated immune thrombotic thrombocytopenia. Haematologica. 2021.

  34. Iba T, Levy JH. Inflammation and thrombosis: roles of neutrophils, platelets and endothelial cells and their interactions in thrombus formation during sepsis. J Thromb Haemost. 2018;16(2):231–41.

    Article  CAS  Google Scholar 

  35. Gould TJ, Vu TT, Swystun LL, Dwivedi DJ, Mai SH, Weitz JI, Liaw PC. Neutrophil extracellular traps promote thrombin generation through platelet-dependent and platelet-independent mechanisms. Arterioscler Thromb Vasc Biol. 2014;34(9):1977–84.

    Article  CAS  Google Scholar 

  36. Barbaro RP, Maclaren G, Boonstra PS, Iwashyna TJ, Slutsky AS, Fan E, Bartlett RH, Tonna JE, Hyslop R, Fanning JJ, et al. Extracorporeal membrane oxygenation support in COVID-19: an international cohort study of the Extracorporeal Life Support Organization registry. Lancet. 2020;396(10257):1071–8.

    Article  CAS  Google Scholar 

  37. Tokodi M, Schwertner WR, Kovacs A, Toser Z, Staub L, Sarkany A, Lakatos BK, Behon A, Boros AM, Perge P, et al. Machine learning-based mortality prediction of patients undergoing cardiac resynchronization therapy: the SEMMELWEIS-CRT score. Eur Heart J. 2020;41(18):1747–56.

    Article  Google Scholar 

Download references

Acknowledgements

The authors and collaborators thank the employees of the participating intensive care units for their immeasurable efforts in the care of COVID-19 patients. The authors thank Nico Pfeifer, Medical Informatics, University of Tübingen, Germany, for the numerous fruitful discussions and valuable input on the application of ML methods. The authors thank Sascha Rehm and Christian Erhardt (meDIC, University Hospital of Tübingen, Germany) for server and REDCap administration and support. The authors also greatly appreciate the contributions of the assistant scientists, data scientists, study nurses and physicians towards all aspects of data collection, data analysis and the general support of this study, namely: H. Gloeckner & M. Keim (Department of Anaesthesiology and Intensive Care Medicine, University Hospital Tübingen, Eberhard-Karls-University Tübingen, Tübingen, Germany), T. Simon (Department of Intensive Care Medicine, University Hospital RWTH Aachen, Aachen, Germany), S. Winands & N. Andrees & G. Vorderwülbecke & K. Steinecke (Department of Anesthesiology and Operative Intensive Care Medicine (CCM, CVK), Charité—Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Germany), K. Willemsen & B. Rahmel (Department of Anesthesiology/Intensive Care Medicine/Pain Therapy, Knappschaftskrankenhaus Bochum, Bochum, Germany), N. Theuerkauf & C. Bode & C.J. Schewe & U. Lohmer (Department of Anaesthesiology and Intensive Care Medicine, University Hospital Bonn, Bonn, Germany), S. May & P. Spieth & A. Gueldner (Department of Anesthesiology and Intensive Care Medicine, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany), T. Dimski & C.Pelletier (Department of Anaesthesiology, University Hospital Düsseldorf, Düsseldorf, Germany), A. Skarabis & M. Overbeck & B. Breuing & S. Kattner (Department of Anesthesiology and Intensive Care Medicine, University Hospital Essen, Essen, Germany), T. Bui (Department of Anaesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Frankfurt, Goethe University, Frankfurt, Germany), J. Kalbhenn (Department of Anesthesiology and Critical Care, Medical Center—University of Freiburg, Freiburg, Germany), A. Kernchen (Center for Anesthesiology, Emergency and Intensive Care Medicine, University of Göttingen, Göttingen, Germany), N. Piasta & M. Gerber (Department of Anesthesiology, University Medicine Greifswald, Greifswald, Germany), K. Ludwig & J. Mühlhaus (Department Cardiology, Angiology and Intensive Care Medicine, University Hospital Halle (Saale), Halle (Saale), Germany), M. Fiedler (Department of Anesthesiology, Heidelberg University Hospital, Heidelberg, Germany), C.Schlegel & A. Seidel (Department of Anesthesiology and Intensive Care, Leipzig University Hospital, Leipzig, Germany), H. Linnen & G. Bergmann (Department of Anesthesiology and Intensive Care, University Medical Center Schleswig-Holstein, Campus Lübeck and University of Lübeck, Lübeck, Germany), L. Bartning & K. Albizky (Department of Anaesthesiology and Intensive Care Therapy, Otto-von-Guericke-University Magdeburg, Magdeburg, Germany), C. Arndt & T. Koch & R. Axt & G. Kix ( University Hospital Marburg, UKGM, Philipps University Marburg, Marburg, Germany), U. Flechtner (Department of Anesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Giessen and Marburg, and Justus-Liebig University Giessen, Giessen, Germany), N. Moßbacher & S. Maluche & P. Feddersen (Departemnt of Anesthesiology and Intensive Care, Technical University of Munich, School of Medicine, Klinikum rechts der Isar, Munich, Germany), S. Kaiser (Department for Anesthesiology, Intensive Care Medicine, Emergency Medicine and Pain Medicine, St. Elisabethen Klinikum, Ravensburg, Germany), K. Meiers (Department of Anesthesiology, Intensive Care Medicine and Pain Medicine, Saarland University Hospital Medical Center, Homburg/Saar, Germany), F. Branz & S. Hoheisen & S. Appel (Department of Anaesthesiology and Intensive Care Medicine, Ulm University, Ulm, Germany), J.M. Defosse & F. Strobel (Department of Anaesthesiology and Intensive Care Medicine, Cologne-Merheim Medical Centre, Witten/Herdecke University, Germany), P. Kranke & C. Lotz (Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Wuerzburg, University Wuerzburg, Wuerzburg, Germany).

Funding

Open Access funding enabled and organized by Projekt DEAL. This work was funded by a grant from the Deutsche Forschungsgemeinschaft DFG RO 3671/8-1 to P.R. O.K. received funding from the Bundesministerium für Bildung und Forschung (BMBF) (DIFUTURE, 01ZZ1804D). Funding of the participating centers: Department of Anaesthesiology, Intensive Care Medicine and Pain Therapy, University Hospital Frankfurt, Goethe University, Frankfurt, Germany: This work was funded by the grant Horizon 2020 Framework Programme, Call: H2020-SC1-PHE-CORONAVIRUS-2020–2-CNECT; Project: 101015930 — ENVISION to K.Z.

Author information

Authors and Affiliations

Authors

Contributions

HM and PR developed the concept of this study including study design, data handling, data analysis, interpretation of results and drafting of the first manuscript. SB performed database planning and set-up. SL, SB and RV performed data analysis, statistics, and visualization. HM, PR, SL and SB verified the underlying data. SL and RV performed the machine-learning analysis. BJG supported database planning and data analysis. OK supported database planning and setup. BJG, SL, SB and NM critically reviewed the manuscript. MK, HH, VM, PM, GM, JB, BN, ML, CD, AE, FS, TR, CP, TS, TK, TBra, DKM, TBre, MB, KZ, EA, MP, OM; CS, DS, MW, FF, CN, FP, TW, CK, GS, TL, AS, AM, MW, BJ, FW, PM and JH reviewed patient charts, entered data into the electronic database and critically reviewed the manuscript. HM and PR finalized the manuscript. All authors approved the final version.

Corresponding authors

Correspondence to Harry Magunia or Peter Rosenberger.

Ethics declarations

Ethics approval and consent to participate

An ethics approval was obtained by all participating centers from their Institutional Review Boards (IRB) prior to data acquisition. German privacy regulations do not require individual patient consent for retrospective data acquisition or use. All prospectively enrolled patients gave informed consent.

Consent for publication

Not applicable.

Competing interests

Harry Magunia: HM received a speaker’s honorarium from CSL Behring (Germany) outside the submitted work. Simone Lederer: SL has no competing interests to report. Raphael Verbuecheln: RV has no competing interests to report. Bryant Joseph Gilot: BJG has no competing interests to report. Michael Koeppen: MK has no competing interests to report. Helene A Haeberle: HH has no competing interests to report. Valbona Mirakaj: VM has no competing interests to report. Pascal Hoffmann: PH has no competing interests to report. Gernot Marx: GM reports personal fees from Philips Health Care, personal fees from B. Braun, during the conduct of the study; and GM is co-founder of Clinomics GmbH, Germany. Johannes Bickenbach: JB reports personal fees from Biotest AG, Germany, during the conduct of the study. Boris Nohe: BN has no competing interests to report. Michael Lay: ML has no competing interests to report. Claudia Spies: CS reports no conflicts of interests within the submitted manuscript, she reports grants from outside the submitted manuscript from Drägerwerk AG & Co. KGaA, grants from Deutsche Forschungsgemeinschaft / German Research Society, grants from Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) /German Aerospace Center, grants from Einstein Stiftung Berlin/ Einstein Foundation Berlin, grants from Gemeinsamer Bundesausschuss / Federal Joint Committee (G-BA), grants from Inneruniversitäre Forschungsförderung / Inner University Grants, grants from Projektträger im DLR / Project Management Agency, grants from Stifterverband/Non-Profit Society Promoting Science and Education, grants from WHOCC, grants from Baxter Deutschland GmbH, grants from Cytosorbents Europe GmbH, grants from Edwards Lifesciences Germany GmbH, grants from Fresenius Medical Care, grants from Grünenthal GmbH, grants from Masimo Europe Ltd., grants from Pfizer Pharma PFE GmbH, personal fees from Georg Thieme Verlag, grants from Dr. F. Köhler Chemie GmbH, grants from Sintetica GmbH, grants from Stifterverband für die deutsche Wissenschaft e.V. / Philips, grants from Stiftung Charité, grants from AGUETTANT Deutschland GmbH, grants from AbbVie Deutschland GmbH & Co. KG, grants from Amomed Pharma GmbH, grants from InTouch Health, grants from Copra System GmbH, grants from Correvio GmbH, grants from Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V., grants from Deutsche Gesellschaft für Anästhesiologie & Intensivmedizin (DGAI, grants from Stifterverband für die deutsche Wissenschaft e.V. / Metronic, grants from Philips ElectronicsNederland BV, grants from BMG, grants from BMBF, grants from BMBF, grants from Deutsche Forschungsgemeinschaft / German Research Society, outside the submitted work; In addition, Dr. Spies has a patent 10 2014 215 211.9 licensed, a patent 10 2018 114 364.8 licensed, a patent 10 2018 110 275.5 licensed, a patent 50 2015 010 534.8 licensed, a patent 50 2015 010 347.7 licensed, and a patent 10 2014 215 212.7 licensed. Andreas Edel: AE has no competing interests to report. Fridtjof Schiefenhövel: FS has no competing interests to report. Tim Rahmel: TR has no competing interests to report. Christian Putensen: CP has no competing interests to report. Timur Sellmann: TS has no competing interests to report. Thea Koch: TK has no competing interests to report. Timo Brandenburger: TBra has no competing interests to report. Detlef Kindgen-Milles: DKM has co competing interests to report. Thorsten Brenner: TBre reports grants from Deutsche Forschungsgemeinschaft (DFG), Dietmar Hopp Stiftung and Stiftung Universitätsmedizin Essen, personal fees from CSL Behring GmbH, Schöchl medical education GmbH, Boehringer Ingelheim Pharma GmbH, Biotest AG, Baxter Deutschland GmbH, Astellas Pharma GmbH, B. Braun Melsungen AG and MSD Sharp & Dohme GmbH outside the submitted work. Marc Berger: MB has no competing interests to report. Kai Zacharowski: KZ has no competing interests to report. Elisabeth Adam: EA has no competing interests to report. Matthias Posch: MP has no competing interests to report. Onnen Moerer: OM has no competing interests to report. Christian S. Scheer: CS has no competing interests to report. Daniel Sedding: DS has no competing interests to report. Markus A. Weigand: MW has no competing interests to report. Falk Fichtner: FF has no competing interests to report. Carla Nau: CN has no competing interests to report. Florian Prätsch: FP has no competing interests to report. Thomas Wiesmann: TW reports personal fees from Pajunk, Germany, personal fees from Vygon, Germany, outside the submitted work. Christian Koch: CK has no competing interests to report. Gerhard Schneider: GS has no competing interests to report. Tobias Lahmer: TL has no competing interests to report. Andreas Straub: AS reports personal fees from CSL Behring GmbH (Munich, Germany), personal fees from Schöchl Medical Education GmbH (Mattsee, Austria), personal fees from Aspen Germany GmbH (Munich, Germany), outside the submitted work. Andreas Meiser: AM reports personal fees from Sedana Medical, Danderyd, Sweden, outside the submitted work. Manfred Weiss: MW has no competing interests to report. Bettina Jungwirth: BJ has no competing interests to report. Frank Wappler: FW has no competing interests to report. Patrick Meybohm: PM has no competing interests to report. Johannes Herrmann: JH has no competing interests to report. Nisar Malek: NM has no competing interests to report. Oliver Kohlbacher: OK has no competing interests to report. Stephanie Biergans: SB has no competing interests to report. Peter Rosenberger: PR has no competing interests to report.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

. Supplementary tables and figures.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Magunia, H., Lederer, S., Verbuecheln, R. et al. Machine learning identifies ICU outcome predictors in a multicenter COVID-19 cohort. Crit Care 25, 295 (2021). https://doi.org/10.1186/s13054-021-03720-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13054-021-03720-4

Keywords