Novel criteria to classify ARDS severity using a machine learning approach

Background Usually, arterial oxygenation in patients with the acute respiratory distress syndrome (ARDS) improves substantially by increasing the level of positive end-expiratory pressure (PEEP). Herein, we are proposing a novel variable [PaO2/(FiO2xPEEP) or P/FPE] for PEEP ≥ 5 to address Berlin’s definition gap for ARDS severity by using machine learning (ML) approaches. Methods We examined P/FPE values delimiting the boundaries of mild, moderate, and severe ARDS. We applied ML to predict ARDS severity after onset over time by comparing current Berlin PaO2/FiO2 criteria with P/FPE under three different scenarios. We extracted clinical data from the first 3 ICU days after ARDS onset (N = 2738, 1519, and 1341 patients, respectively) from MIMIC-III database according to Berlin criteria for severity. Then, we used the multicenter database eICU (2014–2015) and extracted data from the first 3 ICU days after ARDS onset (N = 5153, 2981, and 2326 patients, respectively). Disease progression in each database was tracked along those 3 ICU days to assess ARDS severity. Three robust ML classification techniques were implemented using Python 3.7 (LightGBM, RF, and XGBoost) for predicting ARDS severity over time. Results P/FPE ratio outperformed PaO2/FiO2 ratio in all ML models for predicting ARDS severity after onset over time (MIMIC-III: AUC 0.711–0.788 and CORR 0.376–0.566; eICU: AUC 0.734–0.873 and CORR 0.511–0.745). Conclusions The novel P/FPE ratio to assess ARDS severity after onset over time is markedly better than current PaO2/FiO2 criteria. The use of P/FPE could help to manage ARDS patients with a more precise therapeutic regimen for each ARDS category of severity. Supplementary Information The online version contains supplementary material available at 10.1186/s13054-021-03566-w.


Background
Acute respiratory distress syndrome (ARDS) is an acute and intense inflammatory disease process of the lungs with an associated high mortality rate of about 40% in non-COVID-19 ARDS patients [1,2]. ARDS is a highly heterogeneous syndrome without a specific diagnostic test [3][4][5]. According to the LUNG-SAFE study, ARDS is unrecognized in more than half of patients at the time of fulfillment of ARDS criteria [1]. The current "Berlin definition" is under controversy [5][6][7][8]. The previous American-European Consensus Conference (AECC) [9] and the Berlin definitions are predominantly based on the value of the PaO 2 /FiO 2 ratio at the time of ARDS onset [10].
A working definition of ARDS is essentially required for clinical trials, epidemiologic studies, and biological studies. Moreover, a definition of ARDS is required for clinicians to initiate treatments that would improve clinical outcomes [11], although stratification of ARDS-as defined by Berlin criteria-has been shown not very useful for assessing lung severity [8,12]. The empirical PaO 2 / FiO 2 cut-offs for "severity" of 100, 200, and 300 mmHg are arbitrary and poorly validated [13]. A recently published Reevaluation of Systemic Early Neuromuscular Blockade (ROSE) trial emphasized the variability of these PaO 2 /FiO 2 cut-offs as the investigators did not enroll patients based on the PaO 2 /FiO 2 at the time of ARDS onset, but based on a PaO 2 /FiO 2 < 150 mmHg within the first 48-h after ARDS diagnosis [14,15]. The PaO 2 /FiO 2 ratio strongly depends on ventilator settings, including positive end-expiratory pressure (PEEP), inspiratory/ expiratory time (I:E) ratio, and FiO 2 , and the requirement of a minimum PEEP of 5 cmH 2 O did not substantially improve Berlin prediction compared to AECC [13,16]. Besides, Berlin definition does not account for the nonlinear relationship of PaO 2 and FiO 2 [17] and has a limited predictive accuracy in recent trials [18][19][20][21].
Assessment of severity in ARDS remains a challenge. The relation between oxygenation and prognosis in ARDS varies among published reports [20]. For example, the current mild ARDS category may not be significantly associated with 28-day mortality [22][23][24]. However, although stratification of severity based on Berlin criteria may be helpful to identify severe ARDS patients, it may have less significance to differentiate between mild and moderate ARDS [20]. A recent study identified two different subgroups of moderate ARDS using a 150 mmHg PaO 2 /FiO 2 threshold and may represent a more homogeneous distribution of ARDS patients across subgroups of severity [25][26][27]. Whether ARDS outcome relates to severity of respiratory failure [28], a higher severity is a risk factor for prolonged mechanical ventilation [19]. Since PaO 2 /FiO 2 does not account for PEEP in its calculation, reported PaO 2 /FiO 2 provides a sense of ARDS severity without knowledge of applied PEEP levels.
The main goal of this study was proposing a novel variable [PaO 2 /(FiO 2 xPEEP)] or P/FP E for PEEP ≥ 5 cmH 2 O that, together with corresponding thresholds, could serve as an improved criterion to assess ARDS severity. The thresholds are 60 to 40 mmHg/cmH 2 O for mild, 40 to 20 for moderate, and less than 20 for severe. This new criterion adequately addressed Berlin's definition gap in computing ARDS severity by including PEEP in the new oxygenation ratio. Increasing the PEEP level with the same FiO 2 yields different PaO 2 and SpO 2 [29]. Thus, including PEEP in calculating the degree of oxygenation severity could be better than current Berlin definition. We examined this hypothesis by applying machine learning (ML) approaches for predicting ARDS severity over time.

Study design and patient population
Two clinical databases were used for evaluation. Data of the first 3 ICU days (considering day 1 for representative data within the first 24 h after ARDS onset, day 2 for data within 24-48 h after onset, and day 3 for data within 48-72 h after onset) (N = 2738, 1519, and 1341 patients, respectively) were extracted from a single-center database MIMIC-III (MetaVision, 2008-2012) [30]. The median length of an ICU stay (LOS) of all selected ARDS patients in MIMIC-III was 11.29 days (Q1-Q3: 7.85-17.54). Similarly, data of the first 3 ICU days after ARDS onset (N = 5153, 2981, and 2326 patients, respectively) were extracted from a multicenter database eICU (2014-2015) [31]. The median length of an ICU LOS of all selected ARDS patients in eICU was 11.72 days (6.92-18.84). All selected patients from both databases fulfilled the Berlin criteria for ARDS and were stratified into mild, moderate or severe ARDS [6] and received mechanical ventilation (MV) for > 48 h [32,33]. Disease progression of ARDS in each database was tracked along those 3 ICU days to assess lung severity. Patients younger than 18 years were excluded.

MIMIC-III
MIMIC-III is a large, publicly available database including de-identified health-related data of approximately 60,000 admissions of ICU patients [30]. The input variables include baseline demographic information (age); hemodynamic parameters including mean, maximum and minimum heart rate (HR); ventilator parameters including mean, maximum and minimum respiratory rate (RR), SpO 2 , and PEEP. These predictors on the third ICU day after assessing lung severity, including their description (mean and 95% CI), are presented in Table 1, and Additional file 1: Tables S1 and S2. The main target variable was ARDS severity (where 0 = mild, 1 = moderate, and 2 = severe). ICU mortality (Fig. 1, Additional file 1: Figs. S1 and S2) and duration of MV were also obtained (Additional file 1: Table S3). eICU eICU is a multicenter and publicly available ICU database with high level of detail in the data about more than 200,000 ICU admissions [31]. Input variables included: baseline demographic information (age); ventilator parameters including PEEP; blood gas parameters including FiO 2 , PaO 2 , and PaCO 2 ( Table 2, Additional file 1:  Tables S4 and S5). The main target variable was ARDS severity (where 0 = mild, 1 = moderate, and 2 = severe). ICU mortality (Fig. 2, Additional file 1: Figs. S3 and S4) and duration of MV were also obtained (Additional file 1: Table S6).

Experimental methods
Before starting our analysis, the thresholds of the P/FP E index (with PEEP ≥ 5) were experimentally tuned. We computed the minimum and maximum P/FP E values of the patients in the two databases, which were 2 and 60 mmHg/cmH 2 O, respectively. Then, several cut-offs were studied in order to determine the ones that could be more accurate in the stratification of ARDS severity. For this purpose, we tested round values (to be easily remembered by intensivists) in the range 2-60 and analyzed P/ FP E index of the ARDS severity groups obtained. The partition showing a better separation of the ARDS severity groups obtained was achieved in this study for the following thresholds (with PEEP ≥ 5): 60-40 for mild, 40-20 for moderate, and < 20 for severe.
Our study is based only on ML analysis and not on the conventional statistical hypothesis testing analysis. In general, ML is an exploratory process and a current application of artificial intelligence to generate predictive models. Using this technology, there is not a  one-model-fits-all solution. Precisely, there is no ML method that reaches the highest accuracy for all domains, datasets, or problem types [38]. The optimal model differs from one problem to another based on the characteristics of variables and observations. Our aim was to implement ML models capable of predicting ARDS severity over time to compare the PaO 2 /FiO 2 ratio-as mandated by the current Berlin criteria for ARDS-with the proposed new P/FP E ratio according to the following three scenarios: (1) Scenario I: predicting ARDS severity in the 3 rd ICU day using information captured in the 1st ICU day; (2) Scenario II: predicting ARDS severity in the 3 rd ICU day using information captured in the 2nd ICU day; (3) Scenario III: predicting ARDS severity in the 3 rd ICU day using information captured in the 1st and 2nd ICU days. We implemented three robust supervised ML algorithms using Python 3.7. The ML algorithms were Light Gradient Boosting Machine (LightGBM) [39], Random Forest (RF) [40], and eXtreme Gradient Boosting (XGBoost) [41]. Grid search was used to identify the optimal values for their input parameters. The quality of the prediction models was computed based on a tenfold cross-validation approach. AUC and CORR (correlation between the predicted and actual values of severity level) were used to assess model performance in predicting ARDS severity as a categorical prediction. To provide a meaning to the findings, we used the classification of performance suggested by Hosmer and Lemeshow [42]: "excellent" if AUC ≥ 0.9; "good" if AUC is between 0.8 and 0.9; "fair" if AUC is between 0.7 and 0.8; "poor" if AUC is between 0.6 and 0.7; and "very poor" if AUC is below 0.6. For CORR, we used the interpretation suggested by Mukaka [43] who proposed "very high" for CORR ≥ 0.9 (positive correlation) or CORR ≤ -0.9 (negative correlation); "high" if CORR is between 0.7 and 0.9 (positive) or -0.9 and -0.7 (negative); "moderate" if CORR is between 0.5 and 0.7 (positive) or -0.7 and -0.5 (negative); "low" if CORR is between 0.3 and 0.5 (positive) or -0.5 and -0.3 (negative), and "negligible" otherwise.

Results
The findings of the three classification ML methods for the three predictive scenarios in the two databases are presented in Tables 3 and 4. Table 3 shows the quality of ML predictions for MIMIC-III, confronting the results obtained for PaO 2 /FiO 2 (Table 3(a)) with those obtained for P/FP E (Table 3(b)). Table 4 shows the same comparative results in patients from the eICU database.
For MIMIC-III, the best ML severity predictive model on the third ICU day was obtained by scenario II and by P/FP E with an AUC = 0.788 and CORR = 0.566, using LightGBM algorithm. When PaO 2 /FiO 2 is used, AUC = 0.635 and CORR = 0.19, but these performances were obtained with different algorithms. In qualitative terms, P/FP E ratio improves PaO 2 /FiO 2 ratio from "poor" to "fair" AUC, and from "negligible" to "moderate" CORR.
For the eICU database, the results were slightly better. The best ML severity predictive model was also observed for scenario II. This finding confirms that the best approach to predict ARDS severity on the third ICU day is to consider the condition of the patient in the second ICU day after ARDS onset, rather than the first ICU day or both. For eICU data, the best AUC and CORR values are 0.873 and 0.745 for P/FP E ; and 0.863 and 0.725 for PaO 2 /FiO 2 . These results are qualified as a "good" predictive accuracy and a "high" correlation.
In general, P/FP E ratio has a better behavior in the prediction of ARDS severity than PaO 2 /FiO 2 ratio in terms of AUC and CORR. Whereas PaO 2 /FiO 2 obtained up to 0.635 AUC and up to 0.19 CORR in MIMIC-III, the use of P/FP E reached 0.788 AUC and 0.566 CORR. This represents increments of + 0.153 AUC and + 0.376 CORR and shows the advantages of using the P/FP E ratio.

Discussion
In this large study, we propose a novel variable or formula (P/FP E ) and corresponding thresholds for classifying ARDS severity. We investigated several ML methods to generate severity predictive models in almost 8,000 patients with ARDS over time after ARDS diagnosis. Our findings confirmed that the best approach to predict ARDS severity on the third ICU day is to consider the condition of the patient in the second ICU day after ARDS onset, rather than during the first ICU day as mandated by Berlin criteria. For the MIMIC-III database, predictive models using the P/FP E ratio attained outstanding improvements in terms of AUC (15% improvement) and CORR (37.6% improvement), when compared to the previous PaO 2 / FiO 2 models. For the eICU database, models based on P/FP E also outperformed PaO 2 /FiO 2 predictions, with 14.8% and 2% improvements of AUC and CORR, respectively. The difference in terms of the accuracy between the two databases is remarkable regarding CORR. This is due to the fact that eICU is a multicenter ICU database with high granularity data (i.e., high level of detail in the data) for over 200,000 admissions to ICUs. By contrast, MIMIC-III is a single-center ICU database for approximately 60,000 admissions of ICU patients. Therefore, in all extracted data of the three ICU days, the number of extracted patients from eICU was greater than the number of extracted patients from MIMIC-III. Consequently, this would lead to better ML results in terms of CORR for the eICU database. Overall, the novel P/FP E ratio outperformed the PaO 2 /FiO 2 ratio in all ML applied models and showed that predictions based on the patient condition in the second day after onset are better than predictions based on the first 24 h (7.2-13.8% AUC and 1.5-22% CORR improvements), followed by the predictions based on both the first and the second day conditions (0.1-0.3% AUC and 0.18-14% CORR improvements).
In contrast to our study, most recent studies developed ML approaches to predict the risk of ARDS in critically ill patients prior to ARDS onset [36,44,45], based on single-center databases [36,45] and using one single ML algorithm [36,44]. Consequently, their findings have serious limitations for the generalizability in the context of assessing the prediction of ARDS outcome.
This large study proposes a novel criterion to reclassify ARDS patients in terms of severity by using ML methods on an extensive amount of data from two large datasets of critically ill patients. The relatively good accuracy of P/FP E (when compared to PaO 2 /FiO 2 ) in stratifying ARDS patients could allow to overcome the major clinical drawbacks of the current Berlin definition. Also, this study is implementing ML models for predicting severity over time after ARDS onset. Critically ill patients are an ideal population for clinical database investigations using machine learning algorithms because while the data from ICUs are extensive, the value of many diagnostic and therapeutic interventions remains largely unproven [46]. ARDS is considered one of the major reasons of ICU admission, and it is associated with a high hospital mortality [1]. Despite its high mortality rate and high rates of ICU utilization, ARDS remains critically misdiagnosed and globally under-diagnosed in the ICU settings [1]. Furthermore, increasing ARDS severity is associated with increased mortality rate [6]. The PaO 2 /FiO 2 ratio categorizes ARDS patients according to the severity of their oxygenation deficit without considering the level of applied PEEP in the assessment of lung severity. The PaO 2 /FiO 2 ratio does not appropriately show the severity of ARDS for PEEP ≥ 5. However, the application of PEEP plays a significant role in improving oxygenation. It is well established that changes in PEEP alter the PaO 2 / FiO 2 in lung-injured patients [29]. Attempting to predict lung severity and patient outcomes based solely in PaO 2 / FiO 2 on this basis is inherent flawed. Thus, the stratification of ARDS patients as proposed by the Berlin criteria is useless for assessing severity of lung injury and could be of no benefit for enrolling patients into therapeutic clinical trials. The P/FP E for PEEP ≥ 5 appropriately addressed Berlin's definition gap in computing ARDS severity by including PEEP in the novel ratio. Clearly, our study showed that P/FP E thresholds improved prediction of ARDS severity. This can lead to important medical implications by accurately anticipate specific treatment for each ARDS category, which could eventually decrease ARDS mortality. In other words, P/FP E can represent a good solution for the clinical assessment of ARDS severity and as a guidance for treatment of ARDS.
Our study has several strengths. First, we have analyzed a large population of ARDS patients within their first three ICU days after onset. Second, we have described and validated our findings using both a large single-center database (MIMIC-III) and a large multicenter database (eICU). Third, we have investigated several ML predictive models for ARDS severity over time after ARDS onset. We believe that our approach is generalizable across other ARDS populations. However, we acknowledge some limitations to our study. First, our work is based on a retrospective analysis of data whose results concerning P/FP E benefits should be confirmed in further prospective studies. Second, our analysis is concerned with the evolution and stratification of patients in their third ICU day after ARDS onset. Although the first 72-h are essential in the management and progression of ARDS patients, our study lacks the assessment of a long-term outcome (e.g., ICU mortality, 60-day mortality). Third, further longitudinal studies on complete evolution of ARDS patients could help to find out new evidence(s) on the management of ARDS since our ML results achieved outstanding improvements compared to the current state, with "fair" to "good" predictions of ARDS severity [42]. Forth, one could argue that extracorporeal membrane oxygenation (ECMO) is not considered in this study. ECMO is a clinical outcome and can only temporarily sustain severe ARDS patients to bridge periods of time when oxygenation through the lungs cannot be achieved via MV. Moreover, ECMO is a constrained resource that is not available in all ICUs. Hence, for the purpose of our study, we only considered patients receiving MV for > 48 h [32,33]. Fifth, regarding the potential consequences of using the new ratio at the bedside, further studies are needed to examine whether it could help for clinical decision making and guiding therapy. Our study opens a possibility to better define ARDS severity, as a new research area for patient care improvement.

Conclusions
This large study proposes a novel criterion based on the P/FP E formula to assess ARDS severity using ML, which is significantly better than the current Berlin criteria using baseline PaO 2 /FiO 2 . Clinically, applying the proposed new criteria for ARDS severity enables clinical care physicians to assess lung severity by involving PEEP information. Moreover, being able to better adjust the severity profiles of ARDS patients will potentially improve the selection of more adequate therapeutic regimens for each ARDS category, which could contribute to reduce ARDS mortality. However, additional studies are required in order to confirm this. In both databases (MIMIC-III and eICU) and either in Berlin or P/FP E , scenario II (assessment of oxygenation deficit after 24 h of ARDS diagnosis and routine ICU treatment) was the best severity predictive scenario. From a ML perspective, P/ FP E outperformed PaO 2 /FiO 2 in all ML models predicting ARDS severity after onset over time in all scenarios either in MIMIC-III or eICU. Accordingly, this study can serve as an example of how ML is a worth-considering technology to gain new insights in the development of ARDS predictive models which could contribute to improve ICU resource allocation and mortality reduction.