Skip to main content
Download PDF

Performance of the ROX index in predicting high flow nasal cannula failure in COVID-19 patients: a systematic review and meta-analysis

Critical Care volume 27, Article number: 320 (2023) Cite this article

Abstract

COVID-19 patients with acute hypoxemic respiratory failure (AHRF) benefit from high flow nasal cannula (HFNC) oxygen therapy. However, delays in initiating invasive ventilation after HFNC failure are associated with poorer outcomes. The respiratory oxygenation (ROX) index, combining SpO2/FiO2 and respiratory rate, can predict HFNC failure. This meta-analysis evaluated the optimal ROX index cut-offs in predicting HFNC failure among COVID-19 patients at different measurement timings and clinical settings. Three databases were searched for eligible papers. From each study, we reconstructed the confusion matrices at different cut-offs, fitted linear mixed models to estimate the ROX index distribution function, and derived the area under the summary receiver operator characteristic curve (sAUC) and optimal cut-offs to predict HFNC failure. 24 studies containing 4790 patients were included. Overall sAUC was 0.771 (95% CI: 0.666–0.847) (optimal cut-off: 5.23, sensitivity: 0.732, specificity: 0.690). The cut-off values to achieve 80%, 90% sensitivity, 80%, 90% specificity were 5.70, 6.69, 4.45, 3.37, respectively. We stratified the analysis by ROX measurement time and estimated optimal cut-offs and cut-offs to achieve 80% sensitivity and specificity. For 2–6 h and 6–12 h post-HFNC initiation, we propose the use of 80% specific cut-offs to rule in HFNC failure of < 5.33 and < 3.69, respectively. For 12–24 h post-HFNC initiation, we propose the use of the 80% sensitive cut-off of > 6.07 to rule out HFNC failure. Our analysis confirms the overall utility of the ROX index in risk stratification of COVID-19 patients with AHRF receiving HFNC and provides potentially useful cut-offs for different times from HFNC initiation.

Introduction

The COVID-19 pandemic is an unprecedented global health crisis, with more than 651 million infections and more than 6.6 million deaths [1]. A serious complication and common cause of death in patients with COVID-19 infection is acute hypoxemic respiratory failure (AHRF), which occurs in 15–30% of COVID-19 patients [2]. This can be managed by high flow nasal cannula (HFNC) oxygen therapy [3], a non-invasive method of oxygen supplementation, using a large-bore nasal cannula to administer up to 100% FiO2 at a high flow of 60 L/min [4, 5]. This intervention creates positive airway pressure [6] and lessens the anatomical dead space and the work of breathing [7]. The use of warmed (31–37 °C) [8], humidified oxygen protects the mucosal lining, allows oxygenation with lower transpulmonary driving pressure, and facilitates secretion clearance [9]. HFNC can be used to spare patients from invasive mechanical ventilation [10].

However, HFNC use can delay endotracheal intubation, and such delays are associated with longer invasive mechanical ventilation and a poorer prognosis [11]. In particular, COVID-19 patients have a high failure rate of non-invasive treatment (i.e., worsening in severity so as to require intubation and invasive ventilation) [12]. Hence, prognostic tools to predict HFNC failure have high clinical relevance, to discriminate in a timely manner the patients who are poor candidates for HFNC continuation, from those who can be safely spared from invasive mechanical ventilation. Current risk stratification tools for HFNC failure include respiratory parameters—the oxygen saturation to fraction of inspired oxygen ratio (SpO2/FiO2), respiratory rate (RR), partial pressure of carbon dioxide (PaCO2), and partial pressure of oxygen (PaO2) [13, 14]. However, these parameters in isolation are unable to identify the need for intubation reliably [15].

The respiratory oxygenation (ROX) index is a prognostic index that has gained popularity during the COVID-19 pandemic, and was specifically developed to prognosticate HFNC failure in patients with pneumonia and AHRF [16,17,18]. It combines SpO2/FiO2 and respiratory rate (RR) using the formula \(\left( {\frac{{{{{\text{SpO}}_{2} } \mathord{\left/ {\vphantom {{{\text{SpO}}_{2} } {{\text{FiO}}_{2} }}} \right. \kern-0pt} {{\text{FiO}}_{2} }}}}{{{\text{RR}}}}} \right)\) [15], and can be easily done at the bedside [19]. The original description of the ROX index by Roca et al. measured the ROX at 2, 6, and 12 h after HFNC initiation. The proposed cut-offs were ROX > 4.88 to predict HFNC success at 2, 6, and 12 h, and ROX < 3.85 after 12 h to predict HFNC failure. A ROX of 3.85–4.88 was described as an indeterminate range, and the authors suggested that such patients should have the ROX reassessed at a later time point.

Multiple studies have validated the ROX index in recent years [20, 21]. However, as there are no universally accepted protocols for ROX use, studies on the ROX index have adopted different cut-offs and monitoring intervals. There is thus uncertainty [17, 21] over optimal cut-off values in general, and the specific cut-offs when ROX is measured at different times from HFNC initiation. We note a previous meta-analysis by Prakash et al. [19], which included all studies on ROX as a predictor for HFNC failure up till early 2021. Prakash et al. provided a simple dichotomisation of the ROX index into high versus low categories, with no specific analysis of time from ROX initiation. There are no present meta-analyses to determine the optimal ROX index cut-offs [8, 19, 22], at initiation of HFNC and at subsequent time points.

Hence, in this systematic review and meta-analysis, we aimed to describe the performance of the ROX index in predicting HFNC failure amongst COVID-19 patients at different time points from HFNC initiation. We further aimed to derive optimal cut-off values at various timings to guide the interpretation of ROX index in various clinical settings.

Methods

Search strategy

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [23] informed the design and execution of this study. The protocol was registered on PROSPERO (CRD42023388254). Medline/PubMed, Embase and Cochrane Central Register of Controlled Trials (CENTRAL) were searched from inception to 20 December 2022 for eligible studies with keywords related to Covid-19 and ROX index. The search strategy was developed in collaboration with a medical librarian (Medical Library, National University of Singapore). The search strategy can be found in the Additional file 1. No language filters were applied. Two authors independently carried out the preliminary eligibility screening in a blinded fashion. The authors screened the titles and abstracts before retrieving and reviewing the full texts. Studies were included if they (1) included Covid-19 patients on HFNC and (2) utilised the ROX index. Reviews, commentaries, animal studies and case reports were excluded. A senior author resolved differences by discussion and consensus.

Data extraction and selection criteria

From each study, two authors used a standardised data extraction sheet to extract information on the study period, country, population demographics independently. ROX cut-off values for each study, and the definition of HFNC failure or success were extracted. Since some studies used HFNC success as the outcome, while others used HFNC failure, we standardised HFNC failure as the outcome of interest. HFNC failure included the outcomes of intubation, escalation to mechanical ventilation (including non-invasive and invasive ventilation), and death.

Confusion matrices (a form of the 2 × 2 contingency table) were constructed for each study detailing the number of true positives (patients who scored below the ROX cut-off and experienced HFNC failure), true negatives (patients who scored above the ROX cut-off and experienced HFNC success), false positives (patients who scored below ROX cut-off and experienced HFNC success), and false negatives (patients who scored above the ROX cut-off and experienced HFNC failure).

Statistical analysis

All analysis was done using RStudio (version 2021.9.1.372). Statistical analysis was conducted with meta (version 5.2–0) and diagmeta (version 0.5–0) packages. Using diagmeta which implements the approach outlined by Steinhauser et al. [24] and also previously successfully applied to other acute clinical research questions [25], various linear mixed models were fitted to estimate the distribution function of the ROX index within the included studies. For the linear mixed models that converged, we applied the restricted maximum likelihood criterion and the model that minimised this criterion was selected. Area under the summary receiver operator characteristic curve (sAUC) was derived for different individual time points and time windows, and the optimum cut-off values were calculated. The optimum cut-off value was the cut-off which maximised the weighted sum of sensitivity and specificity.

Subgroup analysis

Analysis of the diagnostic accuracy of the ROX index was conducted in prespecified subgroups: average age of included patients, admission year of patients, corticosteroid usage. Sensitivity analysis was conducted by excluding conference abstracts and letters and including only journal articles. Further sensitivity analysis was conducted for the type of outcome—excluding studies that had progression to NIV or death as an outcome. Pairwise comparison of sAUC values between different subgroups was made in accordance with the methods outlined by Hanley et al. [26, 27].

Risk of bias assessment

Two independent and blinded authors assessed studies for methodological quality, using the Quality Assessment of Prognostic Accuracy Studies (QUAPAS) tool for prognostic studies [28]. The QUAPAS tool assesses the quality of studies across five key domains: participant selection, index test, outcome, flow and timing, and analysis. Disagreements were resolved through discussion with a third author.

Patient and public involvement

No patients or members of the public were directly involved in this research study.

Results

Article search and included studies

The search strategy identified 242 relevant studies (Fig. 1) after removal of duplicates. Fifty studies were included in the full-text review. Our final analysis included 24 [16, 21, 29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50] studies, comprising 4790 patients (Table 1). Of the 24 studies, nine were multicentre studies and 15 were single-centre studies.

Fig. 1
figure 1

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow chart

Full size image
Table 1 Characteristics of included studies
Full size table

Studies came from all major global regions, including one multicontinental study involving two institutions from Europe and South America. Studies from single regions included one from Africa, five from Asia, eleven from Europe, three from North America, and two from South America. One did not report the location of the study.

Six were prospective studies, 15 were retrospective studies, and two were retrospective analyses of prospectively collected data. One study did not report on the study type. Seven studies were published in 2020, eight in 2021, and nine in 2022.

The mean age of included patients ranged from 51.3 to 70. ROX cut-offs investigated ranged from 1.96 to 8.36. The definitions of HFNC success or failure used by each study are shown in Table 1.

Most of the studies were evaluated to be moderate or high in bias (Additional file 1: Fig. S1).

Overall performance of the ROX index

Results from the 24 studies were pooled. ROX had an overall sAUC of 0.771 (95% CI: 0.666–0.847) (Fig. 2), and the overall optimal cut-off value of ROX was 5.23. At this cut-off, sensitivity was 0.732 (95% CI: 0.578–0.846) and specificity was 0.690 (95% CI: 0.539–0.809). The cut-off to achieve 80% sensitivity was 5.70, while the cut-off to achieve 80% specificity was 4.45. The cut-off to achieve 90% sensitivity was 6.69, while the cut-off to achieve 90% specificity was 3.37.

Fig. 2
figure 2

Overall summary receiver operator characteristic curve

Full size image

A sensitivity analysis was conducted by including only original journal articles (i.e. excluding letters and abstracts). In these 19 studies with 4535 patients, sAUC was 0.770 (95% CI: 0.656–0.849), and the optimal cut-off value was 5.27. At this cut-off, sensitivity was 0.741 (95% CI: 0.563–0.864) and specificity was 0.681 (0.515–0.811). No significant difference in sAUC was found (two-tailed p-value: 0.921). Another sensitivity analysis was conducted by including original journal articles and letters (i.e. excluding abstracts). In these 21 studies with 4625 patients, sAUC was 0.764 (95% CI: 0.648–0.845), and the optimal cut-off value was 5.10. At this cut-off, sensitivity was 0.722 (95% CI: 0.552–0.846) and specificity was 0.688 (0.526–0.814). No significant difference in sAUC was found (two-tailed p-value: 0.488).

As a further sensitivity analysis, we analysed the overall optimal cut-offs with the exclusion of Hamou et al. [38] (with the lowest cut-off), and Kim et al. [33] (with the highest cut-off), removed from the analysis separately. Removing Hamou et al., sAUC was 0.766 (95% CI: 0.661–0.841); removing Kim et al., sAUC was 0.779 (95% CI: 0.688–0.848); removing both, sAUC was 0.774 (95% CI: 0.684–0.842). There was no significant difference from the overall sAUC in these three sensitivity analyses (two-tailed p-value 0.620, 0.423 and 0.767, respectively).

Diagnostic accuracy of the ROX index measured within specific time windows

In 5 studies which used ROX measured ≥ 2 h but < 6 h of initiation of HFNC, sAUC was 0.754 (95% CI: 0.604–0.863) (Fig. 3), and the optimal cut-off was 5.71. At this cut-off, sensitivity and specificity were 0.635 (95% CI: 0.411–0.813) and 0.769 (95% CI: 0.465–0.927), respectively (Table 2).

Fig. 3
figure 3

Area under summary receiver operator characteristic curve at various time windows with 95% confidence intervals

Full size image
Table 2 Diagnostic accuracy at various time windows
Full size table

In 8 studies which used ROX measured ≥ 6 h but < 12 h of initiation of HFNC, sAUC was 0.795 (95% CI: 0.287–0.908) (Fig. 3), and the optimal cut-off was 6.50. At this cut-off, sensitivity and specificity were 0.835 (95% CI: 0.389–0.976) and 0.659 (95% CI: 0.192–0.940), respectively (Table 2).

In 7 studies which used ROX measured ≥ 12 h but < 24 h of initiation of HFNC, sAUC was 0.821 (95% CI: 0.713–0.894) (Fig. 3), and the optimal cut-off was 5.78. At this cut-off, sensitivity and specificity were 0.749 (95% CI: 0.575–0.868) and 0.759 (95% CI: 0.511–0.905), respectively (Table 2).

No analysis of the ROX was done for < 2 h from ROX initiation as there were only 5 studies with 5 cut-offs, insufficient to fit the linear mixed models for the ROX index distribution in this time window.

Diagnostic accuracy of the ROX index in clinically important subgroups

Elderly patients

Twenty one studies included data on the average age of patients. No significant difference in sAUC value was found between studies where the mean age was less than 65 and those where the mean age was more than or equal to 65 (two-tailed p-value: 0.067).

Use of corticosteroids

In five studies where more than 75% of patients received corticosteroid treatment, sAUC of ROX was 0.711 (95% CI: 0.293–0.883). In three studies where the proportion of patients who received corticosteroid treatment was less than or equal to 75%, sAUC of ROX was 0.721 (95% CI: 0.005–0.997). sAUC values were not significantly different between the two groups (two-tailed p-value: 0.696).

Admission year of patients

Fifteen studies included patients admitted in 2020 only, while four studies included patients admitted in 2021 only. For this sub-analysis, we did not include four studies which had included patients across both 2020–2021, and one study that did not state the year of admission. sAUC value in the group admitted in 2021 was insignificantly different from that in the group admitted in 2020 (two-tailed p-value: 0.404).

The sAUC values and p-values for between subgroup differences are detailed in Table 3.

Table 3 Area under the summary receiver operator characteristic curve (sAUC) in clinically important subgroups
Full size table

Other subgroups

Three studies of the ROX index were performed in unvaccinated COVID-19 patients. The cut-offs in these few studies widely ranged from 1.96 to 5.63, and this can affect the quality of sAUC estimate. In particular, Hamou et al. used an unusually low cut-off of 1.96. Most studies did not report on the vaccination status of their included patients. There were no studies which made direct comparison of the ROX index between vaccinated and non-vaccinated patients. We decided not to perform further analysis of the ROX index based on vaccination status.

Diagnostic accuracy of the ROX index for different outcomes

The definition of HFNC failure was heterogenous across different studies, with some studies including mortality as part of a composite outcome, and some including progression to NIV as an endpoint. We present the diagnostic accuracy of the ROX index stratified by the definition of HFNC failure in Table 4.

Table 4 Diagnostic accuracy stratified by definition of high flow nasal cannula failure
Full size table

Discussion

HFNC is an important ventilation sparing therapy in COVID-19 patients. However, delayed intubation in HFNC has been shown to lead to increased mortality [11]. Use of the ROX index has allowed for objective assessment of HFNC failure risk, to allow early stratification of patients who can be safely continued on HFNC, from those who are likely to need invasive ventilation. Previous meta-analyses [8, 19] have dichotomised patients to an overall high or low ROX index, to handle different ROX cut-off values used in different studies. On the other hand, our up-to-date systematic review and meta-analysis of the ROX index in COVID-19 patients has included information on all cut-off values and time points described in the constituent studies. We have modelled the sROC of the ROX index in general, as well as at different time points. This has allowed us to estimate the optimal cut-off at each time point, as well as the clinically relevant 80% sensitivity and specificity cut-offs.

Overall, we found that the ROX index had a sAUC of 0.771. The optimal cut-off of the ROX index in general that maximised sensitivity and specificity for HFNC failure was 5.23 (sensitivity: 0.732, specificity: 0.690). This is higher than the cut-off proposed by Roca et al. [18] (who originally developed the ROX index), where a universal cut-off of ROX > 4.88 at 2, 6, and 12 h post-HFNC initiation was suggested to predict HFNC success. Our cut-off derived from analysis of the sAUC agrees more with subsequent studies such as Prakash et al. and Zhou et al. [19, 20], which have suggested an optimal ROX index cut-off of around 5. We suggest that if a general cut-off for the ROX index is sought, without regard for time from initiation, that 5.23 be used as it maximises both sensitivity and specificity. Alternatively, cut-off values of 5.70 and 4.45 are 80% and 80% sensitive and specific, respectively, while cut-off values of 6.69 and 3.37 are 90% sensitive and 90% specific, respectively, in determining HFNC failure without regard for time from initiation. These cut-offs could be used if a provider prefers higher sensitivity or specificity.

For most patients on HFNC therapy, the time from initiation is known, and the ROX can be calculated at different times from HFNC initiation. We provide a further meta-analysis of the performance of the ROX index at different time windows. Providers should be aware of the differences in ROX index performance at different time windows and note the ROX index performs better at time windows from 6 h onwards. We observed that the sAUC improved from 0.754 at 2–6 h, to 0.795 at 6–12 h, to 0.821 at 12–24 h. This echoes the findings of Roca et al. [18], where ROX measured at earlier time points had poorer sensitivity and specificity of < 70%. One possible explanation [20] for the better ROX index performance in later time windows is that patients may be relatively undifferentiated initially, but the additional time afforded them by HFNC therapy allows the underlying pace of COVID-19 disease to become evident, and allows for COVID-19 therapeutics (such as steroids or antiviral drugs) to exert their clinical effect. There were relatively few studies investigating the ROX index in the 0–2 h time window, and we were unable to perform an sAUC analysis. We are unable to comment on the overall diagnostic performance of the ROX index during this window, or to recommend a cut-off.

We suggest that at times closer to HFNC initiation, a cut-off that is highly specific for HFNC failure be used to identify patients at high risk of needing intubation. For these patients, early intubation should be considered. For 2–6 h post-HFNC initiation, we propose the cut-off of < 5.33 (80% specific); for 6–12 h post-HFNC initiation, we propose the cut-off of < 3.69 (80% specific).

On the other hand, at later time points, a cut-off that is highly sensitive for HFNC failure could be used to identify patients with higher likelihood of HFNC success. Such patients might benefit from continued treatment with HFNC. They might even be eligible for non-invasive monitoring with the ROX index given their high likelihood of HFNC success, as opposed to invasive monitoring with arterial blood gas sampling. For 12–24 h post-HFNC initiation, we propose the cut-off of > 6.07 (80% sensitive).

We have also conducted subgroup analyses that examined the ROX index in studies stratified by patient age, use of corticosteroids, and the study period. There were no significant differences in the diagnostic power of the ROX index when comparing among these subgroups, suggesting broad applicability of the ROX index regardless of older age, corticosteroid use, and across different years of the COVID-19 pandemic.

Strengths and limitations

This is the first meta-analysis to estimate optimal cut-off values for the ROX index and evaluate its diagnostic accuracy at different time points and time windows, and to provide high sensitivity and high specificity cut-off values at each time point. This can allow clinicians to make informed choices about the ROX cut-off to use while the patient is undergoing HFNC therapy.

There are some limitations of this analysis. As with any systematic review, we are dependent on the availability of data from the constituent studies. First, publicly available data were only available in aggregate form, instead of individual patient data (IPD), which would be most ideal for estimating diagnostic test performance. This necessitated estimation of the sAUC from the aggregates, rather than directly calculating the AUC. We were thus also unable to analyse the distribution of the ROX index for patients in general, to determine if changes in ROX index within the same patient had any diagnostic value, or to examine the performance of the ROX index in detail for other subgroups (for example, vaccinated versus unvaccinated individuals, immunocompromised individuals, or individuals who received antiviral or immunomodulatory drugs). This lack of IPD necessitated the subgroup analysis to be performed based on study-level characteristics such as mean age and proportion of patients receiving corticosteroids. The lack of IPD, coupled with the sparsity of large studies with cut-off values that achieved 90% sensitivity/90% specificity, is a limitation to reliable calculation of estimated cut-off values for 90% sensitivity/90% specificity in subgroups such as time windows and nature of outcome. We also appreciate that sAUC estimated confidence intervals are relatively large, although this may be improved in the future as more data on HFNC use becomes available.

We are also aware that the absence of standardised reporting protocols for HFNC use can lead to sources of bias that are difficult to account for based on the extant literature. For example, few studies report at which point from onset of COVID-19 illness that the patient presented (which is potentially important as patient presenting very late or very early could have a different disease course [51,52,53]). Other sources of heterogeneity include the definition of HFNC failure, where some studies reported a composite outcome including mortality, on top of the need for intubation or mechanical ventilation. Another source of heterogeneity is the lack of reporting of intubation criteria. The variability in determining when intubation should be initiated would affect the performance of the ROX index. There was an overall lack of standardisation of the time horizon of outcomes in general. As more studies on HFNC and ROX are done, it may be worthwhile for consensus definitions of HFNC success and failure to be agreed on, and minimum reporting criteria for HFNC studies.

We are aware that modified versions of the ROX index exist, such as a modified ROX index [17] incorporating heart rate that was validated in a cohort of 145 patients. There is an overall paucity of data on modified ROX models, and the performance of these alternative indices may be better addressed in future meta-analyses as more data become available.

Conclusion

The ROX index has a good diagnostic accuracy for HFNC failure in COVID-19 patients and performs the best at 6–12 h or later, post-initiation of HFNC. We suggest an optimal cut-off of 5.23 in general, but propose that healthcare providers also contextualise interpretation of the ROX index depending on the time from HFNC initiation. A higher specificity cut-off may be preferred closer to initiation to rule in HFNC failure, whereas a higher sensitivity cut-off could be used further from initiation to rule out HFNC failure and justify continuation of HFNC treatment.

Availability of data and materials

Data used for analysis can be made available upon reasonable request.

References

  1. Karan A, Wadhera RK. Healthcare system stress due to Covid-19: evading an evolving crisis. J Hosp Med. 2021;16(2):127.

    PubMed  PubMed Central  Google Scholar 

  2. Attaway AH, Scheraga RG, Bhimraj A, Biehl M, HatipoÄŸlu U. Severe covid-19 pneumonia: pathogenesis and clinical management. BMJ. 2021;372:n436.

    PubMed  Google Scholar 

  3. Shoukri AM. High flow nasal cannula oxygen and non-invasive mechanical ventilation in management of COVID-19 patients with acute respiratory failure: a retrospective observational study. 2021;15(1).

  4. Leeies M, Flynn E, Turgeon AF, Paunovic B, Loewen H, Rabbani R, et al. High-flow oxygen via nasal cannulae in patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis. Syst Rev. 2017;6(1):202.

    PubMed  PubMed Central  Google Scholar 

  5. Chikata Y, Morinishi K, Nishimura M. Humidification in very-high-flow nasal-cannula therapy in an adult lung Model. Respir Care. 2019;64(7):809–17.

    PubMed  Google Scholar 

  6. Frat JP, Thille AW, Mercat A, Girault C, Ragot S, Perbet S, et al. High-flow oxygen through nasal cannula in acute hypoxemic respiratory failure. N Engl J Med. 2015;372(23):2185–96.

    CAS  PubMed  Google Scholar 

  7. Papazian L, Corley A, Hess D, Fraser JF, Frat J-P, Guitton C, et al. Use of high-flow nasal cannula oxygenation in ICU adults: a narrative review. Intensive Care Med. 2016;42:1336–49.

    CAS  PubMed  Google Scholar 

  8. Junhai Z, Jing Y, Beibei C, Li L. The value of ROX index in predicting the outcome of high flow nasal cannula: a systematic review and meta-analysis. Respir Res. 2022;23(1):33.

    PubMed  PubMed Central  Google Scholar 

  9. Gürün A. High flow nasal cannula in COVID-19: a literature review. Tuberk Toraks. 2020;68:168–74.

    Google Scholar 

  10. Yang X, Yu Y, Xu J, Shu H, Liu H, Wu Y, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. Lancet Respir Med. 2020;8(5):475–81.

    CAS  PubMed  PubMed Central  Google Scholar 

  11. Kang BJ, Koh Y, Lim C-M, Huh JW, Baek S, Han M, et al. Failure of high-flow nasal cannula therapy may delay intubation and increase mortality. Intensive Care Med. 2015;41(4):623–32.

    PubMed  Google Scholar 

  12. Raoof S, Nava S, Carpati C, Hill NS. High-flow, noninvasive ventilation and awake (nonintubation) proning in patients with coronavirus disease 2019 with respiratory failure. Chest. 2020;158(5):1992–2002.

    CAS  PubMed  Google Scholar 

  13. Kim WY, Sung H, Hong SB, Lim CM, Koh Y, Huh JW. Predictors of high flow nasal cannula failure in immunocompromised patients with acute respiratory failure due to non-HIV pneumocystis pneumonia. J Thorac Dis. 2017;9(9):3013–22.

    PubMed  PubMed Central  Google Scholar 

  14. Grieco DL, Maggiore SM, Roca O, Spinelli E, Patel BK, Thille AW, et al. Non-invasive ventilatory support and high-flow nasal oxygen as first-line treatment of acute hypoxemic respiratory failure and ARDS. Intensive Care Med. 2021;47(8):851–66.

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Roca O, Messika J, Caralt B, García-de-Acilu M, Sztrymf B, Ricard JD, et al. Predicting success of high-flow nasal cannula in pneumonia patients with hypoxemic respiratory failure: the utility of the ROX index. J Crit Care. 2016;35:200–5.

    PubMed  Google Scholar 

  16. Blez D, Soulier A, Bonnet F, Gayat E, Garnier M. Monitoring of high-flow nasal cannula for SARS-CoV-2 severe pneumonia: less is more, better look at respiratory rate. Intensive Care Med. 2020;46(11):2094–5.

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Goh KJ, Chai HZ, Ong TH, Sewa DW, Phua GC, Tan QL. Early prediction of high flow nasal cannula therapy outcomes using a modified ROX index incorporating heart rate. J Intensive Care. 2020;8(1):41.

    PubMed  PubMed Central  Google Scholar 

  18. Roca O, Caralt B, Messika J, Samper M, Sztrymf B, Hernández G, et al. An index combining respiratory rate and oxygenation to predict outcome of nasal high-flow therapy. Am J Respir Crit Care Med. 2019;199(11):1368–76.

    PubMed  Google Scholar 

  19. Prakash J, Bhattacharya PK, Yadav AK, Kumar A, Tudu LC, Prasad K. ROX index as a good predictor of high flow nasal cannula failure in COVID-19 patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis. J Crit Care. 2021;66:102–8.

    CAS  PubMed  PubMed Central  Google Scholar 

  20. Zhou X, Liu J, Pan J, Xu Z, Xu J. The ROX index as a predictor of high-flow nasal cannula outcome in pneumonia patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis. BMC Pulm Med. 2022;22(1):121.

    CAS  PubMed  PubMed Central  Google Scholar 

  21. Hu M, Zhou Q, Zheng R, Li X, Ling J, Chen Y, et al. Application of high-flow nasal cannula in hypoxemic patients with COVID-19: a retrospective cohort study. BMC Pulm Med. 2020;20(1):324.

    CAS  PubMed  PubMed Central  Google Scholar 

  22. Claudi D, Arifin H, Prasetya RJ. The role of ROX index in predicting intubation in COVID-19 pneumonia receiving high flow nasal cannula: a systematic review. J Emerg Med Trauma Acute Care. 2022;2022(2).

  23. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

    PubMed  PubMed Central  Google Scholar 

  24. Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies. BMC Med Res Methodol. 2016;16(1):97.

    PubMed  PubMed Central  Google Scholar 

  25. Yau CE, Chen H, Lim BP-Y, Ng M, Ponampalam R, Lim DYZ, et al. Performance of the paracetamol-aminotransferase multiplication product in risk stratification after paracetamol (acetaminophen) poisoning: a systematic review and meta-analysis. Clin Toxicol. 2022;61:1–11.

    Google Scholar 

  26. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36.

    CAS  PubMed  Google Scholar 

  27. Lowry R. VassarStats: statistical computation web site.

  28. Lee J, Mulder F, Leeflang M, Wolff R, Whiting P, Bossuyt PM. QUAPAS: an adaptation of the QUADAS-2 tool to assess prognostic accuracy studies. Ann Intern Med. 2022;175(7):1010–8.

    PubMed  Google Scholar 

  29. Bruna M, Hidalgo G, Castañeda S, Galvez M, Bravo D, Benitez R, et al. Diaphragmatic ultrasound predictors of high-flow nasal cannula therapeutic failure in critically III patients with SARS-CoV-2 pneumonia. J Ultrasound Med. 2022.

  30. Talpos MT, Rasson A, De Terwangne C, Simonet O, Taccone FS, Vallot F. Early prediction of high-flow oxygen therapy failure in COVID-19 acute hypoxemic respiratory failure: a retrospective study of scores and thresholds. Cureus. 2022;14(11):e32087.

    PubMed  PubMed Central  Google Scholar 

  31. Duan J, Zeng J, Deng P, Ni Z, Lu R, Xia W, et al. High-flow nasal cannula for COVID-19 patients: a multicenter retrospective study in China. Front Mol Biosci. 2021;8:639100.

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Chandel A, Patolia S, Brown AW, Collins AC, Sahjwani D, Khangoora V, et al. High-flow nasal cannula therapy in COVID-19: using the ROX index to predict success. Respir Care. 2021;66(6):909–19.

    PubMed  Google Scholar 

  33. Kim JH, Baek A-R, Lee S-I, Kim W-Y, Na YS, Lee BY, et al. ROX index and SpO2/FiO2 ratio for predicting high-flow nasal cannula failure in hypoxemic COVID-19 patients: a multicenter retrospective study. PLoS ONE. 2022;17(5):e0268431.

    CAS  PubMed  PubMed Central  Google Scholar 

  34. Ferrer S, Sancho J, Bocigas I, Bures E, Mora H, Monclou E, et al. ROX index as predictor of high flow nasal cannula therapy success in acute respiratory failure due to SARS-CoV-2. Respir Med. 2021;189:106638.

    PubMed  PubMed Central  Google Scholar 

  35. Calligaro GL, Lalla U, Audley G, Gina P, Miller MG, Mendelson M, et al. The utility of high-flow nasal oxygen for severe COVID-19 pneumonia in a resource-constrained setting: a multi-centre prospective observational study. EClinicalMedicine. 2020;28:100570.

    PubMed  PubMed Central  Google Scholar 

  36. Myers LC, Mark D, Ley B, Guarnieri M, Hofmeister M, Paulson S, et al. Validation of respiratory rate-oxygenation index in patients with COVID-19-related respiratory failure. Crit Care Med. 2022;50(7):E638–42.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Metwaly A, Magdy D. High-velocity nasal insufflation therapy in Patients with COVID-19 pneumonia. Eur Respir J. 2021;58(SUPPL 65).

  38. Ait Hamou Z, Levy N, Charpentier J, Mira JP, Jamme M, Jozwiak M. Use of high-flow nasal cannula oxygen and risk factors for high-flow nasal cannula oxygen failure in critically-ill patients with COVID-19. Respir Res. 2022;23(1):329.

    CAS  PubMed  PubMed Central  Google Scholar 

  39. Vega ML, Dongilli R, Olaizola G, Colaianni N, Sayat MC, Pisani L, et al. COVID-19 Pneumonia and ROX index: time to set a new threshold for patients admitted outside the ICU. Pulmonology. 2022;28(1):13–7.

    PubMed  Google Scholar 

  40. Takeshita Y, Terada J, Hirasawa Y, Kinoshita T, Tajima H, Koshikawa K, et al. High-flow nasal cannula oxygen therapy in hypoxic patients with COVID-19 pneumonia: a retrospective cohort study confirming the utility of respiratory rate index. Respir Investig. 2022;60(1):146–53.

    CAS  PubMed  Google Scholar 

  41. Burnim MS, Wang K, Checkley W, Nolley EP, Xu Y, Garibaldi BT. The effectiveness of high-flow nasal cannula in coronavirus disease 2019 pneumonia: a retrospective cohort study. Crit Care Med. 2022;50(3):E253–62.

    CAS  PubMed  PubMed Central  Google Scholar 

  42. Zucman N, Mullaert J, Roux D, Roca O, Ricard J-D, Longrois D, et al. Prediction of outcome of nasal high flow use during COVID-19-related acute hypoxemic respiratory failure. Intensive Care Med. 2020;46(10):1924–6.

    CAS  PubMed  PubMed Central  Google Scholar 

  43. Valencia CF, Lucero OD, Castro OC, Sanko AA, Olejua PA. Comparison of ROX and HACOR scales to predict high-flow nasal cannula failure in patients with SARS-CoV-2 pneumonia. Sci Rep. 2021;11(1):22559.

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Kljakovic Gaspic T, Pavicic Ivelja M, Kumric M, Matetic A, Delic N, Vrkic I, et al. In-hospital mortality of COVID-19 patients treated with high-flow nasal oxygen: evaluation of biomarkers and development of the novel risk score model CROW-65. Life. 2021;11(8):735.

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Küçük M, Ergan B, Yakar MN, Ergün B, Akdoğan Y, Cantürk A, et al. The predictive values of respiratory rate oxygenation index and chest computed tomography severity score for high-flow nasal oxygen failure in critically Ill patients with coronavirus disease-2019. Balkan Med J. 2022;39(2):140–7.

    PubMed  PubMed Central  Google Scholar 

  46. Panadero C, Abad-Fernández A, Rio-Ramirez MT, Acosta Gutierrez CM, Calderon-Alcala M, Lopez-Riolobos C, et al. High-flow nasal cannula for acute respiratory distress syndrome (ARDS) due to COVID-19. Multidiscip Respir Med. 2020;15(1):693.

    PubMed  PubMed Central  Google Scholar 

  47. De Alba Aparicio M, Carcel Fernandez S, López Martin C, Porras Pantojo M, De La Fuente Martos C, Simón JMS. Predicting the response to high flow nasal cannula as first-line respiratory support in the patients with pneumonia covid-19 and acute respiratory failure. Intensive Care Med Exp. 2020;8(SUPPL 2).

  48. Tzouanatou EM, Kyriazopoulou E, Kranidiotis G, Velentza L, Stamati A, Mavroudis P, et al. Predictors of effectiveness of HFNC oxygenation in severe SARS-COV2 pneumonia in non-ICU hospital wards. Intensive Care Med Exp. 2022;10.

  49. Celejewska-Wójcik N, Polok K, Górka K, Stachura T, Kania A, Nastałek P, et al. High-flow nasal oxygen therapy in the treatment of acute respiratory failure in severe COVID-19 pneumonia: a prospective observational study. Pol Arch Intern Med. 2021;131(7–8):658–65.

    PubMed  Google Scholar 

  50. Xu J, Yang X, Huang C, Zou X, Zhou T, Pan S, et al. A novel risk-stratification models of the high-flow nasal cannula therapy in COVID-19 patients with hypoxemic respiratory failure. Front Med. 2020;7.

  51. Pellaud C, Grandmaison G, Thien HPPH, Baumberger M, Carrel G, Ksouri H, et al. Characteristics, comorbidities, 30-day outcome and in-hospital mortality of patients hospitalised with COVID-19 in a Swiss area—a retrospective cohort study. Swiss Med Wkly. 2020;150:w20314.

    PubMed  Google Scholar 

  52. Faes C, Abrams S, Van Beckhoven D, Meyfroidt G, Vlieghe E, Hens N, et al. Time between symptom onset, hospitalisation and recovery or death: statistical analysis of Belgian COVID-19 patients. Int J Environ Res Public Health. 2020;17(20):7560.

    CAS  PubMed  PubMed Central  Google Scholar 

  53. Goyal D, Inada-Kim M, Mansab F, Iqbal A, McKinstry B, Naasan AP, et al. Improving the early identification of COVID-19 pneumonia: a narrative review. BMJ Open Respir Res. 2021;8(1):e000911.

    PubMed  Google Scholar 

Download references

Acknowledgements

We would like to sincerely thank Ms. Kim Kee Toh for her valuable input in the crafting of the search strategy.

Funding

We do not have any sources of funds to disclose.

Author information

Author notes

    Authors and Affiliations

    1. Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

      Chun En Yau & Adithi Vasudevan

    2. School of Medicine, Dentistry and Nursing, University of Glasgow, Glasgow, UK

      Dawn Yi Xin Lee

    3. Department of Respiratory and Critical Care Medicine, Singapore General Hospital, Singapore, Singapore

      Ken Junyang Goh

    4. Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore

      Evelyn Wong & Andrew Fu Wah Ho

    5. Pre-Hospital and Emergency Research Centre, Duke-National University of Singapore Medical School, Singapore, Singapore

      Andrew Fu Wah Ho

    6. Centre for Population Health Research and Implementation, SingHealth Regional Health System, Singapore, Singapore

      Andrew Fu Wah Ho

    7. Saw Swee Hock School of Public Health, National University of Singapore, Singapore, Singapore

      Andrew Fu Wah Ho

    8. Department of Gastroenterology and Hepatology, Singapore General Hospital, Singapore, Singapore

      Daniel Yan Zheng Lim

    Authors
    1. Chun En Yau
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Dawn Yi Xin Lee
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Adithi Vasudevan
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Ken Junyang Goh
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Evelyn Wong
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Andrew Fu Wah Ho
      View author publications

      You can also search for this author in PubMed Google Scholar

    7. Daniel Yan Zheng Lim
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    CEY, DYXL, AV, KJG, EW, AFWH and DYZL contributed to (1) the conception and design of this project; (2) acquisition of data; (3) drafting and revising the manuscript. CEY, AFWH and DYZL contributed to the analysis and interpretation of data. All authors gave their final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

    Corresponding authors

    Correspondence to Andrew Fu Wah Ho or Daniel Yan Zheng Lim.

    Ethics declarations

    Ethics approval and consent to participate

    No patients or members of the public were directly involved in this research study.

    Consent for publication

    Not applicable.

    Competing interests

    On behalf of all authors, the corresponding author states that there is no conflict of interest.

    Additional information

    Publisher's Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Andrew Fu Wah Ho and Daniel Yan Zheng Lim are co-last authors

    Supplementary Information

    Additional file 1

    . Search strategy. Fig. S1: Risk-of-bias assessment using the quality assessment of prognostic accuracy studies (QUAPAS) tool.

    Rights and permissions

    Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Yau, C.E., Lee, D.Y.X., Vasudevan, A. et al. Performance of the ROX index in predicting high flow nasal cannula failure in COVID-19 patients: a systematic review and meta-analysis. Crit Care 27, 320 (2023). https://doi.org/10.1186/s13054-023-04567-7

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1186/s13054-023-04567-7

    Keywords

    Critical Care

    ISSN: 1364-8535