Skip to main content
Download PDF

Development of an algorithm to aid triage decisions for intensive care unit admission: a clinical vignette and retrospective cohort study

Critical Care volume 20, Article number: 81 (2016) Cite this article

Abstract

Background

Intensive care unit (ICU) admission triage is performed routinely and is often based solely on clinical judgment, which could mask biases. A computerized algorithm to aid ICU triage decisions was developed to classify patients into the Society of Critical Care Medicine’s prioritization system. In this study, we sought to evaluate the reliability and validity of this algorithm.

Methods

Nine senior physicians evaluated forty clinical vignettes based on real patients. The reference standard was defined as the priorities ascribed by two investigators with full access to patients’ records. Agreement of algorithm-based priorities with the reference standard and with intuitive priorities provided by the physicians were evaluated. Correlations between algorithm prioritization and physicians’ judgment of the appropriateness of ICU admissions in scarcity and nonscarcity settings were also evaluated. Validity was further assessed by retrospectively applying this algorithm to 603 patients with requests for ICU admission for association with clinical outcomes.

Results

Agreement between algorithm-based priorities and the reference standard was substantial, with a median κ of 0.72 (interquartile range [IQR] 0.52–0.77). Algorithm-based priorities demonstrated higher interrater reliability (overall κ 0.61, 95 % confidence interval [CI] 0.57–0.65; median percentage agreement 0.64, IQR 0.59–0.70) than physicians’ intuitive prioritization (overall κ 0.51, 95 % CI 0.47–0.55; median percentage agreement 0.49, IQR 0.44–0.56) (p = 0.001). Algorithm-based priorities were also associated with physicians’ judgment of appropriateness of ICU admission (priorities 1, 2, 3, and 4 vignettes would be admitted to the last ICU bed in 83.7 %, 61.2 %, 45.2 %, and 16.8 % of the scenarios, respectively; p < 0.001) and with actual ICU admission, palliative care consultation, and hospital mortality in the retrospective cohort.

Conclusions

This ICU admission triage algorithm demonstrated good reliability and validity. However, more studies are needed to evaluate a difference in benefit of ICU admission justifying the admission of one priority stratum over the others.

Background

Intensive care unit (ICU) admission triage is performed routinely in ICUs worldwide [1, 2], and patients who are refused ICU admission may have a higher risk of death [2]. This triage process is associated with several factors, including clinical characteristics of the patients [2–4], but it is also influenced by nonclinical factors [2, 4, 5]. Moreover, it has been shown that physicians may not follow triage recommendations [3], and concern has been raised that these clinical judgments could mask prejudice or bias [1]. So, development of objective directives and guidelines applying to individual patients, along with the need for an objective triage score, has been discussed in the literature [6, 7].

The Society of Critical Care Medicine (SCCM) has established guidelines for ICU triage [8], which categorizes patients into four priority strata. Priority 1 patients usually have no therapeutic limits and are critically ill unstable patients in need of intensive treatment and monitoring that cannot be provided outside the ICU. Priority 2 patients usually have no therapeutic limits, require intensive monitoring, and may potentially need immediate intervention. Priority 3 patients are critically ill but have a reduced likelihood of recovery because of underlying disease or the nature of their acute illness; these patients may have therapeutic limitations. Priority 4 patients are generally not appropriate for ICU admission, because either (1) they have little or no anticipated benefit from ICU care based on low risk of active intervention that could not safely be administered in a non-ICU setting or (2) they are patients with terminal and irreversible illness and face imminent death. However, these guidelines have not been formally evaluated.

There is no established gold standard for appropriateness of ICU triage, so, to assess the usefulness of any tool aimed at improving this process, it is important to use more than one method of evaluation and to demonstrate the reliability and validity of the tool [9, 10]. Reliability “refers to the degree to which repeated assessments of the same patient with a triage instrument will deliver the same acuity level,” as such yielding reproducible results [9, 11]. Validity would refer “to the degree with which the measured acuity level reflects the patient’s true acuity at the time of triage” [9], or the capacity of an instrument to reflect what it is proposed to measure. Validity is further delineated into three major types: content (i.e., criteria fit with current knowledge), construct (criteria measure what they are supposed to measure), and criterion (new criteria agree with existing standard) [10, 11].

To help in triage decision making for ICU admission in a setting of ICU bed scarcity, the Hospital das Clinicas da Faculdade de Medicina de Sao Paulo (HCFMUSP) assembled a critical care team to come up with a set of defined criteria for ICU admission. This group proposed an algorithm to aid triage decision making based on the SCCM’s guidelines. The algorithm was designed to help standardize, not replace, clinical judgment in ICU triage. We sought to evaluate the reliability and validity of this algorithm.

Methods

Ethics, consent, and permission

This single-center study was approved, and a waiver for informed consent was granted, by the HCFMUSP Research Ethics Committee (approval number 638.864).

Setting

Sao Paulo is the largest city in South America, with an estimated population of 11 million. HCFMUSP is the largest hospital complex in South America, with about 2200 hospital beds. The Central Institute is the main building of the complex, an academic tertiary hospital with about 1100 hospital beds and 110 ICU beds divided among 10 ICUs. Despite the large number of ICU beds, there is a perceived shortage of ICU beds in the HCFMUSP complex due to the fact that it is a referral center for complex patients from the entire Sao Paulo state.

Algorithm development

A team comprised of critical care specialists, bioethicists, and hospital managers was assembled to come up with an instrument to help triage decisions for ICU admissions. Content (logical) validity was achieved, and the final form of the algorithm was developed through a series of consensus meetings. The algorithm was approved by the hospital’s board and was based on the responses to four closed questions.

Question 1 asked whether the ICU request was for active intervention or monitoring. Intervention was defined as the need for vasoactive drugs, mechanical ventilation (invasive or noninvasive), or urgent hemodialysis in unstable patients. Monitoring was defined as the need for active monitoring with possibility of active intervention (e.g., high-risk surgical patients, acute coronary syndromes, postthrombolysis stroke patients).

Question 2 asked about patients’ comorbidities. Comorbidities were classified into four strata: (1) no comorbidities, (2) compensated comorbidities, (3) decompensated comorbidities (frequent hospital admissions in the last few months, unintended weight loss or loss of functionality), and (4) advanced disease with a probable life expectancy of months (metastatic cancer or locally invasive cancer, (advanced heart failure – i.e., American College of Cardiology/American Heart Association stage “D”) chronic obstructive pulmonary disease with hypoxemia and dyspnea at rest without relief with bronchodilators, National Kidney Foundation Kidney Disease Outcomes Quality Initiative chronic kidney disease stage 5 with contraindications to hemodialysis, Child-Pugh class C cirrhosis with contraindications to liver transplant, dementia with total loss of functionality and/or frailty syndrome and/or immobility syndrome, pressure ulcers, malnutrition, loss of sphincter control). These definitions were adapted from hospice indications published by the National Academy of Palliative Care [12].

Question 3 asked about patient’s previous functionality, as defined by activities of daily living (ADL) according to Katz and colleagues [13]. Patients were classified as functionally independent, partially dependent, or severely dependent (capable of performing a maximum of two ADLs).

Question 4 asked about the requesting physician’s most probable intuitive prognosis. Patients were classified as probable survivors without severe disabilities, probable survivors with severe disabilities, or probable nonsurvivors.

Algorithm description

According to the responses to the four closed questions described above, a Microsoft Access 2013® Visual Basic for Applications (Microsoft, Redmond, WA, USA) algorithm was developed to classify patients into one of the four SCCM priority classes. Patients would be ascribed priority 4 if they had decompensated comorbidities with severe dependency, advanced disease with partial or severe dependency, or advanced disease with preserved functionality but an estimated intuitive prognosis of death. Patients were ascribed priority 3 if they had compensated comorbidities but severe dependency, decompensated comorbidities with partial dependence, or advanced disease with preserved functionality and an estimated prognosis of survival. All other patients were ascribed priority 1 or 2 on the basis of whether the ICU admission request was for active intervention or monitoring. The algorithm’s framework is available in Additional file 1.

Study design

An invited convenience sample of 10 senior physicians with experience in emergency medicine or critical care evaluated 40 clinical vignettes developed with information retrieved from representative real patients for whom urgent ICU admission was requested at the HCFMUSP in January 2014. Case vignettes rather than actual charts were chosen to ensure that only information that was available at the moment of ICU request would be presented to the responding physicians.

These vignettes contained real information available at the moment of the patient’s request for ICU admission, such as age, sex, length of hospitalization, comorbidities, previous functional status, acute diagnosis, presence of organ dysfunction, need for advanced life support, and objective reason for ICU admission request. Sample vignettes are available in Additional file 1.

The physicians were asked to ascribe an intuitive prioritization of 1–4 following SCCM guidelines (physician-based priority). In addition, on the basis of information provided with each vignette, the physicians would answer the four closed questions of the algorithm, and then, according to the responses, the algorithm would ascribe a priority (algorithm-based priority). Physicians were blinded to this algorithm classification because it was performed offline.

Moreover, physicians were asked for their clinical judgment of the appropriateness of ICU admission for each clinical vignette. Appropriateness of ICU admission was evaluated in a non-ICU bed scarcity setting using a Likert scale with four levels (1 = completely agree with ICU admission, 2 = agree, 3 = disagree, and 4 = completely disagree). This Likert scale was further dichotomized into two strata (1 or 2 = appropriate admission and 3 or 4 = inappropriate admission). Appropriateness of ICU admission was also evaluated in an ICU bed scarcity setting by asking if the physician would admit the patient in each clinical vignette to the last ICU bed (yes-or-no question).

Reference standard

These 40 clinical vignettes were also classified into the SCCM priority categories by subjective consensus between the two investigators (DNF and JGRR) who had access to the patients’ complete medical records, including outcomes (reference standard). These investigators were blinded to the physician-based and algorithm-based priorities. According to this reference standard, 7 cases reflected priority 1, 13 reflected priority 2, 10 reflected priority 3, and 10 reflected priority 4.

Construct validity analysis

Construct validity is “the degree to which a test measures what it claims, or purport, to be measuring” [9, 11]. In this study, construct validity was determined by assessing how the algorithm would correlate to experts’ prioritization. It was evaluated by comparing priorities ascribed by the algorithm, based on physician’s responses (algorithm-based priority), to priorities ascribed by the two investigators with full access to patients’ medical records (reference standard), and to intuitive priorities ascribed by the physicians who evaluated the vignettes (physician-based priority). It was further assessed by evaluating the correlation of algorithm-based priorities with the clinical judgment of appropriateness of ICU admission in ICU scarcity or nonscarcity settings.

Reliability analysis

Interrater reliability was evaluated by overall agreement among pairwise algorithm-based priorities and among pairwise physician-based priorities. Moreover, to assess the impact of each individual component of the algorithm, interrater agreement among physicians’ responses to each of the four individual questions was evaluated.

Criterion validity analysis

Criterion validity is the extent to which a measure is related to an outcome [9, 11]. Because there is no gold standard for ICU triage, criterion validity was assessed by evaluating the predictive validity of the algorithm (i.e., by comparison with later outcomes believed to be associated with ICU triage) [9, 11].

Criterion validity was evaluated by applying the algorithm to a sample of patients with urgent ICU admission requests between September and December 2013 to assess the association of algorithm-based priorities with clinical outcomes. This sample was retrospectively retrieved from a database with administrative information and responses to the algorithm.

Importantly, the four questions of the algorithm were responded to prospectively at the moment of ICU request by the attending physician, but with the sole intent of evaluating the feasibility of the process. The responses to the questions and the priority ascribed by the algorithm were maintained offline and were not available to the triaging physicians, so they had no impact on actual ICU triage decisions. Patients younger than 16 years of age, repeated ICU requests, and requests for elective surgeries were excluded from the analysis.

We evaluated the association of algorithm-based priorities with hospital mortality, palliative care consultation, and ICU admission. The association of each of the four questions from the ICU request form individually with hospital mortality and ICU admission was also evaluated.

Statistical analysis

Microsoft Excel 2013® (Microsoft) and Microsoft Access 2013® (Microsoft) were used as database software. Statistical analyses were performed with SPSS 13.0™ (SPSS Inc., Chicago, IL, USA) or Epi Info 7™ for Windows (Centers for Disease Control and Prevention, Atlanta, GA, USA). κ-statistic quadratic weighted analyses were performed using the VassarStats website [14].

Estimating a 50 % refusal rate with final values n = 5, α = 0.05, β = 0.20, ρ0 = 0.4, and ρ1 = 0.6, the sample size of observations was estimated as 35 [15]. The classification model designed by Landis and Koch [16] was used for the interpretation of κ. For the clinical outcomes analysis, based on previous findings [17] and preliminary analysis of our data (data not shown), we estimated that, to obtain a power of 80 % and a two-sided confidence level of 95 %, a sample size of 341 patients would be necessary to find an estimated odds ratio for mortality of 2.0, comparing lower priorities with higher priorities, estimating a hospital mortality rate of 30 % in the higher priorities group.

Reliability was evaluated by quadratic weighted κ, proportions (percentage) of agreement, and intraclass correlation coefficient. Categorical variables were described as numbers of cases and proportions. Continuous variables were described as mean ± standard deviation or median (interquartile range [IQR]). Differences in proportions were evaluated with χ2 statistics or Fisher’s exact test as appropriate. Differences in means and medians were evaluated with analysis of variance or the Mann-Whitney U test. A two-tailed p value less than 0.05 was considered statistically significant.

Results

Clinical vignette data

Of 10 physicians invited, 9 (90 %) returned the questionnaire, comprising 36 individual pairs of agreement. All physicians evaluated 40 clinical vignettes, completing 360 triage scenarios. There was incomplete data for 3 triage scenarios, so 357 triage scenarios were evaluated.

Reference standard

Agreement between algorithm-based prioritization and the reference standard was substantial, with a median κ of 0.72 (IQR 0.52–0.77), which was not statistically different from median κ between physician-based priorities and the reference standard, which was 0.62 (IQR 0.57–0.70) (p = 0.258). Agreement between each physician-based priorities and algorithm-based priorities resulted in a median κ of 0.63 (IQR 0.55–0.74).

Correlation with appropriateness of ICU admission

Physicians’ judgment of the appropriateness of ICU admission was correlated with algorithm-based priorities in both non-ICU bed scarcity and ICU bed scarcity settings (Fig. 1). The proportions of triage scenarios that were judged as appropriate for ICU admission in a non-ICU bed scarcity setting were 100 % (86 cases) for priority 1, 94 % (63 cases) for priority 2, 95.2 % (40 cases) for priority 3, and 67.1 % (104 cases) for priority 4 (p < 0.001). The proportions of triage scenarios that would be admitted to the last ICU bed were 83.7 % (72 cases) for priority 1, 61.2 % (41 cases) for priority 2, 45.2 % (19 cases) for priority 3, and 16.8 % (26 cases) for priority 4 (p < 0.001).

Fig. 1
figure 1

Proportion of scenarios in which patients would be admitted to the intensive care unit (ICU), according to algorithm-based priorities, stratified by ICU scarcity or nonscarcity setting. *p < 0.001; † p < 0.001

Full size image

Interrater reliability

In the interrater reliability analysis, overall agreement among algorithm-based priorities was 0.61 (95 % confidence interval [CI] 0.57–0.65), while overall agreement among physician-based priorities was 0.51 (95 % CI 0.47–0.55). Moreover, the median proportions (percentage) of agreement were 0.64 (IQR 0.59–0.70) for algorithm-based priorities and 0.49 (IQR 0.44–0.56) for physician-based priorities (p < 0.001). The average intraclass correlation coefficient for algorithm-based priorities was 0.94 (95 % CI 0.91–0.97), and for physician-based priorities it was 0.90 (95 % CI 0.85–0.94), both indicating good consistency in the prioritization process.

Agreement among individual components of the triage instrument is demonstrated in Additional file 1: Table S1. Individual questions demonstrated substantial interrater agreement (median κ values of 0.77, 0.74, and 0.84 for questions 1, 2, and 3, respectively), with the exception of question 4, with a median κ of 0.52.

Correlation with clinical outcomes

There were 731 requests for ICU admission between September 2013 and December 2013. After excluding repeated requests, patients younger than 16 years of age, and requests for elective surgery, 603 patients were analyzed.

Hospital mortality was 37.8 % (224 patients), and 229 patients (38.0 %) were refused admission to the ICU. The algorithm-based priority distribution was as follows: 265 patients (45.5 %) were ascribed priority 1; 155 patients (26.6 %) were ascribed priority 2; 102 patients (17.5 %) were ascribed priority 3; and 61 patients (10.5 %) were ascribed priority 4.

Hospital mortality was associated with priority level (Fig. 2). Priority 4 patients had a hospital mortality rate of 66.1 % (59 patients), while priorities 1, 2, and 3 patients had mortality rates of 35.5 % (92 patients), 27.6 % (42 patients), and 46.1 % (47 patients), respectively (p < 0.001 by χ2 for trend).

Fig. 2
figure 2

Association of priority categories with hospital mortality, intensive care unit (ICU) admission and palliative care consultation. Priorities 1, 2, 3, and 4 were statistically associated with hospital mortality (p < 0.001 by χ2 for trend), ICU admission (p = 0.035 by χ2 for trend), and palliative care consultation (p = 0.036 by χ2 for trend)

Full size image

The ICU admission rates were 65.2 % (172 patients) for priority 1 patients, 63.2 % (98 patients) for priority 2 patients, 56.9 % (58 patients) for priority 3 patients, and 52.5 % (32 patients) for priority 4 patients (p = 0.035 by χ2 for trend). Palliative care consultation was requested for 51 patients (8.5 %) in the cohort, with 21 (7.9 %) patients in priority 1, 7 (4.5 %) patients in priority 2, 12 (11.8 %) patients in priority 3, and 10 (16.4 %) patients in priority 4, as shown in Fig. 2 (p = 0.036 by χ2 for trend). Individual components of the tool were also evaluated and were associated with hospital mortality and ICU admission, as demonstrated in Fig. 3a and b and Additional file 1: Table S2.

Fig. 3
figure 3

Association of individual questions with hospital mortality (a) and intensive care unit (ICU) admission (b). See supplementary material in Additional file 1 for number of patients in each group. *p < 0.001; † p < 0.05. Question 1: 1 = active intervention, 2 = monitoring. Question 2: 1 = no comorbidities, 2 = compensated comorbidities, 3 = decompensated comorbidities, 4 = advanced disease. Question 3: 1 = functionally independent, 2 = partially dependent, 3 = severely dependent. Question 4: 1 = probable survivor without severe disabilities, 2 = probable survivor with severe disabilities, 3 = probable nonsurvivor

Full size image

Discussion

This ICU triage algorithm has good reliability and validity, achieving construct validity by comparison of the results of the algorithm-based priorities with the reference standard, and by comparison with physicians’ judgment of appropriateness of ICU admission, in both nonscarcity and scarcity settings. Interrater reliability was substantial and was higher for the algorithm-based priorities than for physician-based priorities. The criterion (predictive) validity of this algorithm was further supported by correlation with surrogate clinical outcomes, such as ICU admission, palliative care consultation, and mortality. Moreover, when analyzing the individual components of the algorithm, it was found that each of these components contributed to the overall correlation.

There is growing consensus that clear and objective directives for ICU admission triage could benefit patient and resource allocation decision making [1, 6, 7]. Nevertheless, triage and risk stratification tools have not yet been routinely implemented in the ICU, other than in catastrophic settings [18, 19]. In part, this situation can be explained by the current absence of a validated triage tool for ICU admission, because the SCCM guidelines for ICU triage have never been formally evaluated, other than in small studies in which researchers examined specific priority strata [17, 20]. Moreover, it has been argued that existing physiological scores alone should not be used to predict outcomes [6]. Therefore, an alternative could be to use standardized subjective scores that combine objective data with the physician’s intuitive judgment of prognosis [21–23]. Such an instrument could be used to standardize and reduce biases that could adversely influence proper evaluation and decision making for ICU admission of individual patients.

This study has shown that this algorithm-based ICU triage decision-making tool has good interrater reliability, with a κ of 0.61, outperforming the interrater reliability of physicians’ prioritization. Because there are no ICU triage scales to use as benchmarks, this algorithm’s performance was compared with emergency triage scale performance reported in the literature. The authors of a systematic review of studies evaluating the Manchester Triage Score (MTS) [24] found a wide interrater agreement range. The ranges for unweighted and weighted κ statistics in these studies were 0.31–0.76 and 0.40–0.82, respectively. Of note, most studies were of low or moderate quality. The authors of another systematic review [25] found good reliability for the Canadian Triage and Acuity Scale and the Emergency Severity Index, with κ statistics ranging from 0.7 to 0.95, but found the MTS and the Australasian Triage Scale to be less reliable (κ statistics 0.3–0.6). Farrohknia et al. [26], in analyzing the current evidence on emergency triage scales, found no high-quality study available and also found a wide interrater agreement range, with κ values varying from 0.202 to 0.87.

Although comparable to rates for other published triage tools and higher than senior physicians’ agreement rates, these agreement rates are less than perfect. These agreement rates may be due to the inherent subjectivity associated with ICU triage [1], because disagreement on appropriateness of care may occur in up to 66 % of cases labeled as receiving inappropriate care [27]. To minimize this issue, we chose well-defined closed-end questions that have been shown to correlate with outcomes and to have high interrater reliability. The exception was question 4 (estimated prognosis), which was expected to be more intuitive and less reliable. This problem was mitigated by the fact that, in our algorithm, this specific question has less influence on the final priority. Question 4 would change only the ascribed priority from 3 to 4 in the case of a patient with advanced disease or severe functional dependence that would have been ascribed an intuitive estimate of no survival.

There are also controversies in the literature regarding the interpretation of κ values, and some suggest that the usual interpretation of 0.61–0.80 as substantial agreement [16] may be too lenient [28]. However, authors of many recent papers evaluating triage [24, 26, 29–32] and other scales such as the Glasgow Coma Scale [33] and the American Society of Anesthesiologists physical status classification [34] have used the more traditional interpretation of κ values and have shown similar agreement rates [24, 25, 34], which may help contextualize our results. It is possible that this more stringent approach was meant to be directed for more objective data, such as laboratory values [28]. Moreover, even when the more stringent approach was used, this tool’s performance was at least moderate and was probably better than the physicians’ intuitive priorities. In addition, as suggested by some authors [28], we have also reported other measures of reliability, such as percentage agreement and intraclass correlation coefficient, to help with interpretation of the results.

Validity, or the capacity of the tool to reflect the true triage priority, was evaluated. Because there is no established gold standard for prioritization or appropriateness of ICU, and in accordance with recommendations from the literature [9–11], the validity of this ICU admission triage algorithm was assessed by using a series of methods. In the context of the clinical vignettes, algorithm-based priorities demonstrated substantial agreement with the selected reference standard (two investigators with access to patients’ records), with a κ value of 0.72. In addition, the algorithm was shown to correlate with physicians’ judgment of appropriateness of ICU admission, especially in an ICU scarcity setting, demonstrating a linear relationship between prioritization and the proportion of scenarios that would be admitted to the last ICU bed. These results support the construct validity of the algorithm.

For further validation, this triage algorithm was compared with actual clinical outcomes in a subset of patients with requests for urgent ICU admission. The algorithm correlated with hospital mortality, palliative care consultation, and ICU admission, which supports its criterion (predictive) validity. However, although statistically significant, the difference in ICU admission rates between higher and lower priorities could be expected to be larger. Importantly, this observed difference is a reflection of current practice because the algorithm had no influence on ICU admission rates. Moreover, there is a perceived scarcity of ICU beds in our hospital, which is corroborated by the 30 % ICU refusal rate, so it is possible that this observed difference in admission rates may be due to a high rate of “inappropriate” admissions. Of note, despite the good correlation with outcomes in the retrospective cohort, the algorithm could not be properly evaluated regarding the association with benefit of ICU admission, defined as the difference in outcome with and without critical care [35].

This study has several strengths. It is one of the few to evaluate a triage tool for ICU admission and is, to our knowledge, the first to evaluate a triage algorithm for ICU admission in a resource-limited, noncatastrophic setting. Despite its single-center nature, in this study we evaluated a population of patients in a large academic tertiary hospital with 10 independent ICUs with different patient admission policies. Acknowledging the difficulty of establishing gold standards to ICU admission, we evaluated different surrogates covering different aspects of the process.

The study has several limitations, however. In the validation cohort, a short period of time was evaluated, and, due to the retrospective algorithm-based priority input, it was not possible to adjust for potential confounders and to ascertain all possible biases. Another important limitation is that in this study we evaluated the correlation of priority levels with outcomes, but the study was not designed to predict the impact of ICU admission on mortality in each priority stratum.

In addition, the clinical vignette methodology used, despite being based on real patients, may be less robust than real encounters with patients. Vignettes offer significant advantages, such as quantification of physician performance, ease of use, and low cost [29], and, as such, researchers in several studies have used case vignettes to study clinicians’ attitudes in both triage [29, 36–38] and nontriage settings [39–41]. Nevertheless, the vignette methodology differs from real patients in important ways because it summarizes and standardizes important clinical information and excludes the effect of external factors that could impact the decision-making process, such as time pressure and cognitive load [29]. It could be argued, for instance, that these vignettes’ characteristics could falsely increase the agreement rate in a subjective setting such as ICU triage. However, this is not consensual in the literature, because studies in which researchers have used case vignettes to evaluate decisions to withdraw support in critically ill patients [42] and patients with sepsis [43] and triage for ICU admission of typical ICU cases [44] have demonstrated wide disagreement in individual responses. Because the algorithm improved interrater agreement in the vignette methodology, it is possible to hypothesize that the algorithm would systematize the triage process and that this improvement could be even greater in the unstandardized setting of real patient encounters. Nonetheless, in this study we did not evaluate physicians’ performance with real patients, so we cannot be certain about the impact of the chosen methodology regarding the agreement rate.

Therefore, notwithstanding the good characteristics of this tool, this ICU admission triage algorithm may not be suitable for clinical practice yet, because, despite its correlations with outcomes, it was not properly evaluated for prediction of a differential benefit in each ICU priority stratum that would justify ICU admission of one stratum over the others. Further prospective studies with additional collected variables should be implemented to address the algorithm’s limitations.

Conclusions

An algorithm-based prioritization system for ICU admission derived from the SCCM priority categories demonstrated good reliability and validity, with substantial interrater reliability and correlation with physician’s assessments and with the established reference standard. Algorithm-based priorities were also associated with clinical outcomes such as hospital mortality, palliative care consultation, and ICU admission, with lower priorities correlating with worse outcomes. However, at this stage, this algorithm could not be properly evaluated to predict the differential benefit of critical care for each priority stratum and may still not be ready for implementation. Further prospective studies that address these limitations are warranted.

Key messages

  • Algorithm-based priorities for ICU admission have good interrater reliability and correlate well with the judgment of appropriateness of ICU admission by a panel of experts.

  • Algorithm-based priorities for ICU admission correlated with actual ICU admission, palliative care consultation, and hospital mortality in a retrospective cohort of patients for whom urgent ICU admission was requested.

Abbreviations

ADL:

activities of daily living

CI:

confidence interval

HCFMUSP:

Hospital das Clinicas da Faculdade de Medicina de Sao Paulo

ICU:

intensive care unit

IQR:

interquartile range

MTS:

Manchester Triage Score

SCCM:

Society of Critical Care Medicine

References

  1. Truog RD, Brock DW, Cook DJ, Danis M, Luce JM, Rubenfeld GD, et al. Rationing in the intensive care unit. Crit Care Med. 2006;34:958–63. quiz 971.

    Article  PubMed  Google Scholar 

  2. Sinuff T, Kahnamoui K, Cook DJ, Luce JM, Levy MM. Rationing critical care beds: a systematic review. Crit Care Med. 2004;32:1588–97.

    Article  PubMed  Google Scholar 

  3. Azoulay E, Pochard F, Chevret S, Vinsonneau C, Garrouste M, Cohen Y, et al. Compliance with triage to intensive care recommendations. Crit Care Med. 2001;29:2132–6.

    Article  CAS  PubMed  Google Scholar 

  4. Garrouste-Orgeas M, Montuclard L, Timsit JF, Reignier J, Desmettre T, Karoubi P, et al. Predictors of intensive care unit refusal in French intensive care units: a multiple-center study. Crit Care Med. 2005;33:750–5.

    Article  PubMed  Google Scholar 

  5. Escher M, Perneger TV, Chevrolet JC. National questionnaire survey on what influences doctors’ decisions about admission to intensive care. BMJ. 2004;329:425. doi:10.1136/bmj.329.7463.425.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Christian MD, Fowler R, Muller MP, Gomersall C, Sprung CL, Hupert N, et al. Critical care resource allocation: trying to PREEDICCT outcomes without a crystal ball. Crit Care. 2013;17:107.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Sprung CL, Danis M, Iapichino G, Artigas A, Kesecioglu J, Moreno R, et al. Triage of intensive care patients: identifying agreement and controversy. Intensive Care Med. 2013;39:1916–24. doi:10.1007/s00134-013-3033-6.

    Article  PubMed  Google Scholar 

  8. Task Force of the American College of Critical Care Medicine, Society of Critical Care Medicine. Guidelines for intensive care unit admission, discharge, and triage. Crit Care Med. 1999;27(3):633–8.

    Article  Google Scholar 

  9. Twomey M, Wallis LA, Myers JE. Limitations in validating emergency department triage scales. Emerg Med J. 2007;24:477–9. doi:10.1136/emj.2007.046383.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Streiner DL, Norman GR. “Precision” and “accuracy”: two terms that are neither. J Clin Epidemiol. 2006;59:327–30. doi:10.1016/j.jclinepi.2005.09.005.

    Article  PubMed  Google Scholar 

  11. Angus DC, Seymour CW, Coopersmith CM, Deutschman CS, Klompas M, Levy MM, et al. A framework for the development and interpretation of different sepsis definitions and clinical criteria. Crit Care Med. 2016;44:e113–21. doi:10.1097/CCM.0000000000001730.

    Article  PubMed  Google Scholar 

  12. Tavares De Carvalho R, Parsons HA. Manual de cuidados paliativos ANCP: ampliado e atualizado. 2nd ed. Sao Paulo, Brazi: Academia Nacional de Cuidados Paliativos (ANCP); 2012.

    Google Scholar 

  13. Katz S, Ford AB, Moskowitz RW, Jackson BA, Jaffe MW. Studies of illness in the aged. The index of ADL: a standardized measure of biological and psychosocial function. JAMA. 1963;185:914–9.

    Article  CAS  PubMed  Google Scholar 

  14. Lowry R. VassarStats: website for statistical computation. http://vassarstats.net/. Accessed 14 March 2016.

  15. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17:101–10.

    Article  CAS  PubMed  Google Scholar 

  16. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  CAS  PubMed  Google Scholar 

  17. Caldeira VM, Silva Júnior JM, Oliveira AM, Rezende S, Araújo LA, Santana MR, et al. Criteria for patient admission to an intensive care unit and related mortality rates. Rev Assoc Med Bras. 2010;56:528–34.

    Article  PubMed  Google Scholar 

  18. Christian MD, Hawryluck L, Wax RS, Cook T, Lazar NM, Herridge MS, et al. Development of a triage protocol for critical care during an influenza pandemic. CMAJ. 2006;175:1377–81. doi:10.1503/cmaj.060911.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Cheung WK, Myburgh J, Seppelt IM, Parr MJ, Blackwell N, Demonte S, et al. A multicentre evaluation of two intensive care unit triage protocols for use in an influenza pandemic. Med J Aust. 2012;197:178–81.

    Article  PubMed  Google Scholar 

  20. Shum HP, Chan KC, Lau CW, Leung AK, Chan KW, Yan WW. Triage decisions and outcomes for patients with Triage Priority 3 on the Society of Critical Care Medicine scale. Crit Care Resusc. 2010;12:42–9.

    PubMed  Google Scholar 

  21. Knaus WA, Harrell Jr FE, Lynn J, Goldman L, Phillips RS, Connors Jr AF, et al. The SUPPORT prognostic model: objective estimates of survival for seriously ill hospitalized adults. Ann Intern Med. 1995;122:191–203.

    Article  CAS  PubMed  Google Scholar 

  22. Maghsoudi B, Tabei SH, Zand F, Tabatabaee H, Akbarzadeh A. A model for decision making for intensive care unit admission in source limited hospitals. Iran Red Crescent Med J. 2014;16:e15497. doi:10.5812/ircmj.15497.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Sinuff T, Adhikari NK, Cook DJ, Schunemann HJ, Griffith LE, Rocker G, et al. Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med. 2006;34:878–85. doi:10.1097/01.CCM.0000201881.58644.41.

    Article  PubMed  Google Scholar 

  24. Parenti N, Reggiani ML, Iannone P, Percudani D, Dowding D. A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int J Nurs Stud. 2014;51:1062–9. doi:10.1016/j.ijnurstu.2014.01.013.

    Article  PubMed  Google Scholar 

  25. Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Arztebl Int. 2010;107:892–8.

    PubMed  PubMed Central  Google Scholar 

  26. Farrohknia N, Castren M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resusc Emerg Med. 2011;19:42. doi:10.1186/1757-7241-19-42.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Piers RD, Azoulay E, Ricou B, Dekeyser Ganz F, Decruyenaere J, Max A, et al. Perceptions of appropriateness of care among European and Israeli intensive care unit nurses and physicians. JAMA. 2011;306:2694–703. doi:10.1001/jama.2011.1888.

    Article  CAS  PubMed  Google Scholar 

  28. McHugh ML. Interrater reliability: the κ statistic. Biochem Med (Zagreb). 2012;22:276–82.

    Article  Google Scholar 

  29. Mohan D, Fischhoff B, Farris C, Switzer GE, Rosengart MR, Yealy DM, et al. Validating a vignette-based instrument to study physician decision making in trauma triage. Med Decis Making. 2014;34:242–52. doi:10.1177/0272989X13508007.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Graff I, Goldschmidt B, Glien P, Bogdanow M, Fimmers R, Hoeft A, et al. The German version of the Manchester Triage System and its quality criteria – first assessment of validity and reliability. PLoS One. 2014;9:e88995. doi:10.1371/journal.pone.0088995.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Mirhaghi A, Heydari A, Mazlom R, Ebrahimi M. The reliability of the Canadian Triage and Acuity Scale: meta-analysis. N Am J Med Sci. 2015;7:299–305. doi:10.4103/1947-2714.161243.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Ebrahimi M, Heydari A, Mazlom R, Mirhaghi A. The reliability of the Australasian Triage Scale: a meta-analysis. World J Emerg Med. 2015;6:94–9. doi:10.5847/wjem.j.1920-8642.2015.02.002.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Reith FC, Van den Brande R, Synnot A, Gruen R, Maas AI. The reliability of the Glasgow Coma Scale: a systematic review. Intensive Care Med. 2016;42:3–15. doi:10.1007/s00134-015-4124-3.

    Article  PubMed  Google Scholar 

  34. Riley R, Holman C, Fletcher D. Inter-rater reliability of the ASA physical status classification in a sample of anaesthetists in Western Australia. Anaesth Intensive Care. 2014;42:614–8.

    PubMed  Google Scholar 

  35. Gomersall CD, Joynt GM. What is the benefit in triage? Crit Care Med. 2011;39:911–2.

    Article  PubMed  Google Scholar 

  36. Mohan D, Angus DC, Ricketts D, Farris C, Fischhoff B, Rosengart MR, et al. Assessing the validity of using serious game technology to analyze physician decision making. PLoS One. 2014;9:e105445. doi:10.1371/journal.pone.0105445.

    Article  PubMed  PubMed Central  Google Scholar 

  37. van der Wulp I, van Baar ME, Schrijvers AJ. Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J. 2008;25:431–4. doi:10.1136/emj.2007.055228.

    Article  PubMed  Google Scholar 

  38. Grouse AI, Bishop RO, Bannon AM. The Manchester Triage System provides good reliability in an Australian emergency department. Emerg Med J. 2009;26:484–6. doi:10.1136/emj.2008.065508.

    Article  CAS  PubMed  Google Scholar 

  39. Redelmeier DA, Shafir E. Medical decision making in situations that offer multiple alternatives. JAMA. 1995;273:302–5.

    Article  CAS  PubMed  Google Scholar 

  40. Green MJ, Farber NJ, Ubel PA, Mauger DT, Aboff BM, Sosman JM, et al. Lying to each other: when internal medicine residents use deception with their colleagues. Arch Intern Med. 2000;160:2317–23.

    Article  CAS  PubMed  Google Scholar 

  41. Aberegg SK, Haponik EF, Terry PB. Omission bias and decision making in pulmonary and critical care medicine. Chest. 2005;128:1497–505. doi:10.1378/chest.128.3.1497.

    Article  PubMed  Google Scholar 

  42. Cook DJ. Determinants in Canadian health care workers of the decision to withdraw life support from the critically ill. JAMA. 1995;273:703–8. doi:10.1001/jama.273.9.703.

    Article  CAS  PubMed  Google Scholar 

  43. O’Brien Jr JM, Aberegg SK, Ali NA, Diette GB, Lemeshow S. Results from the National Sepsis Practice Survey: predictions about mortality and morbidity and recommendations for limitation of care orders. Crit Care. 2009;13:R96. doi:10.1186/cc7926.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Dahine J, Mardini L, Jayaraman D. The perceived likelihood of outcome of critical care patients and its impact on triage decisions: a case-based survey of intensivists and internists in a Canadian, quaternary care hospital network. PLoS One. 2016;11:e0149196. doi:10.1371/journal.pone.0149196.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

We acknowledge Leandro Taniguchi, Marcelo Park, Otavio Ranzani, Pedro Mendes, and Sabrina Correa for their insights and for collaboration in the designing and implementation of the study.

Author information

Authors and Affiliations

  1. Medical sciences doctoral program, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Joao Gabriel Rosa Ramos

  2. Intensive Care Unit, Hospital Sao Rafael, Salvador, Brazil

    Joao Gabriel Rosa Ramos

  3. UNIME Medical School, Lauro de Freitas, Brazil

    Joao Gabriel Rosa Ramos

  4. Emergency Department, Hospital das Clinicas, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Beatriz Perondi & Roger Daglius Dias

  5. Intensive Care Unit, Hospital Nove de Julho, Sao Paulo, Brazil

    Leandro Costa Miranda

  6. Bioethics Committee, Hospital das Clinicas, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Claudio Cohen

  7. Discipline of Bioethics, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Claudio Cohen

  8. Pulmonary Division, Heart Institute (InCor), Hospital das Clinicas, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Carlos Roberto Ribeiro Carvalho

  9. Intensive Care Unit, Emergency Medicine Discipline, Hospital das Clinicas, University of Sao Paulo Medical School, Sao Paulo, Brazil

    Irineu Tadeu Velasco & Daniel Neves Forte

  10. Palliative Care Team, Hospital Sirio-Libanes, Sao Paulo, Brazil

    Daniel Neves Forte

Authors
  1. Joao Gabriel Rosa Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beatriz Perondi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roger Daglius Dias
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leandro Costa Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Claudio Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carlos Roberto Ribeiro Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Irineu Tadeu Velasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Daniel Neves Forte
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joao Gabriel Rosa Ramos.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

JGRR, BP, and DNF contributed to the designing, acquisition, analysis and interpretation of data, drafting and revising the manuscript. RDD, LCM, CC, CRRC, and ITV contributed to the conception of the study and critical revision of the manuscript. All authors read and approved the final manuscript.

Additional file

Additional file 1:

Supplementary material. (DOCX 68 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ramos, J.G.R., Perondi, B., Dias, R.D. et al. Development of an algorithm to aid triage decisions for intensive care unit admission: a clinical vignette and retrospective cohort study. Crit Care 20, 81 (2016). https://doi.org/10.1186/s13054-016-1262-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13054-016-1262-0

Keywords

Critical Care

ISSN: 1364-8535