- Open Access
A retrospective cohort pilot study to evaluate a triage tool for use in a pandemic
Critical Care volume 13, Article number: R170 (2009)
The objective of this pilot study was to assess the usability of the draft Ontario triage protocol, to estimate its potential impact on patient outcomes, and ability to increase resource availability based on a retrospective cohort of critically ill patients cared for during a non-pandemic period.
Triage officers applied the protocol prospectively to 2 retrospective cohorts of patients admitted to 2 academic medical/surgical ICUs during an 8 week period of peak occupancy. Each patient was assigned a treatment priority (red -- 'highest', yellow -- 'intermediate', green -- 'discharge to ward', or blue/black -- 'expectant') by the triage officers at 3 separate time points (at the time of admission to the ICU, 48, and 120 hours post admission).
Overall, triage officers were either confident or very confident in 68.4% of their scores; arbitration was required in 54.9% of cases. Application of the triage protocol would potentially decrease the number of required ventilator days by 49.3% (568 days) and decrease the total ICU days by 52.6% (895 days). On the triage protocol at ICU admission the survival rate in the red (93.7%) and yellow (62.5%) categories were significantly higher then that of the blue category (24.6%) with associated P values of < 0.0001 and 0.0003 respectively. Further, the survival rate of the red group was significantly higher than the overall survival rate of 70.9% observed in the cohort (P < 0.0001). At 48 and 120 hours, survival rates in the blue group increased but remained lower then the red or yellow groups.
Refinement of the triage protocol and implementation is required prior to future study, including improved training of triage officers, and protocol modification to minimize the exclusion from critical care of patients who may in fact benefit. However, our results suggest that the triage protocol can help to direct resources to patients who are most likely to benefit, and help to decrease the demands on critical care resources, thereby making available more resources to treat other critically ill patients.
On 11 June, 2009 the World Health Organization acknowledged that the world was facing a pandemic caused by a novel strain of H1N1 influenza . Although to date the overall prevalence of severe H1N1 illness has been low, experiences in Mexico [2, 3] and Manitoba  have increased concern that scarcities of critical care resources, such as mechanical ventilators [5, 6], will occur if a large second wave strikes during the fall in the Northern Hemisphere. Surge response strategies [7–11] will partially mitigate the surge impact, but may be inadequate to fully address health care demands. When faced with scarce resources, the principles of biomedical ethics and international law dictate that triage protocols be used to guide resource allocation [12–14].
In 2006, an expert panel convened by the Ontario Health Plan for an Influenza Pandemic Steering Committee developed a draft critical care triage protocol . The purpose of the protocol is to provide guidance for making triage decisions if critical care services are overwhelmed [5, 6, 16, 17].
The objective of this pilot study was to assess the usability of the draft triage protocol, to estimate its potential impact on patient outcomes, and its potential to increase resource availability based on a cohort of critically ill patients cared for during a non-pandemic period.
Materials and methods
We identified two retrospective cohorts of consecutive patients admitted to two academic medical/surgical intensive care units (ICUs) during an eight-week period of peak occupancy. ICU 'A' is a 16-bed general medical-surgical unit with 800 to 850 annual admissions serving a 472-bed tertiary care hospital and a 220-bed cancer hospital. ICU 'B' is a 23-bed medical-surgical-trauma-neurosurgical ICU with approximately 1200 admissions per year serving a 972-bed tertiary care hospital. Research ethics board approval was obtained at each of the participating institutions. The requirement for informed consent was waived by the research ethics board. Trained research assistants abstracted data from patient charts to create the case summary. An intensivist member of the study team at each site was available to answer clinical questions and review difficult cases. All case report forms were reviewed by the principal investigator, MDC, who requested additional data or site investigator chart review as necessary to clarify missing or unclear information. Data were entered into an Excel spreadsheet (Microsoft Excel 2003, Microsoft Corporation. USA).
An overview of the triage process is presented in Figure 1. Research assistants identified all patients admitted to these two participating ICUs from the ICU admission logs. Patients were enrolled into the study if they met the inclusion criteria for ICU admission defined in the triage protocol [see Additional data file 1] . Patients were then screened to see if they met any of the protocol's initial exclusion criteria or 'minimum qualifications for survival' (MQS) [see Additional data file 1] . Any patients who did not fulfill the triage protocol's initial exclusion criteria were summarized and presented to two of four intensivists who served as 'triage officers' to apply the triage protocol. Three of the four officers were involved in drafting the Ontario triage protocol. All triage officers received one hour of training on how to apply the protocol for this study (three in a single session and the fourth was one-on-one with the principal investigator due to scheduling difficulties). The training focus was triaging sample cases and calibration of the protocol application. Triage officers were instructed to imagine that they were actually conducting triage during a pandemic where demand for critical care services exceeded the available capacity. For study purposes, they were instructed that following the triage protocol was to be considered the 'standard of care' during the pandemic period. Deviations from the protocol would hold the same potential risks and consequences as do deviations from the standard of care during non-pandemic situations.
Patients at each of the two sites were scored independently by two of the triage officers. The triage officers did not review patients from the institution in which they practice, did not discuss these cases with one another during the triage process, and had no prior knowledge of the patients or their outcomes. Patient profiles were presented electronically in PDF format to the triage officers. The profiles included the patient's demographic data, admission diagnosis and limited past medical history. Daily triage reports were presented to the triage officers with the sequential organ failure assessment (SOFA) score component data and a calculated SOFA score, one day at a time, for each of the first five ICU days. The triage officers were instructed to make and record their determination of the patient's treatment priority before advancing to the next day's triage report, although they were allowed to move back in the PDF to review the patient profile and prior day's SOFA scores. Thus, at the time of their triage assessments, the triage officers were blinded to future status, and patient outcome.
Each patient was assigned a treatment priority (red = 'highest', yellow = intermediate', green = 'discharge to ward', or blue/black = 'expectant') by the triage officers at three separate time points (at the time of admission to the ICU, and 48, and 120 hours after admission). In addition, on day 1, 3 and 4 of their ICU admission, patients were assessed by the triage officer for exclusion criteria. Disagreements were resolved by a third intensivist (the principal investigator) scoring the case, in the same manner as described above, blinded to the decisions of the other triage officers and patient outcomes.
Protocol endpoints due to triage protocol exclusion criteria events or reaching discharge criteria beyond day 5 were assessed by a research assistant in conjunction with the principal investigator. We recorded the date and vital status of the patient at discharge from the ICU and hospital, or at day 90 following enrollment if the patient was still in the hospital.
Usability of the draft triage protocol was assessed based on: intensivist rating of confidence in their assignment of triage priority as reported in a brief questionnaire using five-point Likert-style scales assessing completeness of data, relevance of information provided, ability to develop a clear clinical understanding of the case and overall confidence in decision; requirement for third intensivist assessment; and chance-corrected (kappa) in assignment of triage priority. Based on the usability criteria, the triage protocol was considered successful in its current format if overall confidence was rated as confident or very confident in at least 90% of cases and disagreements requiring a third assessment in less then 5% of cases.
We assessed the impact of the triage protocol on resource availability by calculating the difference in days between when a patient achieved an endpoint in the triage protocol, exclusion criteria or prioritized as either blue 'expectant' or green 'dischargeable', compared with the actual date of extubation and discharge from the ICU. This allowed calculation of the number of ICU and ventilator days made available under the triage protocol.
Patient status was recorded at discharge from ICU, discharge from hospital or day 90 following enrollment if the patient remained admitted to hospital. Patients were categorized as deceased or alive at discharge from ICU and hospital, or if still admitted to hospital patients were categorized as: 'requiring ongoing life support', 'undergoing active medical care', or 'awaiting transfer to chronic care/rehabilitation'. Patients who were discharged from hospital alive had their destination of discharge categorized during chart abstraction as: 'home', 'chronic care', 'rehabilitation' or 'transfer to another acute care facility'.
We present descriptive data using mean and standard deviation for continuous variables, or median and interquartile ranges if data were skewed, and proportions and 95% confidence intervals for dichotomous variables. We compared continuous variables using unpaired t-tests (means) or Wilcoxon Rank-Sum (medians), and dichotomous variables using a chi-squared test or a Fisher's exact test if any expected cell frequencies were less than five. All tests were two-sided and P < 0.05 was considered statistically significant. We used Cohen's kappa (κ) with 95% confidence intervals to calculate chance-corrected agreement between triage officers. Based on the usability criteria the triage protocol would be considered successful in its current format if overall confidence is rated as confident or very confident in at least 90% of cases and disagreements requiring third assessment in less then 5% of cases. Missing data for the calculation of the SOFA score were assumed to be normal if the parameter was never collected by the clinicians caring for the patient, or the last reported value was used if the parameter had previously been measured but was missing at the time in question.
A total of 234 patients were included in the cohort (Figure 1 and Table 1), of which 178 (76.1%) met the triage inclusion criteria and would have been admitted to ICU during a pandemic based on the protocol. Of the overall cohort, 39.7% at some point met either the triage exclusion criteria or MQS and thus would have been managed expectantly (triaged blue). The number of patients who met triage inclusion or exclusion criteria was similar between hospitals. The mean age of the cohort was 59.8 years old and was similar between hospitals. A lower percentage of patients in hospital B were women, most likely because it is a trauma center. Overall, 69.2% of patients were mechanically ventilated, while 5.1% were never ventilated but were hypotensive and required vasopressor support. Overall, ICU survival was 76.9% and hospital survival was 70.9%.
The primary outcome measure was the usability of the triage protocol as measured by triage officer confidence in their assignment of priority, requirement for arbitration and agreement with each other. Overall, triage officers were either confident or very confident in 68.4% of their scores and arbitration was required in 54.9% of cases. The agreement between raters (Table 2) was highest on admission (kappa 89.2 (77.9, 100) and 73.7 (55.6, 91.7) for hospitals A and B, respectively. Agreement at 48 hours and 120 hours were substantially lower (48 hours: kappa 60.7 (41.2, 80.2) and 31.5 (16.1, 46.8); 120 hours: kappa 42.4 (28.0, 57.0) and 0 (-17.2, 16.2) for hospitals A and B). The latter analyses were based on fewer patients because only patients who were not triaged blue and were still in the ICU were triaged at these subsequent time points.
Secondary outcome measures of this pilot study included an assessment of the impact of the triage protocol on the availability of critical care resources and measurement of patient outcomes. Total ventilator and ICU days on and off protocol are reported in Table 3. Application of the triage protocol would potentially decrease the number of ventilator days required by 49.3% (568 days) and a decrease of 52.6% (895 days) in the total ICU days. Table 4 presents outcomes based on triage category at admission, 48 hours and 120 hours. On the triage protocol at ICU admission the survival rate in the red (93.7%) and yellow (62.5%) categories were significantly higher then that of the blue category (24.6%) with associated P value of less than 0.0001 and 0.0003, respectively. Further, the survival rate of the red group was significantly higher than the overall survival rate of 70.9% observed in the cohort (P < 0.0001). At 48 hours the survival rates in the red (90.5%) and yellow (94.1%) categories remain significantly higher than that of the blue category (45.4%) with P values of 0.002 and 0.001; however, beyond 48 hours the number of patients remaining in the ICU had decreased substantially, limiting inferences from these analyses.
To examine the outcomes of those patients who would have been triaged to the blue category yet survived in the observation cohort, the outcomes and criteria for those ever triaged to blue category (Table 5) are compared with outcomes for those never triaged to blue category (Table 6). Of the patients triaged to blue, 32.3% survived to hospital discharge. Almost half of the survivors triaged to the blue category had been triaged blue for failing to sufficiently improve their SOFA score at either 48 or 120 hours. The most common exclusion criterion triggering a triage to category blue in those patients who survived to hospital discharge was the presence of metastatic cancer.
Following the triage exercise, triage officers were polled to assess if they believed they were the 'type' of person who could make triage decisions; if they would volunteer to be a triage officer in a pandemic; and their view regarding the training required to be a triage officer. All except triage officer 3 said they believed they were the type of person who could make triage decisions and would volunteer in a pandemic. All stated that specific comprehensive training for intensivists to be triage officers is required.
This study provides insight into the use of triage protocols during pandemic periods, and informs the design for a larger multi-center prospective study to evaluate the Ontario triage protocol. Our results highlight the need to develop a selection process for triage officers and to provide comprehensive training for triage officers. Many cases required arbitration due to disagreement between triage officers 3 and 4. The arbitrator ruled in agreement with triage officer 4 on 95.7% of the decisions.
The overall degree of confidence in triage decisions fell below the a priori target of greater than 90% of decisions being rated as either confident or very confident. Additionally, the rate of arbitration was high, although primarily the result of decisions by one triage officer. A lack of consistency between triage officers threatens the equity of the process. The clinical information provided about patients was much less then intensivists would typically receive in clinical practice, and triage officers reported that the data were insufficient to establish a clear clinical picture in 20.5% of cases (data not shown). Thus, more specific clinical information will have to be provided in a future triaging study, which may be available during prospective application of the protocol, but increasing dependency on information can limit the usability of the protocol. Other possible reasons for lower than anticipated confidence among triage officers and high rate of arbitration is inexperience with the protocol, inadequate training, and the lack of prior studies exploring the impact of triage protocols such as this. Future qualitative research could better illuminate factors that contribute to lack of confidence in triage decision-making using a protocol such as the one studied.
Although not specifically designed or powered to evaluate the impacts of the triage protocol on patient outcome, this study provides some insight into its performance. The primary goal of a tertiary triage protocol is to direct the limited available resources to those who are most likely to benefit from them. On this point, the protocol appears to serve its function with those triaged to the highest priority for ICU care (red) having survival rates significantly higher than the rate in the observation cohort and markedly higher than those triaged blue. The alternative to the use of a protocol would be leaving individual physicians to make allocation decisions on their own. Prior research suggests that such ICU physicians' ability to predict patient outcome without the aid of a decision support tool, such as a triage protocol, is poorer than we have observed in this study . Additionally, failure to use a standard triage protocol to guide decisions regarding the allocation of scarce resources is less efficient and ethically less desirable [19–21].
A second goal of tertiary triage is to make more critical care resources available. On this point, the protocol shows promise. Through the application of the protocol's inclusion criteria, exclusion criteria, MQS and discharge criteria, the demand for ventilators could be decreased by 49.3% and for ICU admission by 52.6% compared with standard practices. As an illustrative example, based upon the 568 days of ventilation made available through the use of the protocol, using rates from the first wave of H1N1 in Canada assuming an average of 10 days of ventilation and an 89% survival rate, 50 lives could potentially be saved by the resources made available.
Ethical frameworks suggest that restrictions placed on the allocation of scarce resources must be proportionate to the expected and observed shortfalls . Thus, it is particularly important to minimize the number of people who are triaged blue but may possibly survive under normal circumstances. Evaluating this outcome is somewhat difficult in this study given that patients were receiving what is essentially optimum care in tertiary ICUs, whereas during a pandemic, triage would be instituted only when emergency mass critical care is being used which requires significantly modified standards of care. Thus, one would expect the mortality rate in the sickest of patients (those excluded or triaged blue) would be higher than seen in this observation cohort. Further, survival in this study was defined as status at hospital discharge, whereas the triage protocol was designed to direct resources to patients with greater than 50% two-year survival. Many of the patients who survived to hospital discharge but would have been triaged blue or excluded, such as those with metastatic malignancies, are unlikely to meet the higher bar of greater than 50% two-year survival. Patients are often repatriated to the facilities from which they were transferred once they no longer require services only available at tertiary centres. As we were unable to collect information from the referring facilities in this study, some of these patients may not have survived to discharge from the referring facility.
This study is limited in that it is a pilot study of a relatively small cohort of critically ill patients from two Ontario hospitals. Although clinical data were presented to triage officers prospectively, data were less than would typically be available in practice, and some data were missing. Our results highlight the need for a central triage committee, comprehensive training, and data management infrastructure so the committee can monitor triage outcomes during a pandemic, adjusting the protocol to correct for either over-triage or under-triage [15, 23–25]. Further, the exclusion criteria and prioritization criteria should be incorporated into triage decisions on a graduated basis depending upon the anticipated shortfall in resource supply. Additionally, the protocol exclusion, MQS and prioritization criteria require further study and modification to minimize the potential for denying resources to those who may benefit from them.
We evaluated the Ontario critical care triage protocol in this pilot study, which generated insights about future triage practices. Although further research is needed, our results suggest that the triage protocol can help to direct resources to patients who are most likely to benefit from them, and help to decrease the demands on critical care resources, thereby making available more resources to treat other critically ill patients. We also documented the need for more comprehensive patient summaries when such decisions are being made, careful triage officer selection, improved triage officer training, and infrastructure to allow timely tracking and analysis of the consequences of triage protocols.
Attention must be paid to the appropriate training and selection of triage officers in to improve confidence in their decisions and agreement between triage officers.
Application of the triage protocol can potentially free up significant critical care resources, which may be re-directed towards managing critically ill patients during a pandemic or disaster.
The current protocol is able to identify patients who are most likely to survive and allow resources to be targeted to this group. However, the protocol does require further investigation and modification to minimize the number of patients who would potentially be excluded critical care in a pandemic but whom may survive if they were to receive critical care.
intensive care unit
minimum qualifications for survival
sequential organ failure assessment.
Zarocostas J: World Health Organization declares A (H1N1) influenza pandemic. BMJ 2009, 338: b2425. 10.1136/bmj.b2425
Perez-Padilla R, de la Rosa-Zamboni D, Ponce de Leon S, Hernandez M, Quiñones-Falconi F, Bautista E, Ramirez-Venegas A, Rojas-Serrano J, Ormsby CE, Corrales A, Higuera A, Mondragon E, Cordova-Villalobos JA: Pneumonia and Respiratory Failure from Swine-Origin Influenza A (H1N1) in Mexico. N Engl J Med 2009, 361: 680-689. 10.1056/NEJMoa0904252
Novel Swine-Origin IA: Emergence of a Novel Swine-Origin Influenza A (H1N1) Virus in Humans. N Engl J Med 2009, 360: 2605-2615. 10.1056/NEJMoa0903810
Kermode-Scott B: Canada has world's highest rate of confirmed cases of A/H1N1, with Aboriginal people hardest hit. BMJ 2009, 339: b2746. 10.1136/bmj.b2746
Christian MD, Devereaux AV, Dichter JR, Geiling JA, Rubinson L: Definitive care for the critically ill during a disaster: current capabilities and limitations: from a Task Force for Mass Critical Care summit meeting, January 26-27, 2007, Chicago, IL. Chest 2008, 133: 8S-17S. 10.1378/chest.07-2707
Osterholm MT: Preparing for the next pandemic. N Engl J Med 2005, 352: 1839-1842. 10.1056/NEJMp058068
Hick JL, Hanfling D, Burstein JL, DeAtley C, Barbisch D, Bogdan GM, Cantrill S: Health care facility and community strategies for patient care surge capacity. Ann Emerg Med 2004, 44: 253-261. 10.1016/j.annemergmed.2004.04.011
Rubinson L, Nuzzo JB, Talmor DS, O'Toole T, Kramer BR, Inglesby TV: Augmentation of hospital critical care capacity after bioterrorist attacks or epidemics: recommendations of the Working Group on Emergency Mass Critical Care. Crit Care Med 2005, 33: 2393-2403. 10.1097/01.CCM.0000173411.06574.D5
Rubinson L, Hick JL, Curtis JR, Branson RD, Burns S, Christian MD, Devereaux AV, Dichter JR, Talmor D, Erstad B, Medina J, Geiling JA, Task Force for Mass Critical Care: Definitive care for the critically ill during a disaster: medical resources for surge capacity: from a Task Force for Mass Critical Care summit meeting, January 26-27, 2007, Chicago, IL. Chest 2008, 133: 32S-50S. 10.1378/chest.07-2691
Rubinson L, Hick JL, Hanfling DG, Devereaux AV, Dichter JR, Christian MD, Talmor D, Medina J, Curtis JR, Geiling JA, Task Force for Mass Critical Care: Definitive care for the critically ill during a disaster: a framework for optimizing critical care surge capacity: from a Task Force for Mass Critical Care summit meeting, January 26-27, 2007, Chicago, IL. Chest 2008, 133: 18S-31S. 10.1378/chest.07-2690
Rubinson L, O'Toole T: Critical care during epidemics. Critical Care 2005, 9: 311-313. 10.1186/cc3533
Baskett PJ: Ethics in disaster medicine. Prehospital Disaster Med 1994, 9: 4-5.
Domres B, Koch M, Manger A, Becker HD: Ethics and triage. Prehospital Disaster Med 2001, 16: 53-58.
Vollmar LC: Military Medical Ethics. Military Medicine In War: The Geneva Conventions Today 2005., Chapter 23:
Christian MD, Hawryluck L, Wax RS, Cook T, Lazar NM, Herridge MS, Muller MP, Gowans DR, Fortier W, Burkle FM: Development of a triage protocol for critical care during an influenza pandemic. CMAJ 2006, 175: 1377-1381.
Ministry of Health and Long-Term Care: Ontario Health Plan for an Influenza Pandemic.[http://www.health.gov.on.ca/english/providers/program/emu/pan_flu/pan_flu_plan.html]
Public Health Agency of Canada: . Canadian Pandemic Influenza Plan 2004.
Rodriguez RM, Wang NE, Pearl RG: Prediction of poor outcome of intensive care unit patients admitted from the emergency department. Crit Care Med 1997, 25: 1801-1806. 10.1097/00003246-199711000-00016
Challen K, Bentley A, Bright J, Walter D: Clinical review: mass casualty triage--pandemic influenza and critical care. Crit Care 2007, 11: 212. 10.1186/cc5732
Kuschner WG, Pollard JB, Ezeji-Okoye SC: Ethical triage and scarce resource allocation during public health emergencies: tenets and procedures. Hosp Top 2007, 85: 16-25. 10.3200/HTPS.85.3.16-25
O'Laughlin DT, Hick JL: Ethical issues in resource triage. Respir Care 2008, 53: 190-197.
University of Toronto Joint Centre for Bioethics Pandemic Influenza Working Group: Stand On Guard For Thee: Ethical considerations in preparedness planning for pandemic influenza.[http://www.jointcentreforbioethics.ca/people/documents/upshur_stand_guard.pdf]
Devereaux A, Christian MD, Dichter JR, Geiling JA, Rubinson L: Summary of suggestions from the Task Force for Mass Critical Care summit, January 26-27, 2007. Chest 2008, 133: 1S-7S. 10.1378/chest.08-0649
Devereaux AV, Dichter JR, Christian MD, Dubler NN, Sandrock CE, Hick JL, Powell T, Geiling JA, Amundson DE, Baudendistel TE, Braner DA, Klein MA, Berkowitz KA, Curtis JR, Rubinson L, Task Force for Mass Critical Care: Definitive care for the critically ill during a disaster: a framework for allocation of scarce resources in mass critical care: from a Task Force for Mass Critical Care summit meeting, January 26-27, 2007, Chicago, IL. Chest 2008, 133: 51S-66S. 10.1378/chest.07-2693
Frykberg ER: Medical management of disasters and mass casualties from terrorist bombings: how can we cope? J Trauma 2002, 53: 201-212. 10.1097/00005373-200208000-00001
Research funding for this project was provided by a grant from the Ontario Ministry of Health and Long Term Care and D. Cook is a Research Chair of the Canadian Institutes for Health Research. We wish to thank the Ontario Ministry of Health and Long Term Care Emergency Management Unit for their support throughout this project. We also wish to acknowledge the leadership provided by the steering committee of the Ontario Health Pandemic Influenza Plan as well as the contributions of all those who collaborated on its development. The findings and conclusions in this article are those of the authors and do not represent official positions of the Ministry of Health and Long Term Care or the Department of National Defense.
The authors declare that they have no competing interests.
MC and DC designed the study and secured funding. MC, CH, NL, RW, MH, and DL participated in data collection. LG, MC and DC conducted the data analysis. MC and DC produced the original manuscript draft. All authors participated in editing and reviewing the manuscript. All authors read and approved the final manuscript.
Electronic supplementary material
Additional file 1: Word file that provides the detailed inclusion and exclusion criteria that form part of the triage protocol. These are in greater detail then were published in the original article detailing the triage protocol, which is referenced in this paper. (DOC 46 KB)
Authors’ original submitted files for images
About this article
Cite this article
Christian, M.D., Hamielec, C., Lazar, N.M. et al. A retrospective cohort pilot study to evaluate a triage tool for use in a pandemic. Crit Care 13, R170 (2009). https://doi.org/10.1186/cc8146
- Sequential Organ Failure Assessment
- Additional Data File
- Sequential Organ Failure Assessment Score
- Triage Protocol
- Triage Officer