A study on the reliability and validity of two four-level emergency triage systems

Introduction

We compared the reliability and validity of two four-level emergency triage systems: the ETS1, used in our emergency department (ED); and the ETS2, a triage algorithm derived from the Emergency Severity Index [1].

Methods

This is an observational retrospective study of 189 patients admitted to our ED in 2 weeks of October 2006. Triage scenarios were designed with medical records. Ten trained triage nurses were randomly assigned either to training in ETS2, or to refresh the ETS1. They independently assigned triage scores to the scenarios, at time zero and after 6 months. Both triage systems have four urgency categories (UC): UC1 = immediate response; UC2, UC3, UC4 = assessment within 20, 60, 120 minutes. We collected demographic and clinical characteristics, nurse triage category, admission status and site, nurse triage forms with presenting complaint, mode and time of arrival, past diseases, vital signs, and pain score. For each scenario we considered 'true triage' the mode of the UC assigned by the nurses. Weighted kappa (K) was used to calculate inter-rater and intra-rater reliability in each of the two groups of nurses. The relationships between the 'true triage' and admission, and admission site were assessed.

Results

The UCs assigned were similar in two groups: 20% versus 21% with UC4, 50% versus 48% with UC3, 28% versus 28% with UC2, 2% versus 3% with UC1. A complete disagreement in UC was found in 3% and 5% cases of ETS1 and ETS2; a complete agreement in 52% and 56% cases of ETS1 and ETS2. Inter-rater reliability among nurses using ETS1 and ETS2 was K = 0.73 (95% CI: 0.59–0.87) and K = 0.79 (95% CI: 0.65–0.93), respectively; intra-rater reliability was: K = 0.82 (95% CI: 0.67–0.96) and K = 0.78 (95% CI: 0.62–0.93) in ETS1 and ETS2. Hospital admission by ETS1 and ETS2 was similar for UC2 (39% vs 37%), UC3 (5% vs 8%), and UC4 (3% vs 0%); 100% of patients with UC1 in ETS1 and 60% in ETS2 were admitted to the ICU.

Conclusion

The two emergency triage systems showed similar reliability and validity. ETS2 is easier to consult but worse in prediction of ICU admission. To our knowledge this is the first study on the intra-rater reliability of two four-level emergency triage systems.