Skip to content

Advertisement

Open Access

Development and validation of a pre-hospital “Red Flag” alert for activation of intra-hospital haemorrhage control response in blunt trauma

  • Sophie Rym Hamada1, 2Email author,
  • Anne Rosa3,
  • Tobias Gauss4,
  • Jean-Philippe Desclefs5,
  • Mathieu Raux6,
  • Anatole Harrois1,
  • Arnaud Follin7,
  • Fabrice Cook8,
  • Mathieu Boutonnet9,
  • the Traumabase® Group,
  • Arie Attias,
  • Sylvain Ausset,
  • Gilles Dhonneur,
  • Olivier Langeron,
  • Catherine Paugam-Burtz,
  • Romain Pirracchio,
  • Bruno Riou,
  • Guillaume de St Maurice,
  • Bernard Vigué,
  • Alexandra Rouquette2, 10 and
  • Jacques Duranteau1
Contributed equally
Critical Care201822:113

https://doi.org/10.1186/s13054-018-2026-9

Received: 5 December 2017

Accepted: 3 April 2018

Published: 5 May 2018

Abstract

Background

Haemorrhagic shock is the leading cause of early preventable death in severe trauma. Delayed treatment is a recognized prognostic factor that can be prevented by efficient organization of care. This study aimed to develop and validate Red Flag, a binary alert identifying blunt trauma patients with high risk of severe haemorrhage (SH), to be used by the pre-hospital trauma team in order to trigger an adequate intra-hospital standardized haemorrhage control response: massive transfusion protocol and/or immediate haemostatic procedures.

Methods

A multicentre retrospective study of prospectively collected data from a trauma registry (Traumabase®) was performed. SH was defined as: packed red blood cell (RBC) transfusion in the trauma room, or transfusion ≥ 4 RBC in the first 6 h, or lactate ≥ 5 mmol/L, or immediate haemostatic surgery, or interventional radiology and/or death of haemorrhagic shock. Pre-hospital characteristics were selected using a multiple logistic regression model in a derivation cohort to develop a Red Flag binary alert whose performances were confirmed in a validation cohort.

Results

Among the 3675 patients of the derivation cohort, 672 (18%) had SH. The final prediction model included five pre-hospital variables: Shock Index ≥ 1, mean arterial blood pressure ≤ 70 mmHg, point of care haemoglobin ≤ 13 g/dl, unstable pelvis and pre-hospital intubation. The Red Flag alert was triggered by the presence of any combination of at least two criteria. Its predictive performances were sensitivity 75% (72–79%), specificity 79% (77–80%) and area under the receiver operating characteristic curve 0.83 (0.81–0.84) in the derivation cohort, and were not significantly different in the independent validation cohort of 2999 patients.

Conclusion

The Red Flag alert developed and validated in this study has high performance to accurately predict or exclude SH.

Keywords

Severe traumaSevere haemorrhageProtocolOrganizationAnticipation

Background

Haemorrhage remains the leading cause of early preventable death in severe trauma [1, 2]. A multidisciplinary analysis showed that approximately 2.5% of the deaths in a trauma centre are preventable or potentially preventable. Among the main causes were haemorrhage (39%) and multiple organ failure (28%), often a consequence of haemorrhagic shock. The main reasons for preventable death due to haemorrhage were delayed recognition and management [3]. Organizational optimization is essential to control bleeding as quickly as possible and to reduce patient mortality [46]. It is therefore crucial to identify during the pre-hospital phase those patients at high risk of severe haemorrhage (SH) to quickly activate a specific intra-hospital standardized haemorrhage control response, connecting the multispecialty trauma team, blood bank, transfusion protocols, interventional radiology and surgery [7].

Thus, to address efficiently the challenge of SH and shape the response, the design of a haemorrhage specific alert is necessary. Standard triage algorithms are designed to guide severe trauma patients to appropriate trauma centres [8, 9] and trigger trauma team activation [10]. The pre-hospital MGAP score (mechanism, Glasgow coma scale, age and arterial pressure) [11] was developed to predict mortality but showed a proper ability to predict SH (area under the curve 0.7, 95% CI 0.66–0.73) [12]. Established haemorrhage scores predict the need for massive transfusion [1315]. The TASH score is probably one of the most widely cited scores to predict massive transfusion [16]. However, massive transfusion only applies to a minority of patients, whereas a timely integrative haemostatic strategy could decrease overall transfusion requirements. The aforementioned massive transfusion scores are only validated with intra-hospital data, which renders their application questionable during the pre-hospital phase. A “Code Red” policy has been implemented in trauma centres across the UK [17, 18] with the pre-arrival organization seen as an integral part of the severe haemorrhage pathway [19]. This activation code consists of three criteria (suspicion or evidence of active haemorrhage, systolic arterial blood pressure < 90 mmHg, failure to respond to a fluid bolus) but its predictive accuracy has not yet been evaluated. On a pragmatic standpoint, this type of alert is of utmost importance for trauma centres since emergency care may compete with elective care because of common facilities and workforces.

The aim of our study was to develop and validate an easy-to-use pre-hospital prediction tool for SH in blunt trauma patients derived from a prediction model. This tool is meant to be used as a binary Red Flag alert to activate a specific intra-hospital severe haemorrhage response. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement was followed to report its results [20].

Methods

Trauma centres and registry

This multicentre observational study used the data collected prospectively from a trauma registry (Traumabase®, traumabase.eu) shared between the six trauma centres of the Paris area in France. These six centres progressively joined the registry between January 2011 and June 2015. Since then, data collection is exhaustive and covers the whole administrative area around Paris, Ile-de-France. The Traumabase® obtained approval from the Advisory Committee for Information Processing in Health Research (CCTIRS, 11.305bis) and from the National Commission on Informatics and Liberties (CNIL, 911461) and meets the requirements of the local and national ethics committee (Comité de Protection des Personnes, Paris VI). The structure of the database integrates algorithms for consistency and coherence. Data monitoring is performed by a central administrator. Sociodemographic, clinical, biological and therapeutic data (from the pre-hospital phase to discharge from the intensive care unit) are systematically collected for all trauma patients. A description of the Emergency Medical System (EMS) and the Trauma System and the characteristics of the Ile-de-France can be found in Hamada et al. [21]. In France, a 24/7 available dispatching physician located in a centralized call centre decides which emergency vector, either a paramedic-staffed ambulance or a physician-staffed mobile intensive care unit, is to be deployed on the basis of the trauma bystanders’ call. Patients brought to dedicated trauma rooms in trauma centres are suspected to be major trauma by the pre-hospital team and are necessarily transported by a physician-staffed ambulance. All patients transported to the trauma rooms of the participating centres were included in the registry (Traumabase®).

Our methodology used a three-step approach: developing a model, deriving a score and transforming the score into a binary alert by choosing the cut-off value.

Patient selection

Every trauma patient registered in the Traumabase® since January 2011 was included in the study. Patients were excluded if they were admitted after secondary transfer, after penetrating trauma or after pre-hospital traumatic cardiac arrest, or if no pre-hospital data were available. Two sub-samples were constituted: the derivation cohort which included all patients admitted from January 2011 to May 2015, and the validation cohort which included all patients admitted from June 2015 to November 2016 (Fig. 1).
Figure 1
Fig. 1

Flowchart of the study. SH severe haemorrhage

Definition of severe haemorrhage

Patients were retrospectively considered to present SH on admission to the trauma centre if any of the following criteria was present: need for any packed red blood cell (RBC) transfusion upon arrival in the resuscitation room, transfusion of 4 packed RBCs or more within the first 6 h [22, 23], blood lactate concentration ≥ 5 mmol/L upon arrival [24], need for immediate haemostatic surgery or interventional radiology before complete injury assessment by whole-body CT scan or death from haemorrhagic shock [25]. These criteria were chosen to reflect the heterogeneity and complexity of SH clinical presentation as there is no consensual definition in the literature. The need for an immediate haemostatic intervention was chosen as a reflection of haemorrhage intensity, and for further anticipation of the need of haemostatic resource mobilization. The resuscitation room transfusion criterion and death secondary to haemorrhagic shock were chosen to select the most severe and actively bleeding patients. The transfusion-related criteria represented the dynamic and evolving character of bleeding and need for transfusion over the first hours. Massive transfusion is usually defined as 10 packed RBCs in the first 24 h [26] but this definition is currently questioned [23, 27] and other criteria (transfusion requirements in the first 6 h, transfusion of at least 3 packed RBCs in 1 h or of 5 packed RBCs in 4 h) have been shown to be better correlate with mortality [22, 27]. The transfusion-related criteria were chosen in this study to be a good balance between these different definitions. Finally, the criterion concerning blood lactate level at admission quantified the magnitude of tissue hypoperfusion related to haemorrhage. Apart from metabolic sources for elevated blood lactate level (e.g. alcohol, fast, ethylene glycol), a level higher than 5 mmol/L during the first 24 h is a risk factor for mortality or multiple organ failure [2830]. For each patient, the presence or absence of SH at admission was adjudicated by SRH, initially blinded to the pre-hospital variables.

Potential pre-hospital predictors

Thirteen predictors of SH were selected or computed using exclusively pre-hospital data from management on scene and during transport. These criteria were selected based on their clinical significance and ease of use in the pre-hospital time-critical setting: age, sex, minimal systolic, diastolic and mean arterial blood pressure (SBP, DBP and MBP), maximal heart rate (HR), minimal oxygen saturation (SpO2), minimal Glasgow Coma Scale (GCS), clinically unstable pelvis, early on-scene point-of-care haemoglobin concentration [31], tracheal intubation and vasopressor administration. The maximal Shock Index was calculated according to the formula: SI = maximal HR / minimal SBP [32].

Other measured variables

Demographic data, trauma characteristics and outcome were also recorded. For the Simplified Acute Physiology Score (SAPS), the worst value of each variable over the first 24 h was taken into account. The Abbreviated Injury Scale (AIS) version 2005 and the Injury Severity Score (ISS) were calculated when the whole-body injury assessment was completed. The expected probability of survival was calculated using the Trauma and Injury Severity Score (TRISS) with the most recent coefficients [33, 34]. Due to missing data, the TRISS was computed by giving a respiratory rate of 20/min in all patients [35]. The MGAP score was computed as a comparison basis for SH prediction [11].

Sample size

We used the entire large cohort (7945 patients) as there is no generally accepted approach to estimate the sample size requirements for derivation and validation studies of risk prediction models. The number of events in our sample far exceeds the required number established using previously published rules (10 events per candidate variable for derivation studies and at least 100 events for validation studies) and therefore is expected to provide robust estimates [20].

Statistical analysis

Continuous data were described as mean ± standard deviation or median (quartiles 1–3) according to their distribution, and categorical variables as count (percentages). The derivation cohort was used to train the prediction model. Univariate analyses were performed to evaluate crude associations between pre-hospital data and the presence of SH using chi-square and Student’s t tests (or the Mann–Whitney test when necessary) depending on the variable type. Each variable with p < 0.2 was retained as a candidate variable and Spearman correlation coefficients were computed to evaluate collinearity (r > 0.8). Candidate continuous variables were dichotomized using receiver operating characteristic (ROC) curves and Youden’s index [36] to identify the cut-off value, except for SpO2 for which the clinically relevant cut-off value of ≤ 90% was chosen. Then, candidate binary variables were entered altogether into a multivariate logistic regression model and selection was performed using a backward stepwise procedure to optimize the Akaike criterion. All variables were tested for pairwise univariate interactions. Multiple imputation via a chained equation was used to handle missing data (R PACkage “mice” V 2.3 [37]). A maximum of 5% missing data was observed for the imputed variables. Model calibration was assessed using the Hosmer and Lemeshow statistic [38]. Model discrimination was assessed using the ROC curve (area under the ROC curve (AUC)) and a bootstrap methodology (1000 samples) [39] was used to quantify any optimism (averaged difference between the apparent AUC of the model developed on each bootstrap sample and its AUC on the original sample) in the final prediction model.

To derive the Red Flag binary alert from this model, a score was computed for each patient of the derivation cohort using various combinations of the number of points attributed to each variable that remained in the final prediction model (one point for each variable or two points for variables with a higher model coefficient than the others). Their predictive accuracy was evaluated using the AUC and the Youden’s index was used to identify the cut-off value for the Red Flag binary alert with the best balance between simplicity of use and predictive performance assessed using sensitivity, specificity, positive and negative predictive values (PPV and NPV) and positive likelihood ratio (+LR). Contingency mosaics were drafted for the values around the determined cut-off value.

Finally, the predictive accuracy of the final prediction model and the predictive performance of the resulting Red Flag binary alert were independently assessed in the validation cohort. Model calibration was graphically checked using a calibration plot representing the agreement between risks of SH predicted by the model and observed proportions in the validation cohort.

Discrimination was explored using the AUC ROC. A bootstrap methodology was used to compute the confidence interval (CI) of the AUCs (R package “pROC” V1.10 [40]). The AUC of Red Flag was compared to the AUC of the MGAP score. All tests that were two-sided at p ≤ 0.05 were considered significant. R 3.3.3 software (R Foundation for Statistical Computing, Vienna, Austria) was used for analysis.

Results

The flowchart of both the derivation (n = 4339) and validation (n = 3606) cohorts is presented in Fig. 1. Patients presenting with initial cardiac arrest, penetrating trauma or no pre-hospital data available were excluded, leaving 3675 (85%) patients for analysis in the derivation cohort and 2999 (83%) in the validation cohort. The distribution of patient across centres is described and illustrated in Additional file 1.

The main characteristics of both cohorts are presented in Table 1. The observed mortality was lower in both groups than the expected mortality according to the TRISS. The two cohorts significantly differed mainly in terms of haemostatic strategies with less surgery (58 vs 51%), more angio-embolization (3% vs 6%) and a shorter intensive care unit length of stay in the validation cohort.
Table 1

Derivation and validation cohort characteristics

 

Derivation

Validation

Comparison

SH

(n = 672)

No SH

(n = 3003)

SH

(n = 415)

No SH

(n = 2584)

SH

p

No SH

p

Demography and outcome

 Age (years)

42 ± 19

37 ± 16

44 ± 19

38 ± 17

ns

ns

 Male (%)

465 (69%)

2340 (78%)

312 (75%)

2017 (78%)

0.040

ns

 BMI (kg/m2)

25.3 ± 5

24.7 ± 4.3

25.2 ± 4.4

24.8 ± 4.7

ns

ns

 SAPS II

45 ± 22

21 ± 15

46 ± 23

21 ± 15

ns

ns

 ICU LOS

18 ± 23

8 ± 15

14 ± 19

7 ± 15

0.001

0.001

Hospital mortality

163 (25%)

137 (5%)

93 (23%)

110 (4%)

ns

ns

Predicted mortality by TRISSa (%)

27

6

27

6

Mechanism of injury (all blunt)

 MVA

133 (20%)

765 (26%)

97 (23%)

617 (24%)

  

 Motorbike

151 (23%)

905 (30%)

95 (23%)

861 (33%)

  

 Pedestrian and bicycle

109 (16%)

381 (13%)

47 (11%)

343 (13%)

ns

ns

 Fall

243 (36%)

792 (26%)

144 (35%)

621 (24%)

  

 Miscellaneous

36 (5%)

150 (5%)

32 (8%)

142 (6%)

  

Severity of injuries

 ISS

30 (18–38)

12 (5–20)

27 (14–41)

12 (5–21)

ns

ns

 Head and neck AIS

2 (0–4)

0 (0–3)

1 (0–3)

0 (0–2)

ns

ns

 Thorax AIS

3 (0–3)

0 (0–3)

3 (0–4)

0 (0–3)

ns

ns

 Abdomen AIS

2 (0–3)

0 (0–2)

2 (0–3)

0 (0–2)

ns

ns

 Extremities pelvis AIS

3 (2–3)

2 (0–2)

3 (1–4)

0 (0–3)

ns

0.001

At admission

 Total pre-hospital time (min)

85 ± 39

80 ± 37

80 ± 37

77 ± 34

0.010

ns

 SBP (mmHg)

102 ± 34

129 ± 23

107 ± 35

130 ± 24

ns

ns

 DBP (mmHg)

62 ± 23

76 ± 16

65 ± 24

79 ± 17

ns

0.001

 Haemoglobin (g/dl)

10.2 ± 2.6

13.4 ± 1.7

10.7 ± 2.7

13.5 ± 1.7

0.010

ns

 Lactate (mmol/L)

4.8 ± 3.4

1.9 ± 0.9

4.9 ± 3.4

2 ± 0.9

ns

0.002

 Prothrombin time (%)

57 ± 22

83 ± 15

61 ± 23

84 ± 15

0.010

ns

Surgery day 1

562 (84%)

1867 (62%)

305 (74%)

1235 (48%)

0.001

0.001

Angio-embolization day 1

111 (17%)

67 (2%)

88 (21%)

86 (3%)

0.001

0.001

SH characteristics

 Immediate surgery

123 (16%)

88 (21%)

ns

 Transfusion in trauma room

425 (63%)

245 (59%)

ns

 Lactates > 5 mmol/L

323 (51%)

194 (52%)

ns

 ≥ 4 RBCs in first 6 h

385 (57%)

204 (49%)

0.010

Results expressed as mean ± standard deviation, n (%) or median (1st quartile–3rd quartile)

SH severe haemorrhage, ns not significant, BMI body mass index, SAPS Simplified Acute Physiology Index, ICU LOS intensive care unit length of stay, TRISS Trauma Injury Severity Score, MVA motor vehicle accident, ISS Injury Severity Score, AIS Abbreviated Injury Scale, SBP systolic blood pressure, DBP diastolic blood pressure RBC red blood cell

a TRISS computed by giving a respiratory rate of 20/min in all patients [35]

Score development

Table 2 presents the results of the univariate analysis performed in the derivation cohort (Additional file 2). All of the pre-hospital variables were significantly associated with SH. Collinear variables were HR, SBP, Shock Index (Shock Index was kept), and SBP, DBP and MBP; the latter was kept as not collinear to the Shock Index. As reported in the lower part of Table 2, when there was no clinically relevant cut-off value, Youden’s index was used to dichotomize continuous variables (resulting thresholds are presented in Additional file 3).
Table 2

Univariate analysis of pre-hospital variables in the derivation cohort

 

SH

(n = 672)

No SH (n = 3003)

Missing values,

n (%)

p

Male

465 (69%)

2258 (78%)

7 (0%)

< 0.001

Age (years)

42 ± 19

37 ± 16

7 (0%)

< 0.001

SBP min (mmHg)

93 ± 30

118 ± 22

50 (1%)

< 0.001

DBP min (mmHg)

55 ± 18

70 ± 15

65 (2%)

< 0.001

MBP min (mmHg)

68 ± 21

86 ± 16

50 (1%)

< 0.001

HR max (/min)

108 ± 27

93 ± 20

73 (2%)

< 0.001

Shock Index (HR/SBP)

1.3 ± 0.8

0.8 ± 0.4

77 (2%)

< 0.001

Capillary haemoglobin (g/dl)

12.8 ± 2.2

14.2 ± 1.7

183 (5%)

< 0.001

SpO2 min (%)

97 (92–100)

98 (96–100)

99 (3%)

< 0.001

Glasgow Coma Scale

14 (7–15)

15 (14–15)

17 (0%)

< 0.001

Pelvic trauma

115 (18%)

106 (4%)

141 (5%)

< 0.001

Vasopressor

216 (33%)

140 (5%)

37 (1%)

< 0.001

Pre-hospital intubation

385 (57%)

692 (23%)

6 (0%)

< 0.001

Binarized variables (Youden’s Index)

 SBP min ≤ 100

421 (64%)

569 (19%)

 

< 0.001

 MBP ≤ 70 mmHg

382 (58%)

448 (15%)

 

< 0.001

 HR max ≥100

418 (64%)

1050 (36%)

 

< 0.001

 Shock Index (HR/SBP) ≥ 1

394 (60%)

419 (14%)

 

< 0.001

 Capillary haemoglobin ≤ 13

382 (59%)

812 (29%)

 

< 0.001

 SpO2 min ≤ 90%a

142 (22%)

189 (6%)

 

< 0.001

 Glasgow Coma Scale ≤13a

321 (48%)

712 (24%)

 

< 0.001

Results expressed as mean ± standard deviation, n (%) or median (1st quartile–3rd quartile)

SH severe haemorrhage, SBP systolic blood pressure, DBP diastolic blood pressure, MBP mean blood pressure, HR heart rate, SpO 2 peripheral oxygen saturation, min minimal, max maximal

aCut-off value not binarized with receiver operating characteristic curves

Ten variables were thus included in the multivariate model and seven variables were selected for the final prediction model (i.e. all but gender, GCS and vasopressor administration) (Table 3). The model goodness-of-fit was good according to the Hosmer–Lemeshow statistic (p = 0.60). The discrimination as evaluated by the AUC was 0.84 (95% CI 0.82–0.86). As internal validation, optimism was evaluated at 0.001 using bootstrap methodology, so the optimism-corrected AUC was 0.84 (95% CI 0.82–0.85).
Table 3

Results of multivariate stepwise analysis

Pre-hospital criteria

Coefficient

OR

95% CI

P

Shock Index > 1

1.32

3.76

2.96–4.78

< 0.001

Pelvic trauma

1.32

3.76

2.68–5.28

< 0.001

Pre-hospital intubation

0.98

2.67

2.17–3.28

< 0.001

Capillary haemoglobin ≤ 13 g/dl

0.92

2.51

2.05–3.08

< 0.001

MBP ≤ 70 mmHg

0.87

2.38

1.88–3.02

< 0.001

Oxygen saturation minimal ≤ 90%

0.59

1.79

1.35–2.39

< 0.001

Age > 50 years

O.42

1.52

1.21–1.92

< 0.001

Model intercept was −3.33 (p < 0.0001)

OR odds ratio, CI confidence interval, MBP mean blood pressure

Table 4 presents the predictive performances of several combinations of points attributed to each variable remaining in the final prediction model. The combination associated with the optimal trade-off between predictive accuracy (AUC 0.83 (0.81–0.84)), performance (sensitivity 75% (72–79%), specificity 79% (77–80%)) and ease of use was the simple sum of the following five criteria: SI ≥ 1, point of care haemoglobin ≤ 13 g/dl, pre-hospital intubation, MBP minimum ≤ 70 mmHg and clinical signs of unstable pelvic fracture at any time during pre-hospital management. The Red Flag binary alert was considered to be activated if this score, ranging from 0 to 5, was superior or equal to 2 points. Its predictive performances are presented in Table 4. The three-variable combination (threshold 1 point, Table 4) offers less performance (Delong test for ROC curves of paired data, p <  0.001), but seems to be an interesting option especially in a non-physician-staffed EMS (less oro-tracheal intubation and no available pre-hospital haemoglobin measurement).
Table 4

Predictive properties of the various combinations studied to identify the Red Flag binary alert

Shading indicates chosen combination

T threshold, Se sensitivity, Sp specificity, PPV positive predictive value, NPV negative predictive value, +LR positive likelihood ratio, –LR negative likelihood ratio, AUC area under the receiver operating characteristic curve, SI Shock Index, Pelvis unstable pelvis, OTI oro-tracheal intubation, Hb point-of-care haemoglobin, MBP mean blood pressure, SpO 2 peripheral oxygen saturation

External validation

Figure 2 shows the calibration plot of the final prediction model in the 2999 patients of the validation cohort. The agreement between predicted probabilities and observed proportions was adequate, except in the group with a predicted risk of SH from 40 to 50% in which the observed proportion of patients with SH was 32%. The AUC of the final prediction model in this population was 0.80 (95% CI 0.78–0.83), similar to the AUC computed in the derivation cohort (p = 0.19) and significantly higher than the AUC of the MGAP score equal to 0.72 (95% CI 0.69–0.73) (p <  0.001). The predictive performances of the Red Flag binary alert assessed in this population were: sensitivity = 70% (95% CI 66–75), specificity = 80% (95% CI 78–81), PPV = 36% (95% CI 34–38) and NPV = 94% (95% CI 94–95). The contingency mosaics drafted for a threshold of 2 and 3 points are shown in Fig. 3.
Figure 2
Fig. 2

Calibration plot of the model in the validation cohort: agreement between observed and predicted proportion of severe haemorrhage (SH) by the model

Figure 3
Fig. 3

a Red Flag alert. b Contingency mosaic according to threshold of activation. FN false negative, FP false positive, Hb haemoglobin MBP mean arterial blood pressure, OTI Oro-tracheal intubation, SI Shock Index, TN true negative, TP true positive

Discussion

In this study, a Red Flag binary alert derived from an efficient combination of pre-hospital criteria was identified with high predictive performances to detect patients at risk of SH. Its high predictive performances were confirmed in internal and external validations. To our knowledge this is the first report of a validated pre-hospital triggered haemorrhage pre-alert. In practice, the presence of any combination of at least two criteria during the pre-hospital care phase among patients with SI (HR/SBP) ≥ 1, unstable pelvic fracture, intubation, point of care haemoglobin ≤ 13 g/dl or MBP ≤ 70 mmHg activates the Red Flag and provides a powerful signal to initiate an adequate intra-hospital standardized haemorrhage control response (massive transfusion protocol and/or immediate haemostatic procedures).

The criteria identified in this study as associated with SH share similarities with some used in previously described haemorrhage control pathways or massive transfusion scores. Unstable pelvic fracture is part of the TASH score, and part of numerous existing scores predicting ongoing haemorrhage [16, 41]. It is a source of internal bleeding that is difficult to control especially in the case of arterial bleeding (20%) but also in the case of venous bleeding, despite pelvic binding. In the TASH score, unstable pelvic fracture accounts for about 20% of the total score (6 points from 28), as in our study (1 from 5). The Shock Index has been demonstrated as a useful sign to diagnose acute hypovolemia and as a good marker of severe haemorrhagic shock [42]. The threshold used in the Red Flag is 1, while the most frequently suggested SI cut-off value to predict massive transfusion is 0.9 in the literature [43]. Also, the threshold of haemoglobin concentration identified in our study was 1 point higher than the threshold used in the TASH score (13 g/dl vs 12 g/dl) [13]. The timing and the technique of measurement used in our study may explain the difference. Indeed, in the present study, haemoglobin concentration was assessed with a point-of-care technique on scene, thus at a very early stage, whereas the TASH score uses the haemoglobin laboratory concentration at hospital admission. Blood pressure is also a key variable in almost all existing predictive scores for severe haemorrhage [16, 41]. Nevertheless, only SBP is used, while the oscillometric sphygmomanometer, used in many EMSs, measures a MBP and extrapolates SBP and DBP via an algorithm [44]. For this reason, MBP was chosen in the Red Flag and the information carried by this variable was found independent of that carried by the SI. Intubation by the pre-hospital team, however, has never been suggested as being associated with severe haemorrhage in previous studies. In our pre-hospital system, physicians are involved in the pre-hospital setting and this may explain this association as it is usually the most severe patients who are intubated during pre-hospital care [45].

The major advantage of this Red Flag alert is its simplicity of use and pragmatism as it is computed with routinely assessed variables and thus directly available criteria for the pre-hospital care team. It allows the rapid identification of patients who require mobilization of important human and material resources to control haemorrhage (advanced immediate resuscitation and/or haemostatic procedures, early and sustained transfusion, etc.). The predictive performances of the previously described “Code Red” have not yet been extensively evaluated [17, 18]. The other existing simple scores are not based on pre-hospital variables [41], or were built to predict outcome such as mortality [11, 46]. The Red Flag could not be compared to the TASH or ABC score. Those latter scores were validated for an intra-hospital setting and include variables such as ultrasound use or blood gas results, variables that are not systematically available in the pre-hospital environment.

The present work is the first to attempt an extensive assessment of the predictive performance of routine pre-hospital data and to include a validation. A code should be easy to remember and the criteria routinely available; both apply to the Red Flag. Indeed, any prediction tool requires evaluation and validation in the very specific setting it will be implemented in. In the case of an inappropriate activation, the complete set of the haemorrhage control infrastructure may be activated and disorganize programmed care for a while. So, any activation code requires a delicate balance between sensitivity and specificity; that is, between the risk of not activating the haemorrhage control pathway when it is needed (false negative, potentially detrimental to the patient) and over-activation (increase in false positive). On the one hand, it is crucial not to miss any haemorrhagic patients and get activated for their arrival, but, on the other, over-activation can generate waste of precious resources and induces team fatigue leading to further non-compliance. Unjustified activation may even reduce the chance of other patients to benefit from the resources inadequately put on standby. An appropriate number of activations, however, maintains team readiness and training. The clinical consequences of this alert will have to be assessed (times, process, outcomes, etc.).

Our study obviously has some limitations. The first is its retrospective design, as it usually precludes the ad hoc choice of the data studied. It might have been interesting to investigate the contribution of other criteria which were not collected in our study: pre-hospital ultrasonography, described as a bleeding characterization criterion in the ABC score [15], or pre-hospital blood lactate dosage described as a predictor of trauma severity [47]. However, this is actually a strength of our study, as it allowed the analysis of pre-hospital data that are routinely collected in practice by the EMS which reinforces the interest in Red Flag as a pragmatic, easy-to-use tool. Moreover, beside this retrospective analysis, data collection was prospective as this study used data from the Traumabase®, and this has limited data loss and biases inherent to retrospective data collection [48]. Furthermore, this study is the first to evaluate the question in a physician-staffed EMS, whereas existing work has been generated in a paramedic-staffed EMS. Transposition of experiences and data from one system to another can be difficult. The external validity of the study could only be assessed by testing and validating it outside the original centres. The characteristics of these centres, however, are quite different with regard to equipment, internal organization and case mix as there is a lot of contrast in demographic characteristics of the different area covered by each centre within the region. So, results from this study are thus likely to be transposable to other centres. Furthermore, demographic and clinical characteristics of our cohort, as well as mortality, were similar to those in the trauma literature [14, 49]. It is noteworthy that this alert does not apply to penetrating trauma and has not been validated for children. Finally, the impact of this Red Flag alert on patient care has not been evaluated and requires a separate prospective study.

Conclusion

We have constructed and validated a simple Red Flag alert for identifying severe blunt trauma patients during the pre-hospital care phase and activating a specific immediate intra-hospital haemorrhage control response prior to arrival. The impact of its use on severe trauma patient care and on resource utilization remains to be determined.

Notes

Abbreviations

+LR: 

Positive likelihood ratio

AIS: 

Abbreviated Injury Scale

AUC: 

Area under the ROC curve

BMI: 

Body mass index

DBP: 

Diastolic arterial blood pressure

EMS: 

Emergency Medical Service

GCS: 

Glasgow Coma Scale

Hb: 

Haemoglobin

ICU LOS: 

Intensive care unit length of stay

ISS: 

Injury Severity Score

MBP: 

Mean arterial blood pressure

MGAP: 

Mechanism, Glasgow, age and arterial pressure

MVA: 

Motor vehicle accident

NPV: 

Negative predictive value

OTI: 

Oro-tracheal intubation

POC: 

Point of care

PPV: 

Positive predictive value

ROC: 

Receiver operating characteristic

SAPS: 

Simplified Acute Physiology Index

SBP: 

Systolic arterial blood pressure

Se: 

Sensitivity

SH: 

Severe haemorrhage

SI: 

Shock Index

Sp: 

Specificity

SpO2

Peripheral oxygen saturation

TRISS: 

Trauma Injury Severity Score

Declarations

Acknowledgements

The authors are grateful to Alexander Fordyce for his English editing. They also thank Samuel Degoul for his statistical input on using the R software.

*The Traumabase® Group: Arie Attias, MD (Department of Anaesthesiology and Critical Care, Hôpital Henri Mondor, APHP, Créteil, France); Sylvain Ausset, MD (Anaesthesiology and Critical Care, Hôpital Interarmées Percy, Clamart, France); Mathieu Boutonnet, MD (Anaesthesiology and Critical Care, Hôpital Interarmées Percy, Clamart, France); Gilles Dhonneur, MD, PhD (Université Paris Est and Department of Anaesthesiology and Critical Care, Hôpital Henri Mondor, APHP, Créteil, France); Jacques Duranteau, MD, PhD (Université Paris Sud and Department of Anaesthesiology and Critical Care, Hôpital Bicêtre, Groupement Hôpitaux Universitaires Paris Sud, AP-HP, Kremlin Bicêtre, France); Olivier Langeron, MD, PhD (Sorbonne Universités, UPMC Univ Paris 06 and Department of Anaesthesiology and Critical Care, Groupe Hospitalier Pitié-Salpêtrière Charles Foix, AP-HP, Paris, France); Catherine Paugam-Burtz, MD, PhD (Université Denis Diderot and Beaujon University Hospital, Hôpitaux Universitaires Paris Nord-Val-De-Seine, Clichy, AP-HP, France); Romain Pirracchio, MD, PhD (Université Paris Descartes and Department of Anaesthesiology and Critical Care, Hôpital Européen Georges Pompidou, APHP, Paris, France); Bruno Riou, MD, PhD (Sorbonne Université, UMRS 1166, IHU ICAN and Department of Emergency Medicine, Groupe Hospitalier Pitié-Salpêtrière-Charles Foix, AP-HP, Paris, France); Guillaume de St Maurice, MD (Anaesthesiology and Critical Care, Hôpital Interarmées Percy, Clamart, France); Bernard Vigué, MD (Department of Anaesthesiology and Critical Care, Hôpital Bicêtre, Groupement Hôpitaux Universitaires Paris Sud, AP-HP, Kremlin Bicêtre, France).

Funding

The Traumabase® has been sponsored by the Regional Health Agency of Ile de France for 2014–2017.

Availability of data and materials

The datasets generated and analysed during the current study are not publicly available due the restrictions around health data demanded by the Advisory Committee for Information Processing in Health Research and the National Commission on Informatics and Liberties. An extract can be available from the corresponding author on reasonable request and with permission of the aforementioned committees.

Authors’ contributions

SRH contributed to the study design, data collection, data analysis and interpretation, literature search, and writing. ARos contributed to the data collection, data analysis and interpretation, literature search, and writing. TG contributed to study design, data collection, literature search, and writing. J-PD, MR, AH, AF, FC, and MB contributed to data collection and critical revision. JD contributed to study design, critical revision and writing. ARou contributed to the data analysis and interpretation, literature search, and writing. The Traumabase® Group contributed to data collection and critical revision. The Traumabase® registry has provided all data used. All of the authors and the Traumabase® Group are responsible for the scientific content. The final version of the manuscript has been validated by the all of the authors and the Traumabase Group.

Ethics approval and consent to participate

The data come from the Traumabase®. This registry obtained approval from the Advisory Committee for Information Processing in Health Research (CCTIRS, 11.305bis) and from the National Commission on Informatics and Liberties (CNIL, 911461), and the analysis meets the requirement of the local and national ethics committee (Comité de Protection des Personnes, Paris VI).

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Université Paris Sud, Department of Anesthesiology and Critical Care, AP-HP, Bicêtre Hôpitaux Universitaires Paris Sud, Le Kremlin Bicêtre, France
(2)
CESP, INSERM, Université Paris-Sud, UVSQ, Université Paris-Saclay, Paris, Paris, France
(3)
Université Paris Sud, Department of Anesthesiology and Critical Care, Assistance Publique-Hôpitaux de Paris (AP-HP), Béclère Hôpitaux Universitaires Paris Sud, Clamart, France
(4)
Hôpitaux Universitaires Paris Nord Val de Seine, Department of Anesthesiology and Critical Care, AP-HP, Beaujon, Clichy, France
(5)
Centre Hospitalier Sud-Francilien, SAMU 91 Emergency Department, Corbeil-Essonnes, France
(6)
Sorbonne Université and Department of Anesthesiology and Critical Care, AP-HP, Hôpitaux Universitaires Pitié-Salpêtrière, Paris, France
(7)
Université Paris Descartes, Department of Anesthesiology and Critical Care, AP-HP, Hôpital Européen Georges Pompidou—Hôpitaux Universitaires Paris Ouest, Paris, France
(8)
Université Paris Est, Department of Anesthesiology and Critical Care, APHP, Hôpital Henri Mondor, Créteil, France
(9)
Department of Anesthesiology and Critical Care, Percy Military Teaching Hospital, Clamart, France
(10)
AP-HP, Bicêtre Hôpitaux Universitaires Paris Sud, Public Health and Epidemiology Department, AP-HP, Bicêtre Hôpitaux Universitaires Paris Sud, Le Kremlin Bicêtre, France

References

  1. Kauvar DS, Wade CE. The epidemiology and modern management of traumatic hemorrhage: US and international perspectives. Crit Care. 2005;9(Suppl 5):S1–9.View ArticlePubMedPubMed CentralGoogle Scholar
  2. Dutton RP, Stansbury LG, Leone S, Kramer E, Hess JR, Scalea TM. Trauma mortality in mature trauma systems: are we doing better? An analysis of trauma mortality patterns, 1997-2008. J Trauma. 2010;69:620–6.View ArticlePubMedGoogle Scholar
  3. Teixeira PGR, Inaba K, Hadjizacharia P, Brown C, Salim A, Rhee P, et al. Preventable or potentially preventable mortality at a mature trauma center. J Trauma. 2007;63:1338–46.View ArticlePubMedGoogle Scholar
  4. Kauvar DS, Lefering R, Wade CE. Impact of hemorrhage on trauma outcome: an overview of epidemiology, clinical presentations, and therapeutic considerations. J Trauma. 2006;60:S3–11.View ArticlePubMedGoogle Scholar
  5. Gerardo CJ, Glickman SW, Vaslef SN, Chandra A, Pietrobon R, Cairns CB. The rapid impact on mortality rates of a dedicated care team including trauma and emergency physicians at an academic medical center. J Emerg Med. 2011;40:586–91.View ArticlePubMedGoogle Scholar
  6. Cotton BA, Gunter OL, Isbell J, Au BK, Robertson AM, Morris JA, et al. Damage control hematology: the impact of a trauma exsanguination protocol on survival and blood product utilization. J Trauma. 2008;64:1177-1182-1183.View ArticleGoogle Scholar
  7. Glen J, Constanti M, Brohi K, Guideline Development Group. Assessment and initial management of major trauma: summary of NICE guidance. BMJ. 2016;353:i3051.View ArticlePubMedGoogle Scholar
  8. van der Wulp I, van Baar ME, Schrijvers AJP. Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J. 2008;25:431–4.View ArticlePubMedGoogle Scholar
  9. Riou B. Le traumatisé grave. Actualités en réanimation préhospitalière. Journées scientifiques des SAMU de France. In: SFEM; 2002. pp. 115-8.Google Scholar
  10. Brown JB, Lerner EB, Sperry JL, Billiar TR, Peitzman AB, Guyette FX. Prehospital lactate improves accuracy of prehospital criteria for designating trauma activation level. J Trauma Acute Care Surg. 2016;81:445–52.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Sartorius D, Le Manach Y, David J-S, Rancurel E, Smail N, Thicoïpé M, et al. Mechanism, glasgow coma scale, age, and arterial pressure (MGAP): a new simple prehospital triage score to predict mortality in trauma patients. Crit Care Med. 2010;38:831–7.View ArticlePubMedGoogle Scholar
  12. Raux M, Sartorius D, Le Manach Y, David J-S, Riou B, Vivien B. What do prehospital trauma scores predict besides mortality? J Trauma Acute Care Surg. 2011;71:754–9.View ArticleGoogle Scholar
  13. Yücel N, Lefering R, Maegele M, Vorweg M, Tjardes T, Ruchholtz S, et al. Trauma Associated Severe Hemorrhage (TASH)-Score: probability of mass transfusion as surrogate for life threatening hemorrhage after multiple trauma. J Trauma. 2006;60:1228–36.View ArticlePubMedGoogle Scholar
  14. McLaughlin DF, Niles SE, Salinas J, Perkins JG, Cox ED, Wade CE, et al. A predictive model for massive transfusion in combat casualty patients. J Trauma. 2008;64:S57–63. discussion S63View ArticlePubMedGoogle Scholar
  15. Nunez TC, Voskresensky IV, Dossett LA, Shinall R, Dutton WD, Cotton BA. Early prediction of massive transfusion in trauma: simple as ABC (assessment of blood consumption)? J Trauma. 2009;66:346–52.View ArticlePubMedGoogle Scholar
  16. Brockamp T, Nienaber U, Mutschler M, Wafaisade A, Peiniger S, Lefering R, et al. Predicting on-going hemorrhage and transfusion requirement after severe trauma: a validation of six scoring systems and algorithms on the TraumaRegister DGU. Crit Care. 2012;16:R129.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Weaver AE, Hunter-Dunn C, Lyon RM, Lockey D, Krogh CL. The effectiveness of a “Code Red” transfusion request policy initiated by pre-hospital physicians. Injury. 2016;47:3–6.View ArticlePubMedGoogle Scholar
  18. Reed MJ, Glover A, Byrne L, Donald M, McMahon N, Hughes N, et al. Experience of implementing a national pre-hospital Code Red bleeding protocol in Scotland. Injury. 2017;48(1):41–6.View ArticlePubMedGoogle Scholar
  19. Khan S, Allard S, Weaver A, Barber C, Davenport R, Brohi K. A major haemorrhage protocol improves the delivery of blood component therapy and reduces waste in trauma massive transfusion. Injury. 2013;44:587–92.View ArticlePubMedGoogle Scholar
  20. Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1.View ArticlePubMedGoogle Scholar
  21. Hamada SR, Gauss T, Duchateau F-X, Truchot J, Harrois A, Raux M, et al. Evaluation of the performance of French physician-staffed emergency medical service in the triage of major trauma patients. J Trauma Acute Care Surg. 2014;76:1476–83.View ArticlePubMedGoogle Scholar
  22. Mitra B, Cameron PA, Gruen RL, Mori A, Fitzgerald M, Street A. The definition of massive transfusion in trauma: a critical variable in examining evidence for resuscitation. Eur J Emerg Med. 2011;18:137–42.View ArticlePubMedGoogle Scholar
  23. Savage SA, Zarzaur BL, Croce MA, Fabian TC. Redefining massive transfusion when every second counts. J Trauma Acute Care Surg. 2013;74:396-400-402.View ArticleGoogle Scholar
  24. Callaway DW, Shapiro NI, Donnino MW, Baker C, Rosen CL. Serum lactate and base deficit as predictors of mortality in normotensive elderly blunt trauma patients. J Trauma. 2009;66:1040–4.View ArticlePubMedGoogle Scholar
  25. Clarke JR, Trooskin SZ, Doshi PJ, Greenwald L, Mode CJ. Time to laparotomy for intra-abdominal bleeding from trauma does affect survival for delays up to 90 minutes. J Trauma. 2002;52:420–5.PubMedGoogle Scholar
  26. Como JJ, Dutton RP, Scalea TM, Edelman BB, Hess JR. Blood transfusion rates in the care of acute trauma. Transfusion. 2004;44:809–13.View ArticlePubMedGoogle Scholar
  27. Savage SA, Sumislawski JJ, Bell TM, Zarzaur BL. Utilizing group-based trajectory modeling to understand patterns of hemorrhage and resuscitation. Ann Surg. 2016;264:1135–41.View ArticlePubMedGoogle Scholar
  28. Montmany Vioque S, Navarro Soto S, Rebasa Cladera P, Luna Aufroy A, Gómez Díaz C, Llaquet BH. Measurement of lactic acid in multiple injury patients and its usefulness as a predictor of multiorgan failure and mortality. Cir Esp. 2012;90:107–13.View ArticlePubMedGoogle Scholar
  29. Régnier M-A, Raux M, Le Manach Y, Asencio Y, Gaillard J, Devilliers C, et al. Prognostic significance of blood lactate and lactate clearance in trauma patients. Anesthesiology. 2012;117:1276–88.View ArticlePubMedGoogle Scholar
  30. Raux M, Le Manach Y, Gauss T, Baumgarten R, Hamada S, Harrois A, et al. Comparison of the prognostic significance of initial blood lactate and base deficit in trauma patients. Anesthesiology. 2017;126:522–33.View ArticlePubMedGoogle Scholar
  31. Sanchis-Gomar F, Cortell-Ballester J, Pareja-Galeano H, Banfi G, Lippi G. Hemoglobin point-of-care testing: the HemoCue system. J Lab Autom. 2013;18:198–205.View ArticlePubMedGoogle Scholar
  32. Mutschler M, Nienaber U, Münzberg M, Wölfl C, Schoechl H, Paffrath T, et al. The Shock Index revisited—a fast guide to transfusion requirement? A retrospective analysis on 21,853 patients derived from the TraumaRegister DGU®. Crit Care. 2013;17:R172.View ArticlePubMedPubMed CentralGoogle Scholar
  33. Boyd CR, Tolson MA, Copes WS. Evaluating trauma care: the TRISS method. Trauma Score and the Injury Severity Score. J Trauma. 1987;27:370–8.View ArticlePubMedGoogle Scholar
  34. Champion HR, Sacco WJ, Copes WS. Injury severity scoring again. J Trauma. 1995;38:94–5.View ArticlePubMedGoogle Scholar
  35. Raux M, Thicoïpé M, Wiel E, Rancurel E, Savary D, David J-S, et al. Comparison of respiratory rate and peripheral oxygen saturation to assess severity in trauma patients. Intensive Care Med. 2006;32:405–12.View ArticlePubMedGoogle Scholar
  36. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–5.View ArticlePubMedGoogle Scholar
  37. Buuren S, Groothuis-Oudshoorn K. mice: Multivariate imputation by chained equations in R. J Stat Softw. 2011 45. Available from: http://doc.utwente.nl/78938/. [cited 2017 Jun 26].
  38. Hosmer Jr DW, Lemeshow S, Sturdivant RX. Applied logistic regression. Wiley; 2013. Available from: http://books.google.com/books?hl=en&lr=&id=64JYAwAAQBAJ&oi=fnd&pg=PA313&dq=info:QYB4u56LccYJ:scholar.google.com&ots=DrkS603smK&sig=hKtWAl9-_-bIkvzkyftnCdfc-cI [cited 2017 Jun 26].
  39. Steyerberg EW, Harrell FE, Borsboom GJ, Eijkemans MJC, Vergouwe Y, Habbema JDF. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–81.View ArticlePubMedGoogle Scholar
  40. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC bioinformatics. 2011;12:77.View ArticlePubMedPubMed CentralGoogle Scholar
  41. Tonglet ML. Early prediction of ongoing hemorrhage in severe trauma: presentation of the existing scoring systems. Arch Trauma Res. 2016;5(4):e33377.View ArticlePubMedPubMed CentralGoogle Scholar
  42. Moffat B, Vogt KN, Inaba K. The Shock Index: is it ready for primetime? Crit Care. 2013;17:196.View ArticlePubMedPubMed CentralGoogle Scholar
  43. Olaussen A, Blackburn T, Mitra B, Fitzgerald M. Review article: shock index for prediction of critical bleeding post-trauma: a systematic review. Emerg Med Australas. 2014;26:223–8.View ArticlePubMedGoogle Scholar
  44. Meldrum SJ. The principles underlying Dinamap—a microprocessor based instrument for the automatic determination of mean arterial pressure. J Med Eng Technol. 1978;2:243–6.View ArticlePubMedGoogle Scholar
  45. Yeguiayan J-M, Garrigue D, Binquet C, Jacquot C, Duranteau J, Martin C, et al. Medical pre-hospital management reduces mortality in severe blunt trauma: a prospective epidemiological study. Crit Care. 2011;15:R34.View ArticlePubMedPubMed CentralGoogle Scholar
  46. Champion HR, Sacco WJ, Copes WS, Gann DS, Gennarelli TA, Flanagan ME. A revision of the Trauma Score. J Trauma. 1989;29:623–9.View ArticlePubMedGoogle Scholar
  47. Jansen TC, van Bommel J, Mulder PG, Rommes JH, Schieveld SJM, Bakker J. The prognostic value of blood lactate levels relative to that of vital signs in the pre-hospital setting: a pilot study. Crit Care. 2008;12:R160.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Euser AM, Zoccali C, Jager KJ, Dekker FW. Cohort studies: prospective versus retrospective. Nephron Clin Pract. 2009;113:c214–7.View ArticlePubMedGoogle Scholar
  49. MacKenzie EJ, Rivara FP, Jurkovich GJ, Nathens AB, Frey KP, Egleston BL, et al. A national evaluation of the effect of trauma-center care on mortality. N Engl J Med. 2006;354:366–78.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2018

Advertisement