Skip to main content

Quality indicators for patients with traumatic brain injury in European intensive care units: a CENTER-TBI study



The aim of this study is to validate a previously published consensus-based quality indicator set for the management of patients with traumatic brain injury (TBI) at intensive care units (ICUs) in Europe and to study its potential for quality measurement and improvement.


Our analysis was based on 2006 adult patients admitted to 54 ICUs between 2014 and 2018, enrolled in the CENTER-TBI study. Indicator scores were calculated as percentage adherence for structure and process indicators and as event rates or median scores for outcome indicators. Feasibility was quantified by the completeness of the variables. Discriminability was determined by the between-centre variation, estimated with a random effect regression model adjusted for case-mix severity and quantified by the median odds ratio (MOR). Statistical uncertainty of outcome indicators was determined by the median number of events per centre, using a cut-off of 10.


A total of 26/42 indicators could be calculated from the CENTER-TBI database. Most quality indicators proved feasible to obtain with more than 70% completeness. Sub-optimal adherence was found for most quality indicators, ranging from 26 to 93% and 20 to 99% for structure and process indicators. Significant (p < 0.001) between-centre variation was found in seven process and five outcome indicators with MORs ranging from 1.51 to 4.14. Statistical uncertainty of outcome indicators was generally high; five out of seven had less than 10 events per centre.


Overall, nine structures, five processes, but none of the outcome indicators showed potential for quality improvement purposes for TBI patients in the ICU. Future research should focus on implementation efforts and continuous reevaluation of quality indicators.

Trial registration

The core study was registered with, number NCT02210221, registered on August 06, 2014, with Resource Identification Portal (RRID: SCR_015582).


Limited evidence is available to direct critical care practice in patients with traumatic brain injury (TBI) [1]. Randomized controlled trials have shown a limited potential to add evidence translatable to clinical practice, and new approaches are being explored to improve care, such as quality of care monitoring. Quality of care registration in patients with TBI could become part of an emerging international intensive care unit (ICU) or trauma registries [2,3,4,5]. When used over time and across centres, large datasets provide a rich source for benchmarking and quality improvement, i.e. with feedback on performance, between-centre discussions on policies, and opportunities to study best practice.

International registries can contribute to improved patient outcome, by identifying areas in need of quality improvement, informing health policies, and increasing transparency and accountability, as shown in other medical fields, like cancer [6], acute coronary syndrome [7], and cystic fibrosis [8]. Benchmarking TBI management between ICUs can only be reliable when standardized quality indicators are used and case-mix correction is applied [5]. Quality indicators can be subdivided into structure, process, and outcome indicators [9]. As no quality indicator set is available for patients with TBI, we recently performed a Delphi study to reach consensus on a quality indicator set [10].

The aim of the current study is to validate the consensus-based quality indicator set. We hereto analyzed patients enrolled in a large dataset of patients with TBI from the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study. Data collected for CENTER-TBI included a comprehensive description of ICU facilities and patient outcomes in 54 centres, thus providing an opportunity to examine the usefulness of the newly developed indicator set [11]. Based on the validation result, the indicator set could be reduced to those that have the greatest potential for implementation.


Quality indicator set

In this validation study, we applied a previously developed quality indicator set based on a Delphi study to the CENTER-TBI study. The quality indicator set consisted of 17 structure, 16 process, and 9 outcome indicators for adult patients with TBI at the ICU. It was acknowledged that this initial set would be in need of further validation [10].


The CENTER-TBI study is a multicentre observational cohort study conducted in Europe, which recruited patients between 2014 and 2018 ( NCT02210221) [11, 12]. The core study contains 4509 patients. Inclusion criteria for the CENTER-TBI study were a clinical diagnosis of TBI, presentation within 24 h of injury, an indication for CT scanning, and the exclusion criterion was a pre-existing (severe) neurological disorder that could confound outcome assessments. We selected ICU patients for this study as the consensus-based indicators were specifically developed for the ICU. So, the inclusion criteria for our study were (1) admitted to the ICU and (2) adults older than 18 years. Processes of ICU care (vitals, treatments, and therapy intensity levels) were obtained on a daily basis. Outcomes were assessed at the ICU and at 3, 6, 12, and 24 months. In addition, questionnaires were completed by participating centres on structures and processes of care (Provider Profiling questionnaires [13]).

Indicator scores

We determined whether the indicators could be calculated from the CENTER-TBI database and whether data collection fitted routine practice.

Structure indicator scores at centre level were calculated based on the Provider Profiling questionnaires and expressed as the number of centres that indicated that the structure was either present or absent.

Process indicators were calculated as the number of patients adherent to the indicator (numerator) divided by the number of patients to which the indicator could have applied per centre (denominator). The denominator could be based on a subset of patients (e.g. excluding patients with leg fractures for the indicator mechanical DVT prophylaxis).

(Crude) outcome indicators were calculated as the event rate of the indicator per centre (numerator) divided by the total number of patients which could have scored on the indicator (denominator). For the Glasgow Outcome Scale Extended (GOSE) and Short Form-36 version 2 (SF-36), the median scores were calculated.

Missing data were disregarded for the denominator so that the indicator adherence scores were based on the number of patients that could be exposed to the indicator. We present the median indicator numbers across centres with interquartile range.

Validation of the quality indicators

The usefulness of the quality indicators was based on three criteria [14]: feasibility [15], discriminability [16, 17], and statistical uncertainty [15, 18, 19]. As no previous studies report thresholds on these criteria, we set a priori thresholds based on consensus.


Feasibility addresses data quality and ease of quality indicator calculation [15].

The feasibility was quantified by the completeness of the variables required to calculate the indicators. We set an arbitrary threshold of > 70% completeness of data (of denominator) to determine feasibility.


To determine discriminability (between-centre variation), we determined the between-centre differences in adherence to quality indicators to evaluate their potential for benchmarking and quality improvement [16, 17].

Between-centre variation for structure indicators was determined by the number of centres having that structure. We set an arbitrary threshold for moderate discriminability at 80–90% and for poor discriminability at 90–100% adherence to structure and process indicators. Such high levels of adherence decrease discrimination between centres.

The between-centre variation of process and outcome indicator scores, adjusted for case-mix and statistical uncertainty, was quantified with the median odds ratio (MOR) [20]. The MOR represents the odds of being adherent to a specific indicator for two patients with the same patient characteristics from two randomly selected centres. The higher the MOR, the larger the between-centre variation (a MOR equal to 1 reflects no variation).

For process and outcome indicators, we considered a low (unadjusted) interquartile range on scores (IQR < 10) or non-significant (adjusted) between-centre differences or a MOR of 1.1 or less as poor discriminability. Case-mix- and uncertainty-adjusted process and outcome indicator scores per centre were presented in caterpillar plots.

Statistical uncertainty

Reliability refers to the reproducibility of a quality indicator and is threatened by unclear indicator definitions [15] and statistical uncertainty [18, 19]. We determined whether we could calculate indicators in a uniform way or made minor changes to definitions. Statistical uncertainty was determined by random variation due to low numbers of events (only applicable to outcome indicators).

Statistical uncertainty for outcome indicators was determined by the median number of events across centres. We set the threshold for high statistical uncertainty at < 10 events.

Statistical analysis

Baseline centre and patient characteristics are described as frequencies and percentages. Between-centre variation of process and outcome indicator scores was calculated with a random-effect logistic regression analysis. We used a random effect model (random effect for centre) to account for the fact that indicator scores in centres with a small number of patients can have extreme values due to random variation. Also, only centres with > 10 admitted ICU patients were included. To correct for case-mix, we used the extended International Mission for Prognosis and analysis of Clinical Trials in TBI (IMPACT) prognostic model: core (age, motor score, pupillary light reactivity), CT (hypoxia, hypotension, epidural hematoma, traumatic subarachnoid hemorrhage, and Marshall CT classification) and lab (first glucose and hemoglobin) [21], and injury severity score (ISS). The MOR was calculated from the τ2 (variance of random effects).

Case-mix- and uncertainty-adjusted process and outcome indicator scores per centre are presented in ‘caterpillar’ plots. p values for determining the significance of the between-centre variation were calculated with a likelihood ratio test comparing a model with and without a random effect for centre. A mixture distribution is required to calculate the p value as the null hypothesis is on the boundary of the parameter space [22].

For the calculation of random effect models, missing data were imputed with multiple (N = 5) imputation with the MICE package from R [23]. Statistical analyses were performed in R statistical software. Neurobot version 2.1 (data extraction date 23-12-2019) was used.


A total of 26 (11 structure, 8 process, and 7 outcome indicators) of the 42 indicators of the Delphi set could be extracted from the CENTER-TBI database. (Additional file 1).

Baseline data

Fifty-four centres from 18 countries were included, totaling 2006 adult patients. The median number of ICU patients included per centre was 23 (IQR12–43, range 2–119). Centres were mostly academic centres (N = 51; 94%) and designated as level I trauma centres (N = 37; 69%). Most centres were located in Northern (N = 20; 37%) or Western Europe (N = 19; 35%) (Table 1).

Table 1 Baseline centre and patient characteristics

Around 28% of patients admitted to ICU were older than 65 years and mostly male (N = 1561; 73%). According to the baseline GCS score, 48% had severe (GCS < 9; N = 915), 16% moderate (GCS 9–12; N = 305), and 48% mild TBI (GCS 13–15; N = 671). The majority of patients (N = 1963; 96%) suffered from polytrauma. The cause of injury was mostly related to road traffic accidents (N = 849; 44%) or incidental falls (N = 802; 42%) (Table 1).


Regarding structure indicators, sub-optimal adherence rates were found for most indicators, including the presence of a neuro-ICU (N = 35; 65%), operation room availability 24 h per day (N = 40; 75%), and presence of a step-down unit (N = 38; 70%) (Additional file 2). Patient-to-nurse ratio’s varied, with reported ratios of 1 (N = 14; 26%), 1–2 (N = 23; 43%), and 2–3 (N = 17; 31%) patients per nurse. Adherence was high for ‘the existence of a protocol including specific guidelines’ (N = 47; 89%), ‘protocol for glucose management’ (N = 43; 81%), ‘the availability of a neurosurgeon within 30 minutes after call’ (N = 49; 93%), and ‘the 24/7 availability of a CT scan and radiologist review’ (N = 50; 91%).

Sub-optimal adherence rates were found for most process indicators, including ICP monitoring in the severe TBI group (median 69%, IQR 44–82), basal caloric intake within 5–7 days (N = 20%, IQR 3–47), and ‘patients that receive DVT prophylaxis with low molecular weight heparins’ (median 63%, IQR 49–78) (Additional file 3). Adherence was high for ‘enteral nutrition within 72 hours’ (median 99%, IQR 87–100).

For outcome, the centres had a median [IQR] ICU mortality of 12% [9–21], ventilator-acquired pneumonia (VAP) incidence of 14% [0–31], and hyperglycemia incidence of 35% [22–45]. The median [IQR] GOSE was 5 [3–7], the SF-36v2 physical component summary (PCS) 46 [37–54], and SF-36v2 mental component summary (MCS) was 46 [36–55] (Additional file 4).


Feasibility of structure indicators was generally high (overall more than 98% available data). Feasibility was low for one process indicator: ‘mechanical DVT prophylaxis within 24 hours’ (43% available data). Feasibility was high for outcome indicators, except for the SF-36 MCS and PCS scores (28% available data) collected after 6 months (due to loss to follow-up) (Additional files 2, 3, 4).

Overall, one process and one outcome indicator showed low feasibility (Table 2).

Table 2 Overview of indicator performance


Variation in scores between centres was low for structure indicators (with little room for improvement) for ‘existence of a protocol’, ‘availability of a neurosurgeon 24/7 within 30 minutes after call’, and ‘24/7 availability of a CT scan and radiologist review’, due to high overall adherence rates among centres (Additional file 2). For process indicators, high variation was found for all indicators (all MORs above 1.5, all p < 0.001) except for ‘surgery within 4 hours in patients with SDH or EDH’ (Fig. 1).

Fig. 1
figure 1

Adjusted random effect estimates per centre for process indicators. This figure shows the between-centre differences for the process indicators (beware of different x-axes). Quality indicator definitions can be found in Additional file 3. On the y-axis, each dot represents a centre. A centre with an average indicator score has log odds 0 (a positive log odds indicates higher indicator scores and a negative log odds lower indicator scores). The between-centre differences are represented by the shape of the caterpillar plots; the variation in the log odds for individual centres and the corresponding confidence intervals (uncertainty). For example, the use of ICP monitoring shows large variation between centres with small confidence intervals, so there is high variation with low statistical uncertainty. While for use of low molecular weight heparin, the variation is large, but the statistical uncertainty is high as well (due to high adherence rates for most centres). The caterpillars were based on non-missing data (after imputation). ‘Use of Low Molecular Weight Heparin’ reflects the indicator ‘Number of patients that receive pharmaceutical prophylaxis with low molecular weight heparins/ total number of TBI patients admitted to the ICU’. ‘Surgery within 4 hours’ reflects the indicator ‘Median door-to-operation time for acute operation of SDH and EDH with surgical indication’. DVT deep venous thrombosis, EDH epidural hematoma, ICU intensive care unit, MOR median odds ratio, SDH subdural hematoma

For outcome indicators, the between-centre variation was significant as well. The variation between centres was especially high for ventilator-acquired pneumonia (VAP) with a MOR of 4.12. Little between-centre variation on the 6-month GOSE was found (MOR = 1.29, p = 0.5) (Fig. 2).

Fig. 2
figure 2

Adjusted random effect estimates per centre for outcome indicators. This figure shows the between-centre differences for the outcome indicators. Quality indicator definitions can be found in Additional file 4. On the y-axis, each dot represents a centre. A centre with an average indicator score has log odds 0 (a positive log odds indicates higher indicator scores and a negative log odds a lower indicator scores). Outcome indicator scores were adjusted for case-mix and ‘statistical uncertainty’ (variation by chance) by using a random effects logistic regression model. The MOR (median odds ratio) represents the between-centre variation: the higher the MOR, the larger the between-centre variation (a MOR equal to 1 reflects no variation). The confidence intervals represent the statistical uncertainty. The caterpillars were based on non-missing data (after imputation). Outcome incidence for decubitus and hypoglycemia was too low to reliably show between-centre variation (high confidence intervals). Impaired SF-36v2 (PCS or MCS) score ≤ 40. CI confidence interval, GOSE Glasgow Outcome Scale Extended, ICU intensive care unit, MOR median odds ratio

Overall, five structure (three with moderate performance), two process, and four outcome indicators showed low discriminability (Table 2).

Statistical uncertainty

Four indicator definitions were slightly changed without changing its content (Additional files 3 and 4, bold definitions). Median event rates for the outcome indicators hyperglycemia, ICU mortality, and ventilator-associated pneumonia (VAP) were respectively 8, 4, and 3 events per centre. Median event rates for hypoglycemia and decubitus were zero. All these event rates reflect high statistical uncertainty (Additional file 4, Table 2).


We showed that it was feasible to obtain most quality indicators from a recently proposed, consensus-based, quality indicator set for traumatic brain injury (TBI) at the ICU based on sufficient data completeness. The suboptimal adherence scores in combination with between-centre variation suggest a potential for quality improvement, specifically for process and outcome indicators. However, statistical uncertainty was generally high for outcome indicators, making them less suitable for quality improvement purposes and benchmarking in particular. Based on the assessment of feasibility, discriminability, and statistical uncertainty, we found nine structure indicators, five process indicators, but none of the outcome indicator out of 26 indicators to be appropriate for quality measurement and improvement in this validation study. Overall, the quality of ICU care can be improved for patients with TBI, and our analysis provides a useful case of how quality indicators for ICU care in TBI can be evaluated in a large database.

To our knowledge, this is the first quality indicator set to be developed and validated in adult patients with TBI admitted to the ICU. We have summarized quality indicators with the potential to be used for benchmarking and quality improvement. First, we recommend reducing the initial set by excluding indicators with a low percentage available data (low feasibility), in a given dataset. The low feasibility on some process indicators might be explained by the complexity and high resource needs of collecting data on process indicators. However, feasibility could be improved with automatic data extraction in the future. Second, quality indicators with high between-centre variation (most quality indicators in this study) and suboptimal adherence rates (discriminability) can be used to improve quality of care and for benchmarking. Third, event rates of outcome indicators were generally low (even over a study duration of 4 years), indicating that outcome indicators have a low potential for quality improvement in this study population due to high statistical uncertainty. However, the threshold of 10 events might be too strict, or alternatively, outcome indicator denominators should be restricted to patients with a more severe injury, greater organ dysfunction, more interventions, or longer length of stay to increase the number of events and to increase statistical power. Over time, registration and use of the quality indicators could provide further insights into their role in quality improvement and benchmarking and allow their re-evaluation and refinement.

Quality of care in critically ill patients with TBI could potentially be improved in various areas, as indicated by a sub-optimal adherence of European ICUs to most quality indicators. The large (adjusted) between-centre variation suggests that some centres significantly outperform others. Wide sharing of best practice and implementation strategies from centres that perform well on quality indicators describing structures and processes of care and/or registering a low incidence of adverse outcomes could improve performance in centres that perform less well.

Previous studies also report large between-centre differences in processes of TBI care across Europe [24,25,26]. This between-centre variation could be explained by variation in adherence to guidelines. Although 89% of centres indicated that they complied with the Brain Trauma Foundation (BTF) guidelines, actual assessment of real-time practice may be different. For example, ICP monitoring in patients with severe TBI (GCS < 9) is one of the higher-level evidence recommendations in the BTF guidelines, but we only found adherence rates of 44–82% (IQR) across centres in our study. This implies that there is much to gain in the reduction of variation in evidence-based care processes. One previous study reported the performance of quality indicators in children with TBI [27]. Although their indicators differed from those in the current study, they found a lower variation in adherence rates (between 68% and 78%). Several registries already exist for general ICU [3, 5]—or trauma care [2, 4]. Some of the outcome indicators we tested are also used in current ICU registries but did not perform well in our study (decubitus ulcers and hypoglycemia). For example, in our study, the outcome score for decubitus ulcers approached 0%, while in Dutch hospitals, decubitus was found in around 6% of patients [16].

This study has several strengths. First, we tested the potential of consensus-based quality indicators in a large clinical dataset, while most previous studies only report a Delphi study to develop quality indicators and only a few studies pilot-tested quality indicators before implementation [28, 29]. Second, the indicator scores were derived from the CENTER-TBI database, which includes a substantial number of patients with TBI across many ICUs. Indeed, this analysis provides the first opportunity to study indicator performance and between-centre variation in TBI management on a larger scale. The CENTER-TBI database has only one exclusion criterion, so it represents a cohort generalizable to the TBI population across Europe.

Our study also has some limitations. Staffing and organizational data were only partly captured in CENTER-TBI. The structure indicators were based on questionnaires which might be imprecise. Patients of all severities (including early deaths) were included for analyses. We recognize that a selection of patients with a longer ICU stay may have increased between-centre comparability, but we mitigated this issue by correcting all between-centre analyses for case-mix severity. We defined feasibility as the completeness of the data, while other aspects of feasibility, such as accessibility, timeliness, and missing data at a centre level, could not be addressed [30]. Statistical uncertainty was reflected in the number of event rates, while also other aspects as intra- and inter rater reliability of medical coders are important but could not be addressed. We decided not to test the construct (correlations between indicators) and criterion validity (association with outcome) of the final indicator set as these are hard to test [31]; for construct validity, predetermined correlations between quality indicators are hard to find between different aspects of processes of care and often do not correlate with outcome; and for criterion validity, the case-mix adjustment would differ per quality indicator and even very complex models cannot adjust for all residual bias (unmeasured confounding). However, ongoing evaluation of these quality indicators in larger datasets could include assessment of such correlations with the outcome.

Future implementation of the quality indicators in a European registry will make it possible to monitor TBI patient data over time and among countries. Feedback from this registry to individual ICUs is essential to make stakeholders be aware of their centre performance and help develop internal quality improvement programmes. No reference standards for the quality indicators have been defined. Our study also illustrates some pitfalls, since some of these indicators are quite complex and difficult to assess retrospectively. Such data collection could, however, be optimized by routine registration of timing of events and processes, automatic data extraction, and clear definitions. Overall, the methods illustrated in this study can be used to optimize future data collection (with uniform indicator definitions and data quality), to calculate quality indicators (adjusted across centres) and to identify areas in need of further research (due to high variation).


This study validated a consensus-base quality indicator set in a large prospective TBI study (CENTER-TBI). Quality of care in critically ill patients with TBI appears amenable to improvement in various areas as indicated by sub-optimal adherence rates and between-centre variation for many quality indicators. Further, our analysis generally shows good feasibility and discriminability but high statistical uncertainty for several outcome indicators. Future research should focus on implementation and quality improvement efforts and continuous reevaluation of the quality indicators.

Availability of data and materials

The datasets generated and/or analyzed during the current study are available via on reasonable request.



Brain trauma foundation


Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury


Computed tomography


Deep venous thrombosis


Epidural hematoma


Glasgow Coma Scale


Glasgow Outcome Scale Extended


Intracranial pressure


Intensive care unit


International Mission for Prognosis and Analysis of Clinical Trials in traumatic brain injury


Interquartile range


Injury severity score


Median odds ratio


Subdural hematoma


Short-Form36 version 2


Ventilator-acquired pneumonia


  1. Maas AIR, Menon DK, Adelson PD, Andelic N, Bell MJ, Belli A, Bragge P, Brazinova A, Buki A, Chesnut RM, et al. Traumatic brain injury: integrated approaches to improve prevention, clinical care, and research. Lancet Neurol. 2017;16(12):987–1048.

    Article  Google Scholar 

  2. Haider AH, Hashmi ZG, Gupta S, Zafar SN, David JS, Efron DT, Stevens KA, Zafar H, Schneider EB, Voiglio E, et al. Benchmarking of trauma care worldwide: the potential value of an International Trauma Data Bank (ITDB). World J Surg. 2014;38(8):1882–91.

    Article  Google Scholar 

  3. Rhodes A, Moreno RP, Azoulay E, Capuzzo M, Chiche JD, Eddleston J, Endacott R, Ferdinande P, Flaatten H, Guidet B, et al. Prospectively defined indicators to improve the safety and quality of care for critically ill patients: a report from the Task Force on Safety and Quality of the European Society of Intensive Care Medicine (ESICM). Intensive Care Med. 2012;38(4):598–605.

    Article  CAS  Google Scholar 

  4. O'Reilly GM, Cameron PA, Joshipura M. Global trauma registry mapping: a scoping review. Injury. 2012;43(7):1148–53.

    Article  Google Scholar 

  5. Salluh JIF, Soares M, Keegan MT. Understanding intensive care unit benchmarking. Intensive Care Med. 2017;43(11):1703–7.

    Article  Google Scholar 

  6. Coleman MP, Forman D, Bryant H, Butler J, Rachet B, Maringe C, Nur U, Tracey E, Coory M, Hatcher J, et al. Cancer survival in Australia, Canada, Denmark, Norway, Sweden, and the UK, 1995-2007 (the International Cancer Benchmarking Partnership): an analysis of population-based cancer registry data. Lancet. 2011;377(9760):127–38.

    Article  CAS  Google Scholar 

  7. Fox KA, Fitzgerald G, Puymirat E, Huang W, Carruthers K, Simon T, Coste P, Monsegu J, Gabriel Steg P, Danchin N, et al. Should patients with acute coronary disease be stratified for management according to their risk? Derivation, external validation and outcomes using the updated GRACE risk score. BMJ Open. 2014;4(2):e004425.

    Article  Google Scholar 

  8. Padman R, McColley SA, Miller DP, Konstan MW, Morgan WJ, Schechter MS, Ren CL, Wagener JS, Investigators, Coordinators of the Epidemiologic Study of Cystic F. Infant care patterns at epidemiologic study of cystic fibrosis sites that achieve superior childhood lung function. Pediatrics. 2007;119(3):e531–7.

    Article  Google Scholar 

  9. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743–8.

    Article  CAS  Google Scholar 

  10. Volovici V, Ercole A, Citerio G, Stocchetti N, Haitsma IK, Huijben JA, Dirven CMF, van der Jagt M, Steyerberg EW, Nelson D, et al. Intensive care admission criteria for traumatic brain injury patients across Europe. J Crit Care. 2019;49:158–61.

    Article  Google Scholar 

  11. Maas AI, Menon DK, Steyerberg EW, Citerio G, Lecky F, Manley GT, Hill S, Legrand V, Sorgner A, Participants C-T, et al. Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI): a prospective longitudinal observational study. Neurosurgery. 2015;76(1):67–80.

    Article  Google Scholar 

  12. Steyerberg EW et al. Case-mix, care pathways, and outcomes in patients with traumatic brain injury in CENTER-TBI: a European prospective, multicentre, longitudinal, cohort study. Lancet neurol. 2019;18(10):923-34.

  13. Cnossen MC, Polinder S, Lingsma HF, Maas AI, Menon D, Steyerberg EW, Investigators C-T, Participants. Variation in structure and process of care in traumatic brain injury: provider profiles of European Neurotrauma Centers participating in the CENTER-TBI study. PLoS One. 2016;11(8):e0161367.

    Article  Google Scholar 

  14. Fischer C. Quality indcators for hospital care; 2015.

    Google Scholar 

  15. Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inform Assoc. 2002;9(6):600–11.

    Article  Google Scholar 

  16. van Dishoeck AM, Looman CW, van der Wilden-van Lier EC, Mackenbach JP, Steyerberg EW. Displaying random variation in comparing hospital performance. BMJ Qual Saf. 2011;20(8):651–7.

    Article  Google Scholar 

  17. van Overveld LFJ, Takes RP, Braspenning JCC, Baatenburg de Jong RJ, de Boer JP, Brouns JJA, Bun RJ, Dik EA, van Dijk BAC, van Es RJJ. Variation in integrated head and neck cancer care: impact of patient and hospital characteristics. J Natl Compr Cancer Netw. 2018;16(12):1491–8.

  18. Lingsma HF, Steyerberg EW, Eijkemans MJ, Dippel DW, Scholte Op Reimer WJ, Van Houwelingen HC, Netherlands Stroke Survey I. Comparing and ranking hospitals based on outcome: results from The Netherlands Stroke Survey. QJM. 2010;103(2):99–108.

    Article  CAS  Google Scholar 

  19. van Dishoeck AM, Lingsma HF, Mackenbach JP, Steyerberg EW. Random variation and rankability of hospitals using outcome indicators. BMJ Qual Saf. 2011;20(10):869–74.

    Article  Google Scholar 

  20. Merlo J, Chaix B, Ohlsson H, Beckman A, Johnell K, Hjerpe P, Rastam L, Larsen K. A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health. 2006;60(4):290–7.

    Article  Google Scholar 

  21. Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JD, et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 2008;5(8):e165 discussion e165.

    Article  Google Scholar 

  22. Stram DO, Lee JW. Variance components testing in the longitudinal mixed effects model. Biometrics. 1994;50(4):1171–7.

    Article  CAS  Google Scholar 

  23. Buuren S. van, Groothuis-Oudshoorn K. Mice: Multivariate Imputation by Chained Equations in R. J Stat Software 45. 2011. p. 1-68.

  24. Cnossen MC, Polinder S, Andriessen TM, van der Naalt J, Haitsma I, Horn J, Franschman G, Vos PE, Steyerberg EW, Lingsma H. Causes and consequences of treatment variation in moderate and severe traumatic brain injury: a multicenter study. Crit Care Med. 2017;45(4):660–9.

    Article  Google Scholar 

  25. Bulger EM, Nathens AB, Rivara FP, Moore M, MacKenzie EJ, Jurkovich GJ, Brain Trauma Foundation. Management of severe head injury: institutional variations in care and effect on outcome. Crit Care Med. 2002;30(8):1870–6.

  26. Alali AS, Fowler RA, Mainprize TG, Scales DC, Kiss A, de Mestral C, Ray JG, Nathens AB. Intracranial pressure monitoring in severe traumatic brain injury: results from the American College of Surgeons Trauma Quality Improvement Program. J Neurotrauma. 2013;30(20):1737–46.

    Article  Google Scholar 

  27. Vavilala MS, Kernic MA, Wang J, Kannan N, Mink RB, Wainwright MS, Groner JI, Bell MJ, Giza CC, Zatzick DF, et al. Acute care clinical indicators associated with discharge outcomes in children with severe traumatic brain injury. Crit Care Med. 2014;42(10):2258–66.

    Article  Google Scholar 

  28. de Carvalho AGR, de Moraes APP, Tanaka LMS, Gomes RV, da Silva AAM. Quality in intensive care units: proposal of an assessment instrument. BMC Res Notes. 2017;10(1):222.

    Article  Google Scholar 

  29. Najjar-Pellet J, Jonquet O, Jambou P, Fabry J. Quality assessment in intensive care units: proposal for a scoring system in terms of structure and process. Intensive Care Med. 2008;34(2):278–85.

    Article  Google Scholar 

  30. Cai L, Zhu Y. The challenges of data quality and data quality assessment in the big data era. Data Sci J. 2015;14:2.

    Article  Google Scholar 

  31. Stocchetti N, Penny KI, Dearden M, Braakman R, Cohadon F, Iannotti F, Lapierre F, Karimi A, Maas A Jr, Murray GD, et al. Intensive care management of head-injured patients in Europe: a survey from the European brain injury consortium. Intensive Care Med. 2001;27(2):400–6.

    Article  CAS  Google Scholar 

  32. Ethical approval in CENTER-TBI;, 397. Accessed 24 Oct 2019.

Download references


First, the authors would like to thank the patients for their participation in the CENTER-TBI study. Also, we thank all principal investigators and researchers of the CENTER-TBI study and for sharing their valuable expertise and for data collection in the ICU stratum (collaboration group, Additional file 5).

Collaboration group: Cecilia Åkerlund1, Krisztina Amrein 2, Nada Andelic3, Lasse Andreassen4, Gérard Audibert5, Philippe Azouvi6, Maria Luisa Azzolini7, Ronald Bartels8, Ronny Beer9, Bo-Michael Bellander10, Habib Benali11, Maurizio Berardino12, Luigi Beretta7, Erta Beqiri13, Morten Blaabjerg14, Stine Borgen Lund15, Camilla Brorsson16, Andras Buki17, Manuel Cabeleira18, Alessio Caccioppola19, Emiliana Calappi19, Maria Rosa Calvi7, Peter Cameron20, Guillermo Carbayo Lozano21, Marco Carbonara19, Ana M. Castaño-León22, Simona Cavallo12, Giorgio Chevallard13, Arturo Chieregato13, Mark Coburn24, Jonathan Coles25, Jamie D. Cooper26, Marta Correia27, Endre Czeiter17, Marek Czosnyka18, Claire Dahyot-Fizelier28, Paul Dark29, Véronique De Keyser30, Vincent Degos11, Francesco Della Corte31, Hugo den Boogert8, Bart Depreitere32, Dula Dilvesi33, Abhishek Dixit34, Jens Dreier35, Guy-Loup Dulière36, Erzsébet Ezer37, Martin Fabricius38, Kelly Foks39, Shirin Frisvold40, Alex Furmanov41, Damien Galanaud11, Dashiell Gantner20, Alexandre Ghuysen42, Lelde Giga43, Jagos Golubovic33, Pedro A. Gomez22, Francesca Grossi31, Deepak Gupta44, Iain Haitsma45, Raimund Helbok9, Eirik Helseth46, Peter J. Hutchinson47, Stefan Jankowski48, Faye Johnson49, Mladen Karan33, Angelos G. Kolias47, Daniel Kondziella38, Evgenios Koraropoulos34, Lars-Owe Koskinen50, Noémi Kovács51, Ana Kowark24, Alfonso Lagares22, Steven Laureys52, Fiona Lecky53,54, Didier Ledoux52, Aurelie Lejeune55, Roger Lightfoot56, Alex Manara58, Costanza Martino59, Hugues Maréchal36, Julia Mattern60, Catherine McMahon61, Tomas Menovsky30, Benoit Misset52, Visakh Muraleedharan62, Lynnette Murray20, Ancuta Negru63, David Nelson1, Virginia Newcombe34, József Nyirádi2, Fabrizio Ortolano19, Jean-François Payen64, Vincent Perlbarg11, Paolo Persona65, Wilco Peul66, Anna Piippo-Karjalainen67, Horia Ples63, Inigo Pomposo21, Jussi P. Posti68, Louis Puybasset69, Andreea Radoi70, Arminas Ragauskas71, Rahul Raj67, Jonathan Rhodes72, Sophie Richter34, Saulius Rocka71, Cecilie Roe73, Olav Roise74,75, Jeffrey V. Rosenfeld76, Christina Rosenlund77, Guy Rosenthal41, Rolf Rossaint24, Sandra Rossi65, Juan Sahuquillo70, Oddrun Sandrød79, Oliver Sakowitz60, 79, Renan Sanchez-Porras79, Kari Schirmer-Mikalsen78, 80, Rico Frederik Schou81, Peter Smielewski18, Abayomi Sorinola82, Emmanuel Stamatakis34 Nino Stocchetti83, Nina Sundström84, Riikka Takala85, Viktória Tamás82, Tomas Tamosuitis86, Olli Tenovuo68, Matt Thomas58, Dick Tibboel77, Christos Tolias88, Tony Trapani19, Cristina Maria Tudora63, Peter Vajkoczy79, Shirley Vallance20, Egils Valeinis 43, Zoltán Vámos37, Gregory Van der Steen30, Jeroen T.J.M. van Dijck66, Thomas A. van Essen66, Roel P. J. van Wijk66, Alessia Vargiolu23, Emmanuel Vega55, Anne Vik80, 90, Rimantas Vilcinis86, Victor Volovici45, Daphne Voormolen57, Petar Vulekovic33, Guy Williams34, Stefan Winzeck34, Stefan Wolf91, Alexander Younsi60, Frederick A. Zeiler34,92, Agate Ziverte43, Tommaso Zoerle19, Hans Clusmann93.

1 Department of Physiology and Pharmacology, Section of Perioperative Medicine and Intensive Care, Karolinska Institutet, Stockholm, Sweden

2 János Szentágothai Research Centre, University of Pécs, Pécs, Hungary

3 Division of Surgery and Clinical Neuroscience, Department of Physical Medicine and Rehabilitation, Oslo University Hospital and University of Oslo, Oslo, Norway

4 Department of Neurosurgery, University Hospital Northern Norway, Tromso, Norway

5 Department of Anesthesiology & Intensive Care, University Hospital Nancy, Nancy, France

6 Raymond Poincare hospital, Assistance Publique – Hopitaux de Paris, Paris, France

7 Department of Anesthesiology & Intensive Care, S Raffaele University Hospital, Milan, Italy.

8 Department of Neurosurgery, Radboud University Medical Center, Nijmegen, The Netherlands

9 Department of Neurology, Neurological Intensive Care Unit, Medical University of Innsbruck, Innsbruck, Austria

10 Department of Neurosurgery & Anesthesia & intensive care medicine, Karolinska University Hospital, Stockholm, Sweden

11 Anesthesie-Réanimation, Assistance Publique – Hopitaux de Paris, Paris, France

12 Department of Anesthesia & ICU, AOU Città della Salute e della Scienza di Torino - Orthopedic and Trauma Center, Torino, Italy

13 NeuroIntensive Care, Niguarda Hospital, Milan, Italy

14 Department of Neurology, Odense University Hospital, Odense, Denmark

15 Department of Public Health and Nursing, Faculty of Medicine and health Sciences, Norwegian University of Science and Technology, NTNU, Trondheim, Norway

16 Department of Surgery and Perioperative Science, Umeå University, Umeå, Sweden

17 Department of Neurosurgery, Medical School, University of Pécs, Hungary and Neurotrauma Research Group, János Szentágothai Research Centre, University of Pécs, Hungary

18 Brain Physics Lab, Division of Neurosurgery, Dept of Clinical Neurosciences, University of Cambridge, Addenbrooke’s Hospital, Cambridge, UK

19 Neuro ICU, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy

20 ANZIC Research Centre, Monash University, Department of Epidemiology and Preventive Medicine, Melbourne, Victoria, Australia

21 Department of Neurosurgery, Hospital of Cruces, Bilbao, Spain

22 Department of Neurosurgery, Hospital Universitario 12 de Octubre, Madrid, Spain

23 NeuroIntensive Care, ASST di Monza, Monza, Italy

24 Department of Anaesthesiology, University Hospital of Aachen, Aachen, Germany

25 Department of Anesthesia & Neurointensive Care, Cambridge University Hospital NHS Foundation Trust, Cambridge, UK

26 School of Public Health & PM, Monash University and The Alfred Hospital, Melbourne, Victoria, Australia

27 Radiology/MRI department, MRC Cognition and Brain Sciences Unit, Cambridge, UK.

28 Intensive Care Unit, CHU Poitiers, Potiers, France

29 University of Manchester NIHR Biomedical Research Centre, Critical Care Directorate, Salford Royal Hospital NHS Foundation Trust, Salford, UK.

30 Department of Neurosurgery, Antwerp University Hospital and University of Antwerp, Edegem, Belgium

31 Department of Anesthesia & Intensive Care, Maggiore Della Carità Hospital, Novara, Italy

32 Department of Neurosurgery, University Hospitals Leuven, Leuven, Belgium

33 Department of Neurosurgery, Clinical centre of Vojvodina, Faculty of Medicine, University of Novi Sad, Novi Sad, Serbia

34 Division of Anaesthesia, University of Cambridge, Addenbrooke’s Hospital, Cambridge, UK

35 Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany

36 Intensive Care Unit, CHR Citadelle, Liège, Belgium

37 Department of Anaesthesiology and Intensive Therapy, University of Pécs, Pécs, Hungary

38 Departments of Neurology, Clinical Neurophysiology and Neuroanesthesiology, Region Hovedstaden Rigshospitalet, Copenhagen, Denmark

39 Department of Neurology, Erasmus MC, Rotterdam, the Netherlands

40 Department of Anesthesiology and Intensive care, University Hospital Northern Norway, Tromso, Norway

41 Department of Neurosurgery, Hadassah-hebrew University Medical center, Jerusalem, Israel.

42 Emergency Department, CHU, Liège, Belgium

43 Neurosurgery clinic, Pauls Stradins Clinical University Hospital, Riga, Latvia

44 Department of Neurosurgery, Neurosciences Centre & JPN Apex trauma centre, All India Institute of Medical Sciences, New Delhi-110029, India

45 Department of Neurosurgery, Erasmus MC, Rotterdam, the Netherlands

46 Department of Neurosurgery, Oslo University Hospital, Oslo, Norway

47 Division of Neurosurgery, Department of Clinical Neurosciences, Addenbrooke’s Hospital & University of Cambridge, Cambridge, UK

48 Neurointensive Care, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK

49 Salford Royal Hospital NHS Foundation Trust Acute Research Delivery Team, Salford, UK

50 Department of Clinical Neuroscience, Neurosurgery, Umeå University, Umeå, Sweden

51 Hungarian Brain Research Program - Grant No. KTIA_13_NAP-A-II/8, University of Pécs, Pécs, Hungary

52 Cyclotron Research Center, University of Liège, Liège, Belgium

53 Centre for Urgent and Emergency Care Research (CURE), Health Services Research Section, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

54 Emergency Department, Salford Royal Hospital, Salford UK

55 Department of Anesthesiology-Intensive Care, Lille University Hospital, Lille, France

56 Department of Anesthesiology & Intensive Care, University Hospitals Southhampton NHS Trust, Southhampton, UK

57 Department of Public Health, Erasmus Medical Center-University Medical Center, Rotterdam, The Netherlands

58 Intensive Care Unit, Southmead Hospital, Bristol, Bristol, UK

59 Department of Anesthesia & Intensive Care,M. Bufalini Hospital, Cesena, Italy

60 Department of Neurosurgery, University Hospital Heidelberg, Heidelberg, Germany

61 Department of Neurosurgery, The Walton centre NHS Foundation Trust, Liverpool, UK

62 Karolinska Institutet, INCF International Neuroinformatics Coordinating Facility, Stockholm, Sweden

63 Department of Neurosurgery, Emergency County Hospital Timisoara, Timisoara, Romania

64 Department of Anesthesiology & Intensive Care, University Hospital of Grenoble, Grenoble, France

65 Department of Anesthesia & Intensive Care, Azienda Ospedaliera Università di Padova, Padova, Italy

66 Dept. of Neurosurgery, Leiden University Medical Center, Leiden, The Netherlands and Dept. of Neurosurgery, Medical Center Haaglanden, The Hague, The Netherlands

67 Department of Neurosurgery, Helsinki University Central Hospital

68 Division of Clinical Neurosciences, Department of Neurosurgery and Turku Brain Injury Centre, Turku University Hospital and University of Turku, Turku, Finland

69 Department of Anesthesiology and Critical Care, Pitié -Salpêtrière Teaching Hospital, Assistance Publique, Hôpitaux de Paris and University Pierre et Marie Curie, Paris, France

70 Neurotraumatology and Neurosurgery Research Unit (UNINN), Vall d’Hebron Research Institute, Barcelona, Spain

71 Department of Neurosurgery, Kaunas University of technology and Vilnius University, Vilnius, Lithuania

72 Department of Anaesthesia, Critical Care & Pain Medicine NHS Lothian & University of Edinburg, Edinburgh, UK

73 Department of Physical Medicine and Rehabilitation, Oslo University Hospital/University of Oslo, Oslo, Norway

74 Division of Orthopedics, Oslo University Hospital, Oslo, Norway

75 Institute of Clinical Medicine, Faculty of Medicine, University of Olso, Oslo, Norway

76 National Trauma Research Institute, The Alfred Hospital, Monash University, Melbourne, Victoria, Australia

77 Department of Neurosurgery, Odense University Hospital, Odense, Denmark.

78 Department of Anasthesiology and Intensive Care Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway

79 Klinik für Neurochirurgie, Klinikum Ludwigsburg, Ludwigsburg, Germany

80 Department of Neuromedicine and Movement Science, Norwegian University of Science and Technology, NTNU, Trondheim, Norway

81 Department of Neuroanesthesia and Neurointensive Care, Odense University Hospital, Odense, Denmark

82 Department of Neurosurgery, University of Pécs, Pécs, Hungary

83 Department of Pathophysiology and Transplantation, Milan University, and Neuroscience ICU, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milano, Italy

84 Department of Radiation Sciences, Biomedical Engineering, Umeå University, Umeå, Sweden

85 Perioperative Services, Intensive Care Medicine and Pain Management, Turku University Hospital and University of Turku, Turku, Finland

86 Department of Neurosurgery, Kaunas University of Health Sciences, Kaunas, Lithuania

87 Intensive Care and Department of Pediatric Surgery, Erasmus Medical Center, Sophia Children’s Hospital, Rotterdam, The Netherlands

88 Department of Neurosurgery, Kings college London, London, UK

89 Neurologie, Neurochirurgie und Psychiatrie, Charité – Universitätsmedizin Berlin, Berlin, Germany

90 Department of Neurosurgery, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway

91 Department of Neurosurgery, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany

92 Section of Neurosurgery, Department of Surgery, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, MB, Canada

93 Department of Neurosurgery, University Hospital of Aachen, Aachen, Germany


This study is funded by the European Commission 7th Framework program (602150). The funder had no role in the design of the study and collection, analysis, interpretation of data, and in writing the manuscript.

Author information

Authors and Affiliations




JH analyzed the data and drafted the manuscript, tables, and figures. EW and DN were closely involved in data analyses and interpretation. JH, HL, and MJ designed the study protocol. HL and MJ supervised the study. All authors were involved in the design of the CENTER-TBI study and read and approved the final version of the manuscript.

Corresponding author

Correspondence to Jilske A. Huijben.

Ethics declarations

Ethics approval and consent to participate

For the CENTER-TBI study, ethical approval was given in each recruiting site; an online overview is available [32].

Consent for publication

Not applicable

Competing interests

AIRM declares consulting fees from PresSura Neuro, Integra Life Sciences, and NeuroTrauma Sciences. DKM reports grants from the UK National Institute for Health Research, during the conduct of the study; grants, personal fees, and non-financial support from GlaxoSmithKline; and personal fees from Neurotrauma Sciences, Lantmaanen AB, Pressura, and Pfizer, outside of the submitted work. All other authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Exclusion of Delphi quality indicators for application to the CENTER-TBI data. This table describes the consensus-based quality indicators (from the Delphi study) that could not be applied to the CENTER-TBI dataset for various reasons.

Additional file 2.

Structure indicator scores. This table shows the calculated structure indicator scores in the CENTER-TBI study. This is calculated at center-level including missing data and complete cases.

Additional file 3.

Process indicator scores. This table shows the calculated process indicator scores in the CENTER-TBI study. This is calculated at patient- and center-level including missing data and complete cases.

Additional file 4.

Outcome indicator scores. This table shows the calculated outcome indicator scores in the CENTER-TBI study. This is calculated at patient- and center-level including missing data and complete cases.

Additional file 5.

CENTER-TBI investigators and participants for the ICU stratum. This file includes the collaborator group: the CENTER-TBI investigators and participants for the ICU stratum and their affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huijben, J.A., Wiegers, E.J.A., Ercole, A. et al. Quality indicators for patients with traumatic brain injury in European intensive care units: a CENTER-TBI study. Crit Care 24, 78 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: