Complication rates as a trauma care performance indicator: a systematic review

Introduction Information on complication rates is essential to trauma quality improvement efforts. However, it is unclear which complications are the most clinically relevant. The objective of this study was to evaluate whether there is consensus on the complications that should be used to evaluate the performance of acute care trauma hospitals. Methods We searched the Medline, EMBASE, Cochrane Central, CINAHL, BIOSIS, TRIP and ProQuest databases and included studies using at least one nonfatal outcome to evaluate the performance of acute care trauma hospitals. Data were extracted in duplicate using a piloted electronic data abstraction form. Consensus was considered to be reached if a specific complication was used in ≥ 70% of studies (strong recommendation) or in ≥ 50% of studies (weak recommendation). Results Of 14,521 citations identified, 22 were eligible for inclusion. We observed important heterogeneity in the complications used to evaluate trauma care. Seventy-nine specific complications were identified but none were used in ≥ 70% of studies and only three (pulmonary embolism, deep vein thrombosis, and pneumonia) were used in ≥ 50% of studies. Only one study provided evidence for the clinical relevance of complications used and only five studies (23%) were considered of high methodological quality. Conclusion Based on the results of this review, we can make a weak recommendation on three complications that should be used to evaluate acute care trauma hospitals; pulmonary embolism, deep vein thrombosis, and pneumonia. However, considering the observed disparity in definitions, the lack of clinical justification for the complications used, and the low methodological quality of studies, further research is needed to develop a valid and reliable performance indicator based on complications that can be used to improve the quality and efficiency of trauma care.


Introduction
Complications following admission for traumatic injury are common and have been shown to increase hospital mortality, length of stay, and costs [1][2][3][4][5]. These complications have been associated with a negative impact on long-term functional capacity and quality of life [6]. Many complications are potentially avoidable and quality improvement strategies aimed at reducing them have been shown to have a positive impact on patient outcome and resource utilization [7,8].
Complications have been identified as a priority for the development of trauma performance indicators [9,10]. Furthermore, US Trauma Quality Improvement Program members have made efforts to track information on complications following trauma using the National Trauma Data Bank according to the National Surgical Quality Improvement program [11,12]. However, despite the widespread availability of routinely collected data on hospital complications, a performance indicator based on complications has yet to be validated specifically in the context of acute trauma care [9,10]. Considering the potential positive and negative consequences of healthcare performance evaluation [13], such a performance indicator should be based on clinically relevant complications identified using standardized definitions [14] and robust methodology including adequate risk adjustment [15].
A recent systematic review synthesized studies evaluating performance indicators in trauma care [9]. However, information specific to the assessment of complications in the context of acute trauma care is limited. The objective of this study was to evaluate whether there is consensus on the complications that should be used to evaluate the performance of acute care trauma hospitals.

Materials and methods
We designed a systematic review of cohort studies evaluating the performance of acute care hospitals for the treatment of general trauma populations using information on complications. This systematic review was conducted following recommendations from the Cochrane handbook for systematic reviews [16] and in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [17]. The study was approved by our institutional research ethics committee. The present systematic review complements a review published by one of the authors (HTS) in 2011. The latter review was designed to document evidence of the reliability and validity of trauma quality indicators at large, whereas the present study is designed to evaluate whether there is consensus on the complications used to evaluate trauma care quality.

Search strategy
We searched the Medline, EMBASE, Cochrane Library, CINAHL, BIOSIS, TRIP and ProQuest databases for peer-reviewed articles and postgraduate academic publications. The websites of the following trauma associations were also searched: American College of Surgeons, American Association for the Surgery of Trauma, Eastern Association for the Surgery of Trauma, American Trauma Society, British Trauma Society, Trauma Association of Canada, Australasian Trauma Society, Western Trauma Association, Trauma.org, Society of Trauma Nurses, and International Trauma Anaesthesia and Critical Care Society. Our initial systematic search strategy was based on nonfatal outcomes at large. We then selected studies based on complications for this systematic review. The search strategy was designed for Medline and EMBASE using keywords and MeSH (Medline) or Emtree (EMBASE) for three groups of terms: trauma, performance, and nonfatal outcomes (see Additional file 1). Keywords were elaborated by a group of experts with methodological and clinical expertise who are co-investigators and collaborators on the project. The systematic search was limited to human studies from high-income countries conducted from the earliest available date of the search engine to 1 July 2011. The bibliographies for all identified studies and reviews were searched manually to identify additional relevant publications. No publication type, year, or language restriction was used.

Study selection
Both prospective and retrospective cohort studies evaluating the performance of acute care hospitals for the treatment of global trauma populations using at least one medical complication were considered eligible. A complication was defined as an additional problem that arises following a procedure, treatment, or illness and is secondary to it [18].
Duplicates were identified and sorted by two independent reviewers with methodological and content expertise (LM and HTS) using EndNote software version X4 (2010; Thomson Reuters, New York, NY, USA). These two reviewers independently evaluated citations identified for eligibility by screening titles, abstracts, and full publications. Disagreement on study eligibility was resolved by consensus, and a third reviewer (AFT) was involved when required. Inter-rater agreement was evaluated with kappa statistics on study eligibility. Articles written in a language other than English were translated.

Data abstraction
The same two reviewers independently extracted data using a standardized data abstraction form, which was piloted on a sample of five representative studies. The data abstraction form was designed to capture information on the study setting and design, complications, and methodological quality. The latter was evaluated using elements selected from the following sources: a tool proposed to evaluate the quality of complication-reporting in the surgical literature [14], the Strengthening the Reporting of Observational Studies in Epidemiology statement [19], the Cochrane risk of bias tool [20], and the Downs and Black tool [21]. The following 10 methodological quality criteria were thus selected by the project steering committee based on the consideration that they are important for the validity and reliability of performance evaluations: definitions of complications provided; justification for the choice of specific complications provided; duration of follow-up indicated; number and percentage of specific complications indicated; severity of complications considered (for example, major versus minor complications); risk adjustment used; data quality assurance efforts reported; adequate treatment of missing data; estimates of variation given; and adequate sample size.
Adequate treatment of missing data implied that the absence/presence and proportion of missing data were reported, and if > 10% of subjects had missing data then imputation techniques that take account of the uncertainty of missing data values (for example, multiple imputation/maximum expectation) or sensitivity analysis were used [22]. Sample size was considered adequate if at least 100 patients per hospital were available for analysis; if not, analysis strategies designed for low-volume centers (for example, shrinkage techniques) were used [15]. Disagreement on abstracted data between reviewers was resolved by consensus or, if necessary, consultation with a third reviewer (AFT).

Analysis
A classification system for complications following traumatic injury proposed elsewhere [23] was used to categorize specific complications identified in this review. Complications were thus grouped into the following 10 categories: pulmonary, cardiovascular, gastrointestinal, hepatic/biliary, hematologic, infection, genitourinary/renal, musculoskeletal/integumentary, neurologic, vascular, psychiatric, and other. According to Cochrane GRADE criteria for clinical recommendations [24], consensus was considered to be reached for a particular complication if at least 70% (strong recommendation) or 50% (weak recommendation) of studies used that complication to evaluate trauma care. Sensitivity analyses were performed to see whether greater consensus was achieved in more recent studies (later than 2005). Studies were considered to be of high methodology quality if at least seven out of 10 quality criteria were respected [25].

Search results
We identified 14,521 citations, of which 112 were selected for full text review after screening of titles and abstracts (Table 1). A total of 22 studies met the inclusion criteria . We had excellent inter-rater agreement on the selection of eligible studies with a kappa statistic of 0.89 (95% confidence interval = 0.79 to 0.99).

Complications
Four studies (18%) did not provide any information on the specific complications used to evaluate performance [31,32,38,44]. Among the other studies, we observed important heterogeneity in the complications used (Table 2). A total of 79 different complications were identified. None of the complications were used in more than 70% of studies (consistent with a strong recommendation) but the following complications were used in at least 50% of studies (weak recommendation): pulmonary embolism (n = 12), deep vein thrombosis (n = 12), and pneumonia (n = 11). Other commonly evaluated complications were sepsis (n = 8), urinary tract infection (n = 8), renal failure (n = 7), myocardial infarction (n = 7), wound infection (n = 7), decubitus ulcer (n = 6), respiratory failure (n = 6), and acute respiratory distress syndrome (n = 6).

Discussion
In this systematic review, we identified 79 specific complications that have been used to evaluate acute care trauma hospitals. None of these complications were consistent with a strong clinical recommendation but pulmonary embolism, deep vein thrombosis, and pneumonia were used in over 50% of studies and were therefore consistent with a weak clinical recommendation. Studies included in our review rarely justified the choice of complications included, and they generally had low methodological quality.
Reducing hospital complications is one of the keys to reducing morbidity, mortality, and resource utilization   [3][4][5]. Evaluating complications is therefore essential to improve the quality of care for patients admitted to acute care institutions for traumatic injuries. However, valid and reliable performance indicators are dependent on standardized definitions and rigorous methodological quality [15,48,49]. Standardized definitions should include a consensus on which complications should be used, how they should be defined, and the timing of evaluation [15,48,49]. First, the results of this study show that there is currently a lack of consensus on which complications should be used to evaluate trauma care; no single complication was used in more than 70% of studies, only two out of 22 studies shared the same list of specific complications [26,39], less than one-half gave justifications for the complications they used to evaluate care, and only one study demonstrated the clinical relevance of included complications [36]. This finding is supported by a previous study in a surgical population [14]. Second, less than one-half of studies provided definitions of specific complications, a problem noted by others [14]. Third, outcomes should be evaluated over a fixed period of time and should include early post-discharge data (for example, complications within the first 28 days of injury) [15]. Indeed, research has shown that major complications do occur after discharge from acute care [50]. In our systematic review, all studies evaluated complications from admission to discharge, which may unfairly advantage hospitals discharging patients early. Assessing complications after discharge is challenging but trauma registries may be linked to hospital discharge datasets to obtain information on complications that have led to hospital readmission.
Rigorous methodological quality includes, among other considerations, data quality and appropriate risk adjustment [15,48]. In this review, data quality assurance was mentioned in only one study [27] and less than one-third of studies used any type of risk adjustment. The problem of data quality or missing data in retrospective evaluations of hospital complications, underlined in our study, has been raised in previous studies [12,51,52]. Authors note that complications which are not recorded either due to differential surveillance or because of poor data quality are assumed to be absent, which leads to an underestimation for the incidence of hospital complications. Robust risk adjustment is essential to valid performance comparisons because of the heterogeneous case mix across trauma centers [15]. Indeed, the risk of complications has been reported to vary according to age, gender, injury severity, and comorbidities [18,53]. Distinguishing preventable from nonpreventable complications is also an important challenge in quality evaluations [48], that was partially addressed by three studies included in this review [26,29,39].

Potential limitations
The results of this systematic review should be interpreted in light of possible limitations. First, despite the exhaustive nature of our search strategy and very good inter-rater agreement on study eligibility, some studies may have been missed. For feasibility reasons, we restricted our website search to major trauma organizations in North America, Europe, and Australasia. By doing so, we may have missed some local studies based on regional trauma organizations. In the event that studies were missed, we may have overlooked some specific complications. Second, missing information in study reports meant that we did not have information on some study characteristics, definitions, and methodological quality criteria. For the latter, lack of information was assumed to mean that they did not meet the criteria and may have led to an underestimation of methodological quality. However, adequate reporting of important information is a reflection of study quality [19]. Third, complications that have been used for performance evaluation of trauma care may not necessarily represent those that researchers and stakeholders consider to be the most clinically relevant. For example, psychiatric complications were only evaluated in two studies and we did not identify any specific psychiatric complications -such as delirium, which is frequent in trauma populations [54]. In addition, the complications identified in this study are likely to have been heavily influenced by data availability, considering that most were based on retrospectively collected administrative or registry data. This influence highlights the importance of using a consensusbased procedure as well as information from a literature review to identify which complications should be used to evaluate trauma care.

Conclusion
According to the GRADE criteria, we can make a weak recommendation on three complications that should be used to evaluate the performance of acute care trauma hospitals: pulmonary embolism, deep vein thrombosis, and pneumonia. However, considering the heterogeneity of definitions used, the lack of clinical justification for the choice of specific complications, and the low methodological quality of included studies, further research is needed to develop a valid and reliable performance indicator based on complications that can be used to improve the quality and efficiency of trauma care.

Key messages
• Evidence in the literature is sufficient to make a weak recommendation on three complications that should be used to evaluate trauma center care; pulmonary embolism, deep vein thrombosis, and pneumonia.
• Definitions of complications are heterogeneous.
• Studies that use complications to evaluate trauma center care are generally of low methodological quality.
• The choice of complications is rarely justified by clinical criteria.
• Further research is needed to develop a valid and reliable performance indicator based on complications that can be used to improve the quality and efficiency of trauma care.