Preventable mortality evaluation in the ICU

Abstract

Mortality is the most widely measured outcome parameter. Improvement of this outcome parameter in critical care is nowadays expected not to come from new technologies or treatment, but from delivering the right care at the right moment in a safe way. The measurement of mortality as an outcome parameter confronts us with a problem in providing follow-up to the results. Especially when proven structure and process interventions are applied already, the cause of a suboptimal performance cannot be deduced easily. One possibility is to evaluate the causes of death and to judge preventability. In this article we explore the opportunities and difficulties of a tool to evaluate preventable mortality in the ICU.

Introduction

Critical illness is still associated with high mortality. The development of new technologies to improve outcome in the ICU has brought advances, but also at least as many disappointments. In 1999 the first appearance of the report To Err is Human confronted us with impressive numbers of patients dying each year in the USA as the result of preventable harm [1]. Since then it has been recognized that delivering the right care at the right moment in a safe way might have a greater impact on outcome than implementing new technology or treatments. As a consequence, many quality improvement efforts have been initiated in critical care. Most of these initiatives are based on the paradigm of Donabedian [2] and stress the importance of implementing proven structures and processes of care as the most effective step to improve outcome [3–6]. Structure and process interventions associated with improved outcome have been performed; for example, an improved outcome in ICUs with high-intensity staffing has been associated with better performance, and this has lead to more critical care patients being cared for continuously by critical care specialists [7]. As process measures, bundles to recognize and treat sepsis and ventilator-associated pneumonia have been widely implemented [8, 9].

Because the intention of quality improvement efforts is objective improvement, measuring safety and performance or outcome has received a tremendous boost and indicators have been defined [10]. Survival is the most important outcome of critical care and several measures have been developed to quantitate and compare mortality rates. However, not only rates are important. If we could analyze the causes of death and judge preventability, we might identify possibilities for improvement [11]. There is an analogy with the airline industry. We have better planes, safer airports, established procedures, and trained pilots; despite this there is still a plane crash once in a while. To improve safety further, every crash is extensively investigated. A similar approach is used in trauma care, evaluating deaths for preventable causes [12]. In critical care, there is a paucity of data about preventable mortality. In the present viewpoint we will explore the opportunities and difficulties of a tool to evaluate preventable mortality in the ICU.

How can preventable mortality in the ICU be defined?

Probably the most difficult issue in evaluating preventable mortality in the ICU is the definition of preventability [13]. In most studies, preventability is defined as an event (death) that would not have occurred if the patient had received ordinary standards of care appropriate for the time of study. The problem is that standards of care are not always clear for the individual, complex ICU patient. Moreover, established standards of care may show variations in time (for example, glucose regulation or activated protein C administration).

Another definition can be that death is preventable (retrospectively) if it would have been possible to prevent dying by using (or not using) some treatment. In the ICU setting we often have to deal with patients with an extensive medical history with a complex course upon which risky procedures are performed. Despite receiving the best possible care, some patients will develop complications or die. The effect of errors cannot always be distinguished from progression of the disease. Patients can die from a preventable event occurring before ICU admission. Patients can also die after ICU admission in the ward due to an event that happened during the ICU admission. Even more important is the fact that mortality in the ICU is mostly multifactorial and is often due to withdrawal or withholding [14]. Sometimes patients choose to die. These patients may have had a preventable event leading to that decision. Evaluation of treatment decisions should be regarded relating to life expectancy and health-related quality of life.

How do we evaluate mortality in the ICU now?

Mortality is the most used outcome measure in the ICU, and has been estimated in several ways.

The mortality rate is the easiest method to evaluate mortality. The definition seems clear and data are already automatically collected in almost all ICUs. Mortality rates give no information about preventability and cannot be compared between units or in time because of differences in severity of illness of the patient population. Local admission and discharge policies also influence ICU mortality.

The standardized mortality ratio or case-mix-adjusted mortality is a more relevant measure to evaluate mortality because risk adjustment is made. Data are collected in many ICUs, and in several countries national databases are in use that make benchmarking possible. Data collection takes more time and education is needed for correct scoring of illness severity [15]. The question remains whether risk adjustment using severity scores is always adequate [16]. Several patient categories are excluded from severity scoring systems. The standardized mortality ratio gives little information about preventability, because statistically unexpected death is not the same as preventable death [17]. The area in which the preventability could exist is not shown. Furthermore, a low standardized mortality ratio does not mean that preventable deaths do not occur [18]. The standardized mortality ratio can therefore be used as a screening tool for excess mortality rather than as a diagnostic tool for causes of mortality that can be prevented [19]. ICU mortality evaluation is also influenced by local admission and discharge policies [20].

Morbidity and mortality conferences can be used to evaluate deceased patients in a unit or hospital. Causes of death can be evaluated and preventability can be discussed. If preventable causes of death are identified, measures can be taken to prevent reoccurrence in the future. These conferences are often based on autopsy reports [21]. Autopsy is only performed in a selected portion (nonrandom sample) of deceased patients [22]. Usually, therefore, only a nonrandom sample of deceased patients is discussed. Morbidity and mortality conferences are time consuming and require a nonpunitive environment to be effective, but are potentially a good method to evaluate quality of care [23].

Regulatory authorities in most western countries require immediate acknowledgement of all lethal incidents. By definition these lethalities are preventable and require analysis and preventive measures. Root-cause analysis is often performed in these cases, which means all factors (roots) that resulted in harm are identified in order to evaluate what has to be improved or changed to prevent reoccurrence. Sometimes there are even legal consequences. The number of never events reported, however, is far less than the actual number of preventable incidents. For incidence estimation of preventable death and identification of improvement options, this type of registration is insufficient.

Can we use hazards and/or harm registration in the ICU for mortality evaluation? (Table 1)

Table 1 Terminology and definitions used to describe hazards and/or harm in patient care

Adverse events can cause morbidity and mortality. Adverse event examination as a trigger (clue) might therefore identify patients with a preventable cause of death [24]. Implementation of system changes following analysis of event reports has the potential to improve outcome by preventing the event [25].

Event reporting systems are a valuable tool to identify hazards and to learn from them by performing (root-cause) analysis. Routine reporting systems, however, have a poor performance in identifying patients with harm [26]. The most important problem of these systems is under-reporting, a possible cause of which is fear of blame or punishment and lack of clear definitions. The reporting rate can be increased by anonymity, regular feedback, and the existence of a safety climate [25]. In the ICU, defining events and deciding which events have to be registered is a challenge because of the multidisciplinarity and complexity of the patients and the environment. The correlation between an event and death is not always clear and there might be preventable deaths for which no adverse event has been registered because it was not recognized.

The Institute of Healthcare Improvement has developed an intensive care trigger tool that can be used as an adjunct to voluntary reporting of adverse events [27]. With this method a random selection of patient records is retrospectively screened for the presence of pre-determined criteria by trained reviewers to identify harmful events. Unexpected factors leading to adverse events will be missed.

Systematic evaluation of the incidence of selected clearly defined events can be used as a screening tool as part of a quality system. The main advantage of this method is that more accurate information is obtained because of focusing on a well-defined subject and rates of events can be measured. Benchmarking is possible and evaluation over time of improvement measures can be studied. Disadvantages are that this method is usually time-consuming and costly. Only a sample of the patients is evaluated and all hazards not defined as a trigger point are left out of the evaluation.

Prospective evaluation of adverse events by direct observation is the best available method to identify adverse events. A high number of adverse events has been identified [28] but usually no rates of events are obtained because not the whole population at risk is identified. Correlation between event and death cannot always be shown. Far more events than deaths will be found. Direct observation was found especially valuable in detecting near misses [28], as they are less frequently reported or documented in patient charts than adverse events but near misses have by definition no influence on mortality. Direct observation of performance therefore probably presents the least under-reporting but is very labor intensive, and the effect on preventable mortality is to be established.

In summary, hazards and/or harm registration in the ICU is valuable to identify hazards and learn from them by performing analysis, but is not sufficient for evaluation of preventable mortality.

Lessons from experiences outside the ICU

Trauma care

Trauma care has a longstanding tradition for evaluation of care [29]. Preventable deaths have been used for many years as a performance indicator and as a tool for improving the delivery of optimal care. Most experience has been gained at hospital level and national level [12, 30]. Several study designs have been used, with preference for cohort studies and case series [12, 30, 31].

Review of hospital charts is a frequently used method. Early studies were questioned about reliability and validity because of low reproducibility of implicit judgments of single reviewers. Later studies used panels of reviewers and more explicit judgment criteria. Data extraction by a professional followed by review of a summary of the patient record to reduce workload is an accepted method, but has not been widely tested for reliability and accuracy. An internal review panel might be more rigorous in identifying preventable harm than an external review panel [32]. A recent review of quality indicators to evaluate adult trauma care demonstrated evidence of reliability and validity and improved outcomes after implementation only for peer review of preventable death [31].

Another method used in trauma care for identifying possible preventable deaths is determining statistically unexpected death using the modified Trauma and Injury Severity Score. This method was found to have high sensitivity but moderate specificity for identifying deaths judged to be preventable by peer review [12].

Important differences between trauma care and critical care are the lower variation of patient categories in trauma care and the much smaller numbers of major trauma patients and deaths.

Related experience in other healthcare settings

Retrospective chart review is the most widely applied and thoroughly studied method for measurement of patient safety. This method is used on a national level in large studies evaluating preventable deaths and adverse events in the hospital, such as those leading to the To Err is Human report [1, 33–36]. In these studies, selections of patient charts were reviewed and results were extrapolated. Although several comments are published about the reliability of this method of investigation [37–39], it remains probably the best benchmark for estimating the extent of adverse events and preventable deaths at this level.

Quality and selection of patient charts is of utmost importance. Effective chart review depends on the quality of the patient chart. Only what is documented can be evaluated. Adverse events may therefore be missed, such as medication errors or team performance problems. In the large studies, charts were first reviewed on trigger points. These trigger points are predetermined screening criteria known to be sensitive to the occurrence of an adverse event, such as unplanned readmission [33–36]. Only the charts of patients with trigger points present were reviewed.

Review can be performed in several ways, but the choice has an impact on the results. A nondirected (holistic or implicit) review process will give broader information but lower inter-rater reliability. A criterion-based (or explicit) review process will be less sensitive, but has better reproducibility [40]. Reviewers can be external or internal, involved or not involved in patient care, doctors, nurses or others. How many reviewers preferably should be used is not known. Training of reviewers increases inter-rater reliability, as does discussion between reviewers, but whether this is due to congruence or precision is not clear [41, 42].

The goal of the review process has been defined differently in respective studies. Is a high inter-rater agreement required for valid incidence estimation? Or are as many preventable deaths reported as possible to identify as many improvement options as possible? If more reviewers per chart are used, a majority opinion, a panel consensus, or a unanimous decision can be used. These different methods will give large variations in the incidence of preventable mortality.

These studies show that the incidence of preventable mortality in the hospital, depending on the methodology, lies around 4% of total hospital mortality [35].

Practices outside healthcare

High-reliability organizations - such as aviation, aircraft carriers, the nuclear industry, and the oil industry - have extensive quality and safety programs despite the fact that they have very low numbers of incidents. The use of protocols, checklists, teamwork, and a focus on safety are strongly imbedded in these organizations. If an incident occurs despite all of the safety measures, retrospective analysis is performed immediately to evaluate the causes and reasons for failure and to find possibilities for improvement and prevention. Failure is not hidden, but rather is used as a way to gain insight into the performances and weaknesses of the system [43]. Although there are many similarities between high-reliability organizations and critical care, there are also some important differences - such as more variation and uncertainty of outcome in patients, harm at an individual patient level versus many casualties, including one's own personnel, and different team structures [44]. Death cannot always be considered a failure in critical care. We can, however, learn from high-reliability organizing and use evaluation of our deceased patients as a way to further improvement.

Consequences for the evaluation of preventable mortality in the ICU

From high-reliability organizations we learn about safety culture, standard operating procedures, and incident evaluation. Many of these items can be put in place in the ICU. In high-reliability organizations, however, all catastrophes with low incidence can be analyzed thoroughly, which we cannot copy to the ICU environment because of the considerable (expected) mortality. Possible ways to analyze preventable mortality in the ICU can be learnt from trauma care and the studies that have been performed to develop benchmarks and nationwide incidence of preventable in-hospital mortality. Retrospective chart review is the method most widely applied and studied. Preselecting based on trigger points such as statistically unlikely death or occurrence of adverse events will probably miss too many preventable deaths in the ICU setting. From trauma care we learn that charts can be extracted by an experienced healthcare worker to reduce the workload for the reviewers. In this way all reviewers can still evaluate all deceased patients. Experiences with preventable mortality evaluation teach us that the definition of preventability is cumbersome in healthcare. This will be even more the case in critical care. The inter-rater variability will be large, but larger judging committees will identify more possibilities for improvement. An implicit review method is subjective and will increase inter-rater variability but will probably identify more preventability. The incidence of preventable mortality is probably at least 4% of all deaths because more preventable death is expected in the ICU environment with the high risk for errors.

Definition difficulties, low incidence, and large inter-rater variability will prohibit the use of a preventable mortality score as a benchmark.

Proposal for a preventable death evaluation system in the ICU

We consider it worthwhile to start a project for the development of an evaluation system for preventable mortality in the ICU. Based on the experience from other fields we have the following suggestions.

Case selection

All deceased patients in the unit without preselection should be evaluated. In our unit, with approximately 3,000 admissions per year and 10% mortality, this would mean that we probably have to analyze 300 cases to find 12 preventable deaths.

Data extraction

An experienced healthcare worker can extract the patient data.

Review committee

A multidisciplinary team consisting of about five nurses and doctors (internal and external) should be able to review all extracted records and identify most preventable events leading to death.

Definition of preventable death

A broad and intuitive definition should be used to identify as many probabilities for improvement as possible. Death should occur in the ICU, otherwise facts cannot be retrieved reliably. The illness must be survivable and life expectancy should be taken into account.

Aim

The aim should be quality improvement. We therefore need to use a method that gives the maximum amount of information for improvement.

Conclusion

Discussion is important but consensus between all reviewers to conclude preventability is not required as too many opportunities for improvement may be missed that way.

Consequences

The conclusions are for internal use only because statistical reliability is insufficient. Possible improvements based on the results should be implemented.

Possible limitations and pitfalls

First, we have to present definitions and criteria for which deaths in the ICU are preventable and which deaths are not [13]. As previously discussed, a nondirected review process, based on the reviewer's own professional judgment, will provide broader information but lower inter-rater reliability; while a criterion-based review process, based on explicit standards, will be less sensitive but will have better reproducibility. In our institution we chose to start with the use of a nondirected review process because we think standards of care are often not clear in the complex ICU environment and many unexpected factors influencing outcome might be present. In the future we intend to evaluate the reproducibility and usefulness of this nondirected review process.

Second, if we succeed in defining a preventable death, we have to define what actions to take next. This important subject is beyond the scope of the present viewpoint and is discussed elsewhere [4]. One of the big challenges for healthcare organizations is not to identify harm in individual cases, but to create a learning organization. The preventable death evaluation system should be used to maximize improving patient care and to minimize adding to the long list of reports detailing the failures of modern healthcare.

Third, an important pitfall, is how to prepare the patient data and how to present these data to the reviewers. We started this process by letting the first and second authors of this article screen the charts and summarize the cases into anonymized (for patient and caregiver information) case vignettes. All available information considered necessary for identification of preventable death was extracted from the records. However, this is also a subjective process. In the future our research will focus on how and whether this case abstraction process can be both sensitive and specific enough to be worthwhile.

Conclusion

The ICU should be the safest place possible for our patients. Structure and process should be shaped according to widely accepted standards. Outcome should be measured and compared with existing benchmarks with all limitations in mind. Adverse events should be registered, evaluated and, whenever possible, prevented. After performing and evaluating care according to all standards, retrospective evaluation of mortality might reveal unnoticed preventable causes of death. These causes should be known to the ICU team so analysis can be performed and improvement actions can be taken.

Gaining insight into preventable mortality in the ICU is difficult. In this article we have evaluated possible methods to identify modifiable causes of death. Retrospective case review seems a feasible method for preventable mortality evaluation in the ICU. Ample evaluation of our own daily practice might reveal opportunities to prevent reoccurrence of events leading to death and thus improve the outcome of the critically ill patient, thereby creating the safest place possible.