Preventable mortality evaluation in the ICU

Mortality is the most widely measured outcome parameter. Improvement of this outcome parameter in critical care is nowadays expected not to come from new technologies or treatment, but from delivering the right care at the right moment in a safe way. The measurement of mortality as an outcome parameter confronts us with a problem in providing follow-up to the results. Especially when proven structure and process interventions are applied already, the cause of a suboptimal performance cannot be deduced easily. One possibility is to evaluate the causes of death and to judge preventability. In this article we explore the opportunities and difficulties of a tool to evaluate preventable mortality in the ICU.


Introduction
Critical illness is still associated with high mortality. Th e development of new technologies to improve outcome in the ICU has brought advances, but also at least as many disappointments. In 1999 the fi rst appearance of the report To Err is Human confronted us with impressive numbers of patients dying each year in the USA as the result of preventable harm [1]. Since then it has been recognized that delivering the right care at the right moment in a safe way might have a greater impact on outcome than implementing new technology or treatments. As a consequence, many quality improvement eff orts have been initiated in critical care. Most of these initiatives are based on the paradigm of Donabedian [2] and stress the importance of implementing proven structures and processes of care as the most eff ective step to improve outcome [3][4][5][6]. Structure and process interventions associated with improved outcome have been performed; for example, an improved outcome in ICUs with high-intensity staffi ng has been associated with better performance, and this has lead to more critical care patients being cared for continuously by critical care specialists [7]. As process measures, bundles to recognize and treat sepsis and ventilator-associated pneumonia have been widely implemented [8,9].
Because the intention of quality improvement eff orts is objective improvement, measuring safety and performance or outcome has received a tremendous boost and indicators have been defi ned [10]. Survival is the most important outcome of critical care and several measures have been developed to quantitate and compare mortality rates. However, not only rates are important. If we could analyze the causes of death and judge preventability, we might identify possibilities for improvement [11]. Th ere is an analogy with the airline industry. We have better planes, safer airports, established procedures, and trained pilots; despite this there is still a plane crash once in a while. To improve safety further, every crash is extensively investigated. A similar approach is used in trauma care, evaluating deaths for preventable causes [12]. In critical care, there is a paucity of data about preventable mortality. In the present viewpoint we will explore the opportunities and diffi culties of a tool to evaluate preventable mortality in the ICU.

How can preventable mortality in the ICU be defi ned?
Probably the most diffi cult issue in evaluating preventable mortality in the ICU is the defi nition of preventability [13]. In most studies, preventability is defi ned as an event (death) that would not have occurred if the patient had received ordinary standards of care appropriate for the time of study. Th e problem is that standards of care are not always clear for the individual, complex ICU patient. Moreover, established standards of care may show variations in time (for example, glucose regulation or activated protein C administration).
Another defi nition can be that death is preventable (retrospectively) if it would have been possible to prevent dying by using (or not using) some treatment. In the ICU setting we often have to deal with patients with an extensive medical history with a complex course upon which risky procedures are performed. Despite receiving the best possible care, some patients will develop complications or die. Th e eff ect of errors cannot always be distinguished from progression of the disease. Patients can die from a preventable event occurring before ICU admission. Patients can also die after ICU admission in the ward due to an event that happened during the ICU admission. Even more important is the fact that mortality in the ICU is mostly multifactorial and is often due to withdrawal or withholding [14]. Sometimes patients choose to die. Th ese patients may have had a preventable event leading to that decision. Evaluation of treatment decisions should be regarded relating to life expectancy and health-related quality of life.

How do we evaluate mortality in the ICU now?
Mortality is the most used outcome measure in the ICU, and has been estimated in several ways.
Th e mortality rate is the easiest method to evaluate mortality. Th e defi nition seems clear and data are already automatically collected in almost all ICUs. Mortality rates give no information about preventability and cannot be compared between units or in time because of diff erences in severity of illness of the patient population. Local admission and discharge policies also infl uence ICU mortality.
Th e standardized mortality ratio or case-mix-adjusted mortality is a more relevant measure to evaluate mortality because risk adjustment is made. Data are collected in many ICUs, and in several countries national databases are in use that make benchmarking possible. Data collection takes more time and education is needed for correct scoring of illness severity [15]. Th e question remains whether risk adjustment using severity scores is always adequate [16]. Several patient categories are excluded from severity scoring systems. Th e standardized mortality ratio gives little information about preventability, because statistically unexpected death is not the same as preven table death [17]. Th e area in which the preventa bility could exist is not shown. Furthermore, a low standardized mortality ratio does not mean that preventable deaths do not occur [18]. Th e standardized mortality ratio can there fore be used as a screening tool for excess mortality rather than as a diagnostic tool for causes of mortality that can be prevented [19]. ICU mortality evaluation is also infl uenced by local admission and discharge policies [20].
Morbidity and mortality conferences can be used to evaluate deceased patients in a unit or hospital. Causes of death can be evaluated and preventability can be discussed. If preventable causes of death are identifi ed, measures can be taken to prevent reoccurrence in the future. Th ese conferences are often based on autopsy reports [21]. Autopsy is only performed in a selected portion (nonrandom sample) of deceased patients [22]. Usually, therefore, only a nonrandom sample of deceased patients is discussed. Morbidity and mortality conferences are time consum ing and require a non punitive environment to be eff ective, but are potentially a good method to evaluate quality of care [23].
Regulatory authorities in most western countries require immediate acknowledgement of all lethal incidents. By defi nition these lethalities are preventable and require analysis and preventive measures. Root-cause analysis is often performed in these cases, which means all factors (roots) that resulted in harm are identifi ed in order to evaluate what has to be improved or changed to prevent reoccurrence. Sometimes there are even legal consequences. Th e number of never events reported, however, is far less than the actual number of preventable incidents. For incidence estimation of preventable death and identifi cation of improvement options, this type of registration is insuffi cient.
Can we use hazards and/or harm registration in the ICU for mortality evaluation? (Table 1) Adverse events can cause morbidity and mortality. Adverse event examination as a trigger (clue) might there fore identify patients with a preventable cause of death [24]. Implementation of system changes following analysis of event reports has the potential to improve outcome by preventing the event [25].
Event reporting systems are a valuable tool to identify hazards and to learn from them by performing (rootcause) analysis. Routine reporting systems, however, have a poor performance in identifying patients with harm [26]. Th e most important problem of these systems is under-reporting, a possible cause of which is fear of blame or punishment and lack of clear defi nitions. Th e reporting rate can be increased by anonymity, regular feedback, and the existence of a safety climate [25]. In the ICU, defi ning events and deciding which events have to be registered is a challenge because of the multi disciplinarity and complexity of the patients and the environment. Th e correlation between an event and death is not always clear and there might be preventable deaths for which no adverse event has been registered because it was not recognized.
Th e Institute of Healthcare Improvement has developed an intensive care trigger tool that can be used as an adjunct to voluntary reporting of adverse events [27]. With this method a random selection of patient records is retrospectively screened for the presence of predetermined criteria by trained reviewers to identify harm ful events. Unexpected factors leading to adverse events will be missed.
Systematic evaluation of the incidence of selected clearly defi ned events can be used as a screening tool as part of a quality system. Th e main advantage of this method is that more accurate information is obtained because of focusing on a well-defi ned subject and rates of events can be measured. Benchmarking is possible and evaluation over time of improvement measures can be studied. Disadvantages are that this method is usually time-consuming and costly. Only a sample of the patients is evaluated and all hazards not defi ned as a trigger point are left out of the evaluation.
Prospective evaluation of adverse events by direct observation is the best available method to identify adverse events. A high number of adverse events has been identifi ed [28] but usually no rates of events are obtained because not the whole population at risk is identifi ed. Corre lation between event and death cannot always be shown. Far more events than deaths will be found. Direct observation was found especially valuable in detecting near misses [28], as they are less frequently reported or documented in patient charts than adverse events but near misses have by defi nition no infl uence on mortality. Direct observation of performance therefore probably presents the least under-reporting but is very labor intensive, and the eff ect on preventable mortality is to be established.
In summary, hazards and/or harm registration in the ICU is valuable to identify hazards and learn from them by performing analysis, but is not suffi cient for evaluation of preventable mortality.

Trauma care
Trauma care has a longstanding tradition for evaluation of care [29]. Preventable deaths have been used for many years as a performance indicator and as a tool for improving the delivery of optimal care. Most experience has been gained at hospital level and national level [12,30].
Several study designs have been used, with prefer ence for cohort studies and case series [12,30,31].
Review of hospital charts is a frequently used method. Early studies were questioned about reliability and validity because of low reproducibility of implicit judgments of single reviewers. Later studies used panels of reviewers and more explicit judgment criteria. Data extraction by a professional followed by review of a summary of the patient record to reduce workload is an accepted method, but has not been widely tested for reliability and accuracy. An internal review panel might be more rigorous in identifying preventable harm than an external review panel [32]. A recent review of quality indicators to evaluate adult trauma care demonstrated evidence of reliability and validity and improved outcomes after implementation only for peer review of preventable death [31].
Another method used in trauma care for identifying possible preventable deaths is determining statistically unexpected death using the modifi ed Trauma and Injury Severity Score. Th is method was found to have high sensitivity but moderate specifi city for identifying deaths judged to be preventable by peer review [12].
Important diff erences between trauma care and critical care are the lower variation of patient categories in trauma care and the much smaller numbers of major trauma patients and deaths.

Related experience in other healthcare settings
Retrospective chart review is the most widely applied and thoroughly studied method for measurement of patient safety. Th is method is used on a national level in large Table 1

. Terminology and defi nitions used to describe hazards and/or harm in patient care
Patient safety incident: event or circumstance that could have resulted in, or did result in, unnecessary harm to a patient Adverse event (harmful incident): • Injury or harm related to (or from) the delivery of care (Institute of Healthcare Improvement) • A patient safety incident; undesirable health event that may or may not be related to the treatment • Any injury due to medical management, rather than to the underlying disease [28] • In several studies: an incident that resulted in death, life-threatening illness, disability at time of discharge, admission to hospital or prolongation of hospital stay Medical error: • An adverse event that is preventable, inaccurate or incomplete diagnosis and/or treatment • Failure of a planned action to be completed as intended or the use of a wrong plan to achieve an aim [28] Sentinel event (Joint Commission on Accreditation of Healthcare Organizations): a serious medical error; any unanticipated event in a healthcare setting resulting in death or serious physical or psychological injury Never event: a serious reportable event; occurrence that should never happen in a hospital and can be prevented Near miss: a patient safety incident; event or situation that could have resulted in an accident, injury or illness but did not, either by chance or through timely intervention Critical incident: adverse event with the potential to harm patients, staff or visitors [45] Complication: unfavorable evolution of a disease, health condition or medical treatment studies evaluating preventable deaths and adverse events in the hospital, such as those leading to the To Err is Human report [1,[33][34][35][36]. In these studies, selections of patient charts were reviewed and results were extra polated. Although several comments are published about the reliability of this method of investigation [37][38][39], it remains probably the best benchmark for estimating the extent of adverse events and preventable deaths at this level. Quality and selection of patient charts is of utmost importance. Eff ective chart review depends on the quality of the patient chart. Only what is documented can be evaluated. Adverse events may therefore be missed, such as medication errors or team performance problems. In the large studies, charts were fi rst reviewed on trigger points. Th ese trigger points are predetermined screening criteria known to be sensitive to the occurrence of an adverse event, such as unplanned readmission [33][34][35][36]. Only the charts of patients with trigger points present were reviewed.
Review can be performed in several ways, but the choice has an impact on the results. A nondirected (holistic or implicit) review process will give broader information but lower inter-rater reliability. A criterionbased (or explicit) review process will be less sensitive, but has better reproducibility [40]. Reviewers can be external or internal, involved or not involved in patient care, doctors, nurses or others. How many reviewers preferably should be used is not known. Training of reviewers increases inter-rater reliability, as does discussion between reviewers, but whether this is due to congruence or precision is not clear [41,42].
Th e goal of the review process has been defi ned diff erently in respective studies. Is a high inter-rater agreement required for valid incidence estimation? Or are as many preventable deaths reported as possible to identify as many improvement options as possible? If more reviewers per chart are used, a majority opinion, a panel consensus, or a unanimous decision can be used. Th ese diff erent methods will give large variations in the incidence of preventable mortality.
Th ese studies show that the incidence of preventable mortality in the hospital, depending on the methodology, lies around 4% of total hospital mortality [35].

Practices outside healthcare
High-reliability organizations -such as aviation, aircraft carriers, the nuclear industry, and the oil industry -have extensive quality and safety programs despite the fact that they have very low numbers of incidents. Th e use of protocols, checklists, teamwork, and a focus on safety are strongly imbedded in these organizations. If an incident occurs despite all of the safety measures, retrospective analysis is performed immediately to evaluate the causes and reasons for failure and to fi nd possibilities for improve ment and prevention. Failure is not hidden, but rather is used as a way to gain insight into the performances and weaknesses of the system [43]. Although there are many similarities between high-reliability organizations and critical care, there are also some important diff erences -such as more variation and un certainty of outcome in patients, harm at an individual patient level versus many casualties, including one's own personnel, and diff erent team structures [44]. Death cannot always be considered a failure in critical care. We can, however, learn from high-reliability organizing and use evaluation of our deceased patients as a way to further improvement.

Consequences for the evaluation of preventable mortality in the ICU
From high-reliability organizations we learn about safety culture, standard operating procedures, and incident evaluation. Many of these items can be put in place in the ICU. In high-reliability organizations, however, all catastrophes with low incidence can be analyzed thoroughly, which we cannot copy to the ICU environment because of the considerable (expected) mortality. Possible ways to analyze preventable mortality in the ICU can be learnt from trauma care and the studies that have been performed to develop benchmarks and nationwide incidence of preventable in-hospital mortality. Retrospective chart review is the method most widely applied and studied. Preselecting based on trigger points such as statistically unlikely death or occurrence of adverse events will probably miss too many preventable deaths in the ICU setting. From trauma care we learn that charts can be extracted by an experienced healthcare worker to reduce the workload for the reviewers. In this way all reviewers can still evaluate all deceased patients. Experiences with prevent able mortality evaluation teach us that the defi nition of preventability is cumbersome in healthcare. Th is will be even more the case in critical care. Th e interrater variability will be large, but larger judging committees will identify more possibilities for improvement. An implicit review method is subjective and will increase inter-rater variability but will probably identify more pre ventability. Th e incidence of preventable mortality is probably at least 4% of all deaths because more prevent able death is expected in the ICU environment with the high risk for errors.
Defi nition diffi culties, low incidence, and large interrater variability will prohibit the use of a preventable mortality score as a benchmark.

Proposal for a preventable death evaluation system in the ICU
We consider it worthwhile to start a project for the development of an evaluation system for preventable mortality in the ICU. Based on the experience from other fi elds we have the following suggestions.

Case selection
All deceased patients in the unit without preselection should be evaluated. In our unit, with approximately 3,000 admissions per year and 10% mortality, this would mean that we probably have to analyze 300 cases to fi nd 12 preventable deaths.

Data extraction
An experienced healthcare worker can extract the patient data.

Review committee
A multidisciplinary team consisting of about fi ve nurses and doctors (internal and external) should be able to review all extracted records and identify most preventable events leading to death.

Defi nition of preventable death
A broad and intuitive defi nition should be used to identify as many probabilities for improvement as possible. Death should occur in the ICU, otherwise facts cannot be retrieved reliably. Th e illness must be survivable and life expectancy should be taken into account.

Aim
Th e aim should be quality improvement. We therefore need to use a method that gives the maximum amount of information for improvement.

Conclusion
Discussion is important but consensus between all reviewers to conclude preventability is not required as too many opportunities for improvement may be missed that way.

Consequences
Th e conclusions are for internal use only because statistical reliability is insuffi cient. Possible improvements based on the results should be implemented.

Possible limitations and pitfalls
First, we have to present defi nitions and criteria for which deaths in the ICU are preventable and which deaths are not [13]. As previously discussed, a nondirected review process, based on the reviewer's own professional judgment, will provide broader information but lower inter-rater reliability; while a criterion-based review process, based on explicit standards, will be less sensitive but will have better reproducibility. In our institution we chose to start with the use of a nondirected review process because we think standards of care are often not clear in the complex ICU environment and many unexpected factors infl uencing outcome might be present. In the future we intend to evaluate the reproducibility and usefulness of this non directed review process.
Second, if we succeed in defi ning a preventable death, we have to defi ne what actions to take next. Th is important subject is beyond the scope of the present viewpoint and is discussed elsewhere [4]. One of the big challenges for healthcare organizations is not to identify harm in individual cases, but to create a learning organization. Th e preventable death evaluation system should be used to maximize improving patient care and to minimize adding to the long list of reports detailing the failures of modern healthcare.
Th ird, an important pitfall, is how to prepare the patient data and how to present these data to the reviewers. We started this process by letting the fi rst and second authors of this article screen the charts and summarize the cases into anonymized (for patient and caregiver information) case vignettes. All available information considered neces sary for identifi cation of preventable death was extracted from the records. However, this is also a subjective process. In the future our research will focus on how and whether this case abstraction process can be both sensitive and specifi c enough to be worthwhile.

Conclusion
Th e ICU should be the safest place possible for our patients. Structure and process should be shaped according to widely accepted standards. Outcome should be measured and compared with existing benchmarks with all limitations in mind. Adverse events should be registered, evaluated and, whenever possible, prevented. After performing and evaluating care according to all standards, retrospective evaluation of mortality might reveal unnoticed preventable causes of death. Th ese causes should be known to the ICU team so analysis can be performed and improvement actions can be taken.
Gaining insight into preventable mortality in the ICU is diffi cult. In this article we have evaluated possible methods to identify modifi able causes of death. Retrospective case review seems a feasible method for preventable mortality evaluation in the ICU. Ample evaluation of our own daily practice might reveal opportunities to prevent reoccurrence of events leading to death and thus improve the outcome of the critically ill patient, thereby creating the safest place possible.