Open Access

Judging quality of current septic shock definitions and criteria

  • Manu Shankar-Hari1, 2Email author,
  • Guido Bertolini3,
  • Frank M. Brunkhorst4,
  • Rinaldo Bellomo5,
  • Djillali Annane6,
  • Clifford S. Deutschman7, 8 and
  • Mervyn Singer9
Critical Care201519:445

https://doi.org/10.1186/s13054-015-1164-6

Published: 25 December 2015

Abstract

Septic shock definitions are being revisited. We assess the feasibility, reliability, and validity characteristics of the current definitions and criteria of septic shock. Septic shock is conceptualised as cardiovascular dysfunction, tissue perfusion and cellular abnormalities caused by infection. Currently, for feasibility, septic shock is identified at the bedside by using either hypotension or a proxy for tissue perfusion/cellular abnormalities (e.g., hyperlactatemia). We propose that concurrent presence of cardiovascular dysfunction and perfusion/cellular abnormalities could improve validity of septic shock diagnosis, as we are more likely to identify a patient population with all elements of the illness concept. This epidemiological refinement should not affect clinical care and may aid study design to identify illness-specific biomarkers and interventions.

Introduction

The illness concept, with its description (definition), clinical symptoms, signs and laboratory tests (criteria), and consequences (outcome), inform clinical reasoning and diagnosis. Diagnostic certainty is important for patients, clinicians and researchers as it helps to identify risk factors, select optimal treatment, devise new therapies, and prognosticate. In this context, septic shock is a clinical syndrome with protean clinical manifestations and biochemical abnormalities but with no universally accepted gold standard for diagnosis [1]. In the absence of a gold standard test, the clinician-determined probability of the illness, the definition and the corresponding criteria are fundamental to diagnosing syndromes. After outlining the septic shock illness concept, this article evaluates how bedside operationalization of current definitions of septic shock definitions and their corresponding clinical criteria influence the validity, reliability, and feasibility attributes [2].

Illness concept

Infection triggers a series of host responses, dysregulation of which results in organ dysfunction (sepsis) [35]. Septic shock, the most severe form of sepsis, is highly complex, with concurrent myocardial, vascular (macro- and micro-), tissue perfusion and cellular level abnormalities. Myocardial abnormalities include both cardiomyocyte injury [4] and structural myocardial dysfunction [6, 7]. Vascular abnormalities affect the endothelium, sub-endothelial tissues, microcirculation and vasomotor tone. These alter systemic vascular resistance, increase endothelial permeability, impair tissue oxygen delivery (tissue hypoxia) and utilization (cellular dysoxia), and modify cellular metabolism (Fig. 1) [815].
Fig. 1

Simplified overview of septic shock biology

Current definitions and criteria for septic shock

The 1992 and 2001 Sepsis Consensus Conference definitions [16, 17] and the definition used by the Surviving Sepsis Campaign (SSC) guidelines [18] currently provide the most-cited case definitions and criteria for making a diagnosis of septic shock. Yet all differ for both definition and criteria (e.g., use of different cutoffs for hypotension) and variable use of markers of impaired tissue perfusion (e.g., raised lactate, base deficit). There is also an implicit reliance upon clinician-driven interventions such as fluid therapy or inotropes. Thus, there are potentially varying combinations of clinical criteria and differences in the interventional targets used to diagnose septic shock. This heterogeneity highlights the limitations of current operationalization methods [1921].

Judging quality attributes of septic shock definitions and criteria

Validity

Validity is the ability to capture what the investigator truly seeks to measure. Validity assessments can be either qualitative (face validity, content validity) or quantitative (criterion validity, construct validity).

Face validity

“Face validity” refers to whether patients identified by the criteria appear to match the illness concept at face value. At present, hypotension is a core element of the criteria identifying septic shock. However, the presence of hypotension in the context of infection does not necessarily define septic shock. Hypotension in an infected patient could be related to pre-illness medications (e.g., antihypertensive), comorbidities (e.g., heart failure) or concurrent interventions (e.g., sedatives) or a combination of these. In other words, we remain uncertain whether hypotension as the only criterion truly represents septic shock, especially when used without caveats to address potential confounders (e.g., medications or heart failure). Furthermore, we cannot quantify this uncertainty, as there is no gold standard test for septic shock.

Content validity

Content validity assesses whether the clinical criteria encompass all components of the illness. For septic shock, these components are cardiovascular dysfunction, hypoperfusion and cellular abnormalities. We highlighted above that hypotension alone is an incomplete proxy for cardiovascular dysfunction to define septic shock. Equally, imperfect proxies of tissue hypoperfusion such as base deficit, confusion or oliguria are inadequate in isolation to define septic shock [16]. Similarly, we currently lack a reliable proxy to measure the complex cellular abnormalities that occur in septic shock. Hyperlactatemia is often used as a proxy; however, this is usually found in all types of shock [8] and the serum level is variably related to multiple confounders, including the rate of tissue utilization (lactate clearance), accelerated β-adrenergic-driven aerobic glycolysis, liver dysfunction and co-existing anaerobic glycolysis [11, 2225].

Given the lack of a gold standard diagnostic test for septic shock, the performance characteristics of these variables (sensitivity, specificity, and predictive values) could be based on prognosis (e.g., short-term mortality), with or without a clinical reclassification risk assessment or blinded clinician adjudication [26]. If we alter the existing clinical criteria, the reclassification risk refers to the proportion of patients reclassified into new risk-of-death categories [26]. Levy et al., using the SSC dataset, gave a simple descriptive demonstration of this concept by using lactate as an additional marker for septic shock in a cohort of patients with an overall mortality of 38.4 % [27]. The authors reported categories of shock as lactate of more than 4 mmol/l only (29.9 % mortality), vasopressor use only (36.7 % mortality), or a combination of lactate of more than 4 mmol/l plus vasopressor use (46.1 % mortality). A similar exercise conducted on data collected from English intensive care units (ICUs) reported mortality rates of 26.2 % for patients with an isolated lactate level of more than 4 mmol/l, 31.4 % for refractory hypotension only, and 55.5 % for the combination [28]. Kaukonen et al. recently evaluated the performance characteristics of the four systemic inflammatory response syndrome (SIRS) criteria in identifying sepsis [29]. They highlighted that 12 % of patients admitted to ICUs with organ dysfunction consequent to presumed infection were SIRS-negative (i.e., had fewer than two criteria of SIRS). Thus, if sepsis criteria are developed without requiring SIRS variables, different risk categories within patient populations may be identified (i.e., reclassification).

While cardiovascular organ dysfunction has long been central to the definition of septic shock, the criteria used to identify it remain variable. Levy et al. [17] specified the use of either Sepsis Organ Failure Assessment (SOFA) or multiple organ dysfunction score (MODS) to quantify organ dysfunction in patients with severe sepsis [30, 31]. However, these two scoring systems differ, particularly in terms of cardiovascular dysfunction criteria. The SOFA score uses both mean arterial pressure and the dose of vasoactive drugs being administered. By contrast, the MODS is solely physiology-based, using a pressure-adjusted heart rate parameter derived from heart rate, central venous pressure and mean arterial pressure to reflect fluid-unresponsive hypotension. Under the MODS scoring system, organ dysfunction is a continuum with no category differentiating dysfunction from failure [3032]. Cardiovascular dysfunction is quantified by a continuous variable derived by using regression analyses to define the variables and their weights. In contrast, SOFA, which was developed by using expert opinion, views cardiovascular dysfunction as occurring in discrete steps, with a score of 1 or 2 being coded as organ “dysfunction” and 3 or 4 coded as organ “failure”. Thus, the cutoffs and relative weights of variables within these scores are unlikely to stratify cardiovascular dysfunction similarly [33].

Criterion and construct validity

Criterion validity encompasses both concurrent and predictive validities. Concurrent validity refers to the ability of the definition and criteria to discriminate groups, whereas predictive validity is the ability to predict (future) outcomes. A closely related concept is construct validity; this refers to an assessment of how well the definitions are converted into measurable criteria to identify septic shock in clinical practice.

There are many examples in the literature of poor criterion validity for current definitions and criteria of septic shock. For example, a cohort study of nearly 8000 patients defined by the 1992 criteria as having septic shock [16] reported an overall crude hospital mortality of 52.4 % [20]. However, mortality ranged from 21.1 to 84.5 % when stratified on anatomical site of infection, and this variation persisted despite adjustments for confounders such as age, comorbidities and organism type. Given that cardiovascular dysfunction is the core criterion used for diagnosis, the predictive validity of current definitions of septic shock is weak, being heavily dependent on how it is operationalized at the bedside [19].

Outcome is also affected by how the individual components of septic shock (cardiovascular dysfunction, hypoperfusion and cellular abnormalities) are determined. Mortality in a single dataset varied from 45 to 60 % depending on what diagnostic criteria were applied [34]. Similarly, hospital mortality rates in septic shock patients admitted to ICUs in Australia and New Zealand (171 ICUs; n = 6757) (ANZICS data, R. Bellomo, personal communication) and Italy (221 ICUs; n = 4959) (GiViTI data, Italian ICU registry; Margherita project, G. Bertolini, personal communication) during 2012 were reported as 22 and 57.9 %, respectively. The Australasian case definition was predicated upon an APACHE III (Acute Physiology and Chronic Health Evaluation III) diagnosis coding of infection plus recording of either a decrease in mean blood pressure of less than 65 mmHg or systolic blood pressure of less than 90 mmHg at any time within the first 24 h of admission. In contrast, the Italian criteria used those provided by the 2001 Consensus definitions [17]. Even when data were extracted from representative national datasets by using the International Classification of Diseases coding system, the hospital mortality using the septic shock code was much higher in Germany (60.5 % in 2011) [35] than in the USA (42.1 % between 2004 and 2009) [21]. Are outcomes in Germany much worse, or do the differences merely reflect coding practices or variable patient pathways (e.g., transfer to post-acute care hospitals) [36]?

Secular trends in septic shock outcomes are depicted in Fig. 2. Among other explanations [37], this improving trend in mortality may also represent an enhanced detection of a less severe “septic shock” population using hypotension alone as a criterion (i.e., stage migration, or the Will Rogers phenomenon [38]).
Fig. 2

Secular trends in septic shock mortality. Cub-REA Data were provided by Philippe Aegerter, Bertrand Guidet and D. Annane for the Cub-REA network, which used International Classification of Diseases (ICD)-9 and ICD-10 codes and the Bone definition [16]. GiViTI data were provided by Bertolini et al. using 2001 Consensus Definitions [16, 17]. ANZICS data, provided by Bellomo et al. (personal communications), used hypotension as the definition for septic shock. Brunkhorst et al. (personal communications) provided German data for 2011, using ICD codes for the diagnosis of septic shock

Reliability

Reliability refers to agreement between observers and by the same observer during repeated measurements (i.e., consistency and reproducibility). With septic shock, some measurements, such as blood pressure and lactate levels, are likely to have reasonably high inter-observer, intra-observer and intra-subject reliability subject to certain caveats and exclusion of methodological errors. On the other hand, adequacy of fluid resuscitation and initiation of vasopressor therapy are far more subjective because of a lack of consensus on triggers and end-points. Potentially, we can test the reliability attributes of these variables (e.g., consistency within and between observers) by using test-retest reliability and the related kappa statistic.

Feasibility

Feasibility is a composite concept that depends on the purpose of the diagnosis; it is a compromise between validity and reliability [2]. For a high-mortality condition such as septic shock, ease of detection is key. The SSC guidelines have increased feasibility by emphasizing cardiovascular dysfunction criteria for the diagnosis of septic shock [18]. However, relaxing diagnostic criteria will almost certainly increase the rate of false-positive diagnoses. Conversely, if the complexity and number of criteria to be met for diagnosis are increased to improve validity, then feasibility will almost certainly be reduced. For example, the incidence of septic shock was halved from 9.1 to 4.4 % when liberal criteria (i.e., refractory hypotension) were replaced with restrictive criteria (i.e., refractory hypotension with non-cardiovascular dysfunction) [34].

How could we address these challenges?

The above discussion highlights variability in the criteria used to identify septic shock in published research. The European Society of Intensive Care Medicine-Society of Critical Care Medicine (ESICM-SCCM) Task Force on new definitions of sepsis is currently undertaking a systematic review of the literature to explore how these septic shock definitions and criteria are operationalized. Observational studies reporting the incidence and outcomes of septic shock are ideal target publications as randomized controlled trials often have trial-specific criteria. The second step in this process is to generate agreement on the updated illness concept which should reflect all three domains of biology (i.e., cardiovascular dysfunction, cellular abnormalities and evidence of impaired tissue perfusion). Based on this illness concept and by applying the principle of parsimony, the minimum number of bedside variables that have face and content validity could be determined. Inter-observer reliability of septic shock case detection could be improved by using a single validated criterion for cardiovascular dysfunction (e.g., need for vasopressor therapy to maintain target blood pressure) and a proxy for likely cellular and impaired tissue perfusion abnormality (e.g., lactate). These steps could be achieved with qualitative research methods, such as a Delphi process [39]. An important research question in this context is whether septic shock illness criteria should have predictive validity [19]. If we believe that septic shock is the most severe form of sepsis, then an argument could be made for having predictive validity as a characteristic of any updated septic shock criteria. This research question could be addressed by developing a parsimonious model using combinations and cutoffs of the variables for outcomes such as short-term mortality.

Conclusions

Septic shock conceptually comprises an illness with new onset or worsening cardiovascular dysfunction, impaired tissue perfusion and cellular abnormalities caused by infection. Individually, none of these abnormalities truly reflects the complex illness concept but they may do so in combination, potentially improving the validity and reliability of a diagnosis of septic shock. As there are no specific treatments for this condition, this epidemiological refinement will not affect clinical care and may aid the design of studies that can identify illness-specific biomarkers and interventions.

Abbreviations

ICU: 

Intensive care unit

MODS: 

Multiple organ dysfunction score

SIRS: 

Systemic inflammatory response syndrome

SOFA: 

Sepsis Organ Failure Assessment

SSC: 

Surviving Sepsis Campaign

Declarations

Acknowledgements

MS-H and MS acknowledge the support of the UK National Institute for Health Research (NIHR) Biomedical Research Centre scheme. MS is a recipient of a UK NIHR Senior Investigator Fellowship.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Intensive Care Medicine, 1st Floor, East Wing, St Thomas’ Hospital, Guy’s and St Thomas’ NHS Foundation Trust
(2)
Division of Asthma, Allergy and Lung Biology, King’s College London
(3)
Laboratory of Clinical Epidemiology and GiViTI Coordinating Centre, IRCCS-Istituto di Ricerche Farmacologiche “Mario Negri”
(4)
Paul-Martini-Research Group for Clinical Sepsis Research, Center for Clinical Studies, Jena University Hospital
(5)
Department of Intensive Care and Medicine, Austin Health
(6)
Department of Intensive Care Medicine, Hôpital Raymond Poincaré (AP-HP), Laboratory of Cell Death, Inflammation & Infection, UMR1173 University of Versailles SQY & INSERM
(7)
Departments of Pediatrics and Molecular Medicine, Hofstra-North Shore-Long Island Jewish-Hofstra School of Medicine
(8)
Feinstein Institute for Medical Research
(9)
Bloomsbury Institute of Intensive Care Medicine, University College London

References

  1. Shankar-Hari M, Deutschman CS, Singer M. Do we need a new definition of sepsis? Intensive Care Med. 2015;41:909–11.PubMedView ArticleGoogle Scholar
  2. Rubenfeld GD. Epidemiology of acute lung injury. Crit Care Med. 2003;31(4 Suppl):S276–84.PubMedView ArticleGoogle Scholar
  3. Xiao W, Mindrinos MN, Seok J, Cuschieri J, Cuenca AG, Gao H, et al. A genomic storm in critically injured humans. J Exp Med. 2011;208:2581–90.PubMedPubMed CentralView ArticleGoogle Scholar
  4. Takasu O, Gaut JP, Watanabe E, To K, Fagley RE, Sato B, et al. Mechanisms of cardiac and renal dysfunction in patients dying of sepsis. Am J Respir Crit Care Med. 2013;187:509–17.PubMedPubMed CentralView ArticleGoogle Scholar
  5. Marshall JC. Sepsis: rethinking the approach to clinical research. J Leukoc Biol. 2008;83:471–82.PubMedView ArticleGoogle Scholar
  6. Kimmoun A, Ducrocq N, Mory S, Delfosse R, Muller L, Perez P, et al. Cardiac contractile reserve parameters are related to prognosis in septic shock. Biomed Res Int. 2013;2013:930673.PubMedPubMed CentralView ArticleGoogle Scholar
  7. Zanotti-Cavazzoni SL, Hollenberg SM. Cardiac dysfunction in severe sepsis and septic shock. Curr Opin Crit Care. 2009;15:392–7.PubMedView ArticleGoogle Scholar
  8. Vincent JL, De Backer D. Circulatory shock. N Engl J Med. 2013;369:1726–34.PubMedView ArticleGoogle Scholar
  9. Abraham E, Shoemaker WC, Bland RD, Cobo JC. Sequential cardiorespiratory patterns in septic shock. Crit Care Med. 1983;11:799–803.PubMedView ArticleGoogle Scholar
  10. Vincent JL, Gris P, Coffernils M, Leon M, Pinsky M, Reuse C, et al. Myocardial depression characterizes the fatal course of septic shock. Surgery. 1992;111:660–7.PubMedGoogle Scholar
  11. Hotchkiss RS, Karl IE. Reevaluation of the role of cellular hypoxia and bioenergetic failure in sepsis. JAMA. 1992;267:1503–10.PubMedView ArticleGoogle Scholar
  12. Kreymann G, Grosser S, Buggisch P, Gottschall C, Matthaei S, Greten H. Oxygen consumption and resting metabolic rate in sepsis, sepsis syndrome, and septic shock. Crit Care Med. 1993;21:1012–9.PubMedView ArticleGoogle Scholar
  13. Carre JE, Orban JC, Re L, Felsmann K, Iffert W, Bauer M, et al. Survival in critical illness is associated with early activation of mitochondrial biogenesis. Am J Respir Crit Care Med. 2010;182:745–51.PubMedPubMed CentralView ArticleGoogle Scholar
  14. Langley RJ, Tsalik EL, van Velkinburgh JC, Glickman SW, Rice BJ, Wang C, et al. An integrated clinico-metabolomic model improves prediction of death in sepsis. Sci Transl Med. 2013;5:195ra95.PubMedPubMed CentralGoogle Scholar
  15. De Backer D, Orbegozo Cortes D, Donadello K, Vincent JL. Pathophysiology of microcirculatory dysfunction and the pathogenesis of septic shock. Virulence. 2014;5:73–9.PubMedPubMed CentralView ArticleGoogle Scholar
  16. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101:1644–55.PubMedView ArticleGoogle Scholar
  17. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2001;2003(31):1250–6.Google Scholar
  18. Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41:580–637.PubMedView ArticleGoogle Scholar
  19. Iwashyna TJ, Govindan S. Did they just prove that a diagnosis of “septic shock” is meaningless? Am J Respir Crit Care Med. 2014;189:1156–7.PubMedPubMed CentralView ArticleGoogle Scholar
  20. Leligdowicz A, Dodek PM, Norena M, Wong H, Kumar A, Kumar A, et al. Association between source of infection and hospital mortality in patients who have septic shock. Am J Respir Crit Care Med. 2014;189:1204–13.PubMedView ArticleGoogle Scholar
  21. Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med. 2013;41:1167–74.PubMedView ArticleGoogle Scholar
  22. Levy B, Gibot S, Franck P, Cravoisy A, Bollaert PE. Relation between muscle Na + K+ ATPase activity and raised lactate concentrations in septic shock: a prospective study. Lancet. 2005;365:871–5.PubMedView ArticleGoogle Scholar
  23. Vander Heiden MG, Cantley LC, Thompson CB. Understanding the Warburg effect: the metabolic requirements of cell proliferation. Science. 2009;324:1029–33.PubMedPubMed CentralView ArticleGoogle Scholar
  24. Tapia P, Soto D, Bruhn A, Alegria L, Jarufe N, Luengo C, et al. Impairment of exogenous lactate clearance in experimental hyperdynamic septic shock is not related to total liver hypoperfusion. Crit Care. 2015;19:188.PubMedPubMed CentralView ArticleGoogle Scholar
  25. Gore DC, Jahoor F, Hibbert JM, DeMaria EJ. Lactic acidosis during sepsis is related to increased pyruvate production, not deficits in tissue oxygen availability. Ann Surg. 1996;224:97–102.PubMedPubMed CentralView ArticleGoogle Scholar
  26. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem. 2008;54:17–23.PubMedView ArticleGoogle Scholar
  27. Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, et al. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010;36:222–31.PubMedPubMed CentralView ArticleGoogle Scholar
  28. Mouncey PR, Osborn TM, Power GS, Harrison DA, Sadique MZ, Grieve RD, et al. Trial of early, goal-directed resuscitation for septic shock. N Engl J Med. 2015;372:1301–11.PubMedView ArticleGoogle Scholar
  29. Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015;372:1629–38.PubMedView ArticleGoogle Scholar
  30. Marshall JC, Cook DJ, Christou NV, Bernard GR, Sprung CL, Sibbald WJ. Multiple organ dysfunction score: a reliable descriptor of a complex clinical outcome. Crit Care Med. 1995;23:1638–52.PubMedView ArticleGoogle Scholar
  31. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22:707–10.PubMedView ArticleGoogle Scholar
  32. Cook R, Cook D, Tilley J, Lee K, Marshall J, Canadian Critical Care Trials Group. Multiple organ dysfunction: baseline and serial component scores. Crit Care Med. 2001;29:2046–50.PubMedView ArticleGoogle Scholar
  33. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207.PubMedPubMed CentralView ArticleGoogle Scholar
  34. Klein Klouwenberg PM, Ong DS, Bonten MJ, Cremer OL. Classification of sepsis, severe sepsis and septic shock: the impact of minor variations in data capture and definition of SIRS criteria. Intensive Care Med. 2012;38:811–9.PubMedView ArticleGoogle Scholar
  35. Heublein S, Hartmann M, Hagel S, Hutagalung R, Brunkhorst FM. Epidemiology of sepsis in German hospitals derived from administrative databases. 6th International Congress on Sepsis and Multiorgan Dysfunction, September 4-6, Weimar. Infection. 2013;41:7–23.Google Scholar
  36. Damuth E, Mitchell JA, Bartock JL, Roberts BW, Trzeciak S. Long-term survival of critically ill patients treated with prolonged mechanical ventilation: a systematic review and meta-analysis. Lancet Respir Med. 2015;3:544–53.PubMedView ArticleGoogle Scholar
  37. Rhee C, Gohil S, Klompas M. Regulatory mandates for sepsis care—reasons for caution. N Engl J Med. 2014;370:1673–6.PubMedView ArticleGoogle Scholar
  38. Feinstein AR, Sosin DM, Wells CK. The Will Rogers phenomenon. Stage migration and new diagnostic techniques as a source of misleading statistics for survival in cancer. N Engl J Med. 1985;312:1604–8.PubMedView ArticleGoogle Scholar
  39. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inform Manage. 2004;42:15–29.View ArticleGoogle Scholar

Copyright

© Shankar-Hari et al. 2015