Skip to main content

Developing a feasible and valid scoring system for critically ill patients in resource-limited settings

The Original Article was published on 17 October 2017

In the study by Haniffa et al., the authors have brilliantly proposed “TropICS” as the first multinational prognostic model for critically ill patients in resource-limited settings. The study used data from four South East Asian nations, involving a large number of unselected ICU patients for development and validation of three prognostic models. Of the three proposed models, TropICS performed better than APACHE II and SAPS II in terms of discrimination and calibration. It is interesting to see them test the three models, which varied in inclusion of clinical, laboratory and treatment variables, which were selected based on multivariate logistic regression. The models have tried to address the variable availability and affordability of the parameters measured [1].

However, it would be prudent to note that the complete case availability for APACHE II was only 15% [1]. Moreover, as diagnostic, therapeutic, and prognostic techniques in the ICU evolve over time, the scoring systems need to be updated [2]. Newer APACHE versions, such as APACHE IV, which is based on a larger database and in which the selection of variables and their weights is based on multiple logistic regression, perform better than older versions [3]. These factors together might have caused the lower performance of APACHE II compared to the other proposed models.

Disease-specific scoring systems are increasingly being used. The SOFA score has been proposed to quantify organ dysfunction, as per Sepsis-3 [4]. Some of the parameters of the SOFA score like PaO2, serum creatinine, and bilirubin level have poor availability (less than 50%) in the databases of the new models, raising questions about the feasibility and validity of using the SOFA score to quantify organ dysfunction in resource-poor settings [1]. There may be a need to develop a simpler and more feasible scoring system to recognize sepsis in resource-limited settings [5]. Laboratory variables of TropICS, blood urea and hemoglobin level, were available for only 50% of the patients [1]. Considering the wide variability of availability of resources between low and middle-income countries (LMICs) and even within the same nation, TropICS needs to be validated across these settings before assuming global applicability in places with limited resources.

Rather than depending on a single score at admission to ICU, change in score over time (like the delta SOFA score) may reflect the progression of organ dysfunction over time and can be helpful for better prognostication [2]. Future studies may attempt to explore the utility of the delta score for TropICS as well.

Authors’ response

We would like to thank Dr. Shrestha for his insightful comments. Our systematic review attributes the variable performance of prognostic models in LMICs to three main factors: missing data and subsequent normal imputation; differing endpoints due to unknown hospital outcomes; and the inadvertent inclusion of paediatric patients [6]. Other reasons more commonly attributed to poor performance of prognostic models in such settings include differing case-mix when compared to high income countries (HICs) and a predominantly healthy and younger population. These are, however, often difficult to evaluate given the limited investment in infrastructure (for example medical registries) enabling the systematic collection of data. We agree that the reason for the poor performance of APACHE II in our dataset may be due to missing data [7]. However, for the same reason, newer and more complicated versions such as APACHE III or IV are unlikely to fare any better given their greater data collection burden. It is also important to note that such missingness is not limited to LMICs or indeed to the ICU [8]. The development of simplified models such as TropICS, R-MPM and qSOFA can be seen as attempts to overcome this impediment.

As variability between and within countries is likely to persist, and as all models require regular validation and refinement (for example by recalibration), prognostic model selection and assessment are perhaps better guided, at least initially, by data availability; by matching models and their variants to the degree of missingness [7]. Subsequent validation using the classic statistical tests of discrimination, calibration and accuracy can then follow. Further research to investigate the impact of the extent of missingness on prognostic models and strategies to mitigate this seemingly ubiquitous problem, beyond traditional statistical methods, is ongoing in multiple LMIC settings by our group.

In addition to the quest to refine statistical methodology to improve model performance, we recommend that the validation of such tools should be seen in the context of ‘real world performance and impact on patient outcomes’ [9]. To have greater applicability to improving the quality and equity of care provided, the priorities of clinicians, researchers and administrators in the target setting need to be better understood. Could the biggest barrier to the validity and applicability of such tools be the disconnect of the necessary stakeholders from processes established as indispensable in HICs: benchmarking, audit and quality improvement, and risk stratification for clinical trials?

Abbreviations

APACHE:

Acute Physiology and Chronic Health Evaluation

ICU:

Intensive Care Unit

SAPS:

Simplified Acute Physiology Score

SOFA:

Sequential Organ Failure Assessment.

TropICS:

Tropical Intensive Care Score

References

  1. Haniffa R, Mukaka M, Munasinghe SB, et al. Simplified prognostic model for critically ill patients in resource limited settings in South Asia. Crit Care. 2017;21:250.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med. 2006;34:1297–310.

    Article  PubMed  Google Scholar 

  4. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Shrestha GS. Composition of the sepsis definitions task force. JAMA. 2016;316:459–60.

    Article  PubMed  Google Scholar 

  6. Haniffa R, Isaam I, De Silva AP, Dondorp AM, De Keizer NF. Performance of critical care prognostic scoring systems in low and middle income countries - a systematic review. Critical Care. doi:10.1186/s13054-017-1930-8.

  7. Haniffa R, De Silva AP, Weerathunga P, Mukaka M, Athapattu P, Munasinghe S, Mahesh B, Mahipala P, De Silva T, Abayadeera A, Jayasinghe S. Applicability of the APACHE II model to a lower middle income country. J Crit Care. 2017;42:178–83.

    Article  PubMed  Google Scholar 

  8. Afessa B, Keegan MT, Gajic O, Hubmayr RD, Peters SG. The influence of missing components of the Acute Physiology Score of APACHE III on the measurement of ICU performance. Intensive Care Med. 2005;31(11):1537–43.

    Article  PubMed  Google Scholar 

  9. Baker T, Gerdin M. The clinical usefulness of prognostic prediction models in critical illness. Eur J Intern Med. 2017;45:37–40.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

None.

Funding

None.

Availability of data and materials

Not applicable.

Author information

Authors and Affiliations

Authors

Contributions

The author contributed to concept, designing, literature search, manuscript writing, editing and review.

Corresponding author

Correspondence to Gentle Sunder Shrestha.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The author declares that he/she has no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional information

See related research by Haniffa et al., https://ccforum.biomedcentral.com/articles/10.1186/s13054-017-1843-6

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shrestha, G.S. Developing a feasible and valid scoring system for critically ill patients in resource-limited settings. Crit Care 22, 2 (2018). https://doi.org/10.1186/s13054-017-1902-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13054-017-1902-z