Skip to main content

Advertisement

Trial sequential analysis suggested the potential overestimated effect of carbonic anhydrase inhibitor for respiratory failure and metabolic alkalosis

Article metrics

The original article was published in Critical Care 2018 22:275

Meta-analyses of randomized controlled trials (RCTs) used to be considered as the optimum evidence to guide clinical practices. Generally, a high-quality meta-analysis with conclusive information should meet the minimum requirements of a well-conducted RCT, which includes prospective protocol development, limitation of bias, and adequate sample size [1]. Conversely, meta-analyses based on limited RCTs may trigger the potential overestimation of the authentic intervention effect owing to weak statistical power [1]. More interestingly, increasing studies indicated that pooled results with false positive were frequently existed in published meta-analyses including many Cochrane ones [2, 3].

Trial sequential analysis (TSA) was introduced to monitor potential random error, false positive, and false negative in meta-analyses of RCTs [3]. Moreover, it was recommended that TSA should be performed to assess the “imprecision” of outcomes of interest in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) [4]. A recent meta-analysis indicated that carbonic anhydrase inhibitor (CAI) may have a positive effect on respiratory failure and metabolic alkalosis [5]. Considering that limited trials with small information size included in the study, we assumed that the effect of CAI for respiratory failure and metabolic alkalosis may be overestimated. Subsequently, we performed TSA for one of outcomes (i.e. PaCO2) with the most included RCTs to estimate whether the evidence is enough reliable and credible. TSA in Fig. 1 showed that the cumulative Z-curve did not cross the trial sequential monitoring boundary for benefit and the required information size boundary, which suggested that the current evidence (the positive effect of CAI on PaCO2) was inconclusive. In addition, TSA on PaCO2 showed that the required information size (347 patients) is not reached due to weak statistical power. So, the effects of CAI therapy for patients with respiratory failure and metabolic alkalosis may very likely be overrated.

Fig. 1
figure1

Trial sequential analysis (TSA) of 5 trials testing the effect of CAI therapy on PaCO2 in patients with respiratory failure and metabolic alkalosis. TSA of 5 trials (black square fill icons) showing that the line of cumulative Z-curve crossed the conventional boundary for benefit, but not the lines of the trial sequential monitoring boundary for benefit and required information size (RIS), which establish inconclusive evidence and suggest that further trials are needed. (X-axis, number of patients; Y-axis, cumulative Z-score; horizontal green dotted lines, conventional boundaries for benefit or harm; sloping full red lines with black square fill icons, trial sequential monitoring boundaries for benefit or harm; full blue line with black square fill icons, Z-curve; vertical red full line, required information size boundary)

Collectively, for meta-analyses of RCTs with limited information size, TSA is a good choice to monitor the potential overestimation of the overall pooled effect. Furthermore, it is worthwhile to further discussion whether TSA should be routinely performed in meta-analyses of RCTs.

Authors' response

AuthorGroup Author AuthorName GivenNameBassem GivenNameY. FamilyNameTanios Author AuthorName GivenNamePierre GivenNameK. FamilyNameBou-Khalil Author AuthorName GivenNameSamir GivenNameS. FamilyNameMallat Author AuthorName GivenNameElie GivenNameA. FamilyNameAkl

We thank Meng-Si Luo et al. for their interest in our work, and for highlighting one the limitations of the literature in this field, namely the limited number of published randomized controlled trials (RCTS). In our conclusion, we do acknowledge this important limitation and the need for larger, well designed, and RCTs addressing clinically important outcomes such as mortality, duration of hospital stay, and duration of mechanical ventilation. [5]

Meng-Si Luo et al. propose the use of Trial Sequential Analysis (TSA) to assess the ‘imprecision’ of outcomes. In a recent expert panel consensus statement, the Cochrane scientific committee recommended against the use of sequential methods for the main analysis, or to draw main conclusions. The statement encourages authors to interpret evidence based on the estimated magnitude of the effect of intervention and its uncertainty (usually quantified using a confidence interval), rather than focusing primarily on the rejection of the null hypothesis of no treatment effect (like that used by Meng-Si Luo et al. in their TSA) [6].

Our approach was consistent with the recent Cochrane recommendations, as we did not draw binary interpretation of the effect estimate as “significant” or “non-significant”. Instead, we presented our results using a confidence interval and assessed heterogeneity using the I2 statistic, then used the GRADE methodology to rate the certainty of evidence. In applying GRADE, we judged imprecision as a reason for rating down the certainty of evidence for all the clinically important outcomes. [6, 7] If we were to analyze the certainty of evidence for the change in PaCO2 using the GRADE methodology, we would have similarly rated down the certainly of evidence for imprecision (and further down due to significant heterogeneity resulting in a low certainty evidence). Indeed, our meta-analysis found a mean reduction of - 4.98 mmHg in the carbonic anhydrase inhibitors group (95% CI -9.66 to -0.3; I2 = 95%). The confidence interval includes both values indicating a clinically significant benefit and values indicating no effect [8].

One of the main reasons Cochrane adopted the above recommendation is to “support the decision maker and end user by providing the best and latest evidence, but that interpretation of that evidence should be left to the user to make within their own context.” We are happy our approach was consistent with that recommendation given the aim of our review is to support clinicians providing care to critically ill patients.

Abbreviations

CAI:

Carbonic anhydrase inhibitor

GRADE:

Grading of Recommendations Assessment, Development and Evaluation

RCT:

Randomized clinical trial

TSA:

Trial sequential analysis

References

  1. 1.

    Pogue J, Yusuf S. Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet. 1998;351:47–52.

  2. 2.

    Brok J, Thorlund K, Gluud C, et al. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J Clin Epidemiol. 2008;61:763–9.

  3. 3.

    Imberger G, Thorlund K, Gluud C, et al. False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review. BMJ Open. 2016;6:e011890.

  4. 4.

    Jakobsen JC, Wetterslev J, Winkel P, et al. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.

  5. 5.

    Tanios BY, Omran MO, Noujeim C, et al. Carbonic anhydrase inhibitors in patients with respiratory failure and metabolic alkalosis: a systematic review and meta-analysis of randomized controlled trials. Crit Care. 2018;22:275.

  6. 6.

    Agendas and Minutes: Cochrane Scientific Committee available from Cochrane.org https://methods.cochrane.org/sites/default/files/public/uploads/tsa_expert_panel_guidance_and_recommendation_final.pdf.

  7. 7.

    Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.

  8. 8.

    Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011;64(12):1283–93.

Download references

Acknowledgements

Not application.

Funding

Not applicable.

Availability of data and materials

Not applicable.

Author information

M-SL was responsible for the conception of the letter and wrote the manuscript. H-ZL and G-JH conceived and wrote this manuscript. LW was responsible for the conception of the letter and revised the manuscript. All authors had read and approved this final manuscript.

Correspondence to Guan-Jiang Huang or Lun Wu.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional information

This comment refers to the article available at https://doi.org/10.1186/s13054-018-2207-6

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark