Skip to main content

Advertisement

  • Letter
  • Open Access

Trial sequential analysis suggested the potential overestimated effect of carbonic anhydrase inhibitor for respiratory failure and metabolic alkalosis

Critical Care201923:138

https://doi.org/10.1186/s13054-019-2384-y

  • Received: 20 December 2018
  • Accepted: 6 March 2019
  • Published:

The original article was published in Critical Care 2018 22:275

Meta-analyses of randomized controlled trials (RCTs) used to be considered as the optimum evidence to guide clinical practices. Generally, a high-quality meta-analysis with conclusive information should meet the minimum requirements of a well-conducted RCT, which includes prospective protocol development, limitation of bias, and adequate sample size [1]. Conversely, meta-analyses based on limited RCTs may trigger the potential overestimation of the authentic intervention effect owing to weak statistical power [1]. More interestingly, increasing studies indicated that pooled results with false positive were frequently existed in published meta-analyses including many Cochrane ones [2, 3].

Trial sequential analysis (TSA) was introduced to monitor potential random error, false positive, and false negative in meta-analyses of RCTs [3]. Moreover, it was recommended that TSA should be performed to assess the “imprecision” of outcomes of interest in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) [4]. A recent meta-analysis indicated that carbonic anhydrase inhibitor (CAI) may have a positive effect on respiratory failure and metabolic alkalosis [5]. Considering that limited trials with small information size included in the study, we assumed that the effect of CAI for respiratory failure and metabolic alkalosis may be overestimated. Subsequently, we performed TSA for one of outcomes (i.e. PaCO2) with the most included RCTs to estimate whether the evidence is enough reliable and credible. TSA in Fig. 1 showed that the cumulative Z-curve did not cross the trial sequential monitoring boundary for benefit and the required information size boundary, which suggested that the current evidence (the positive effect of CAI on PaCO2) was inconclusive. In addition, TSA on PaCO2 showed that the required information size (347 patients) is not reached due to weak statistical power. So, the effects of CAI therapy for patients with respiratory failure and metabolic alkalosis may very likely be overrated.
Fig. 1
Fig. 1

Trial sequential analysis (TSA) of 5 trials testing the effect of CAI therapy on PaCO2 in patients with respiratory failure and metabolic alkalosis. TSA of 5 trials (black square fill icons) showing that the line of cumulative Z-curve crossed the conventional boundary for benefit, but not the lines of the trial sequential monitoring boundary for benefit and required information size (RIS), which establish inconclusive evidence and suggest that further trials are needed. (X-axis, number of patients; Y-axis, cumulative Z-score; horizontal green dotted lines, conventional boundaries for benefit or harm; sloping full red lines with black square fill icons, trial sequential monitoring boundaries for benefit or harm; full blue line with black square fill icons, Z-curve; vertical red full line, required information size boundary)

Collectively, for meta-analyses of RCTs with limited information size, TSA is a good choice to monitor the potential overestimation of the overall pooled effect. Furthermore, it is worthwhile to further discussion whether TSA should be routinely performed in meta-analyses of RCTs.

Authors' response

  • ,
  • ,
  • and
  • We thank Meng-Si Luo et al. for their interest in our work, and for highlighting one the limitations of the literature in this field, namely the limited number of published randomized controlled trials (RCTS). In our conclusion, we do acknowledge this important limitation and the need for larger, well designed, and RCTs addressing clinically important outcomes such as mortality, duration of hospital stay, and duration of mechanical ventilation. [5]

    Meng-Si Luo et al. propose the use of Trial Sequential Analysis (TSA) to assess the ‘imprecision’ of outcomes. In a recent expert panel consensus statement, the Cochrane scientific committee recommended against the use of sequential methods for the main analysis, or to draw main conclusions. The statement encourages authors to interpret evidence based on the estimated magnitude of the effect of intervention and its uncertainty (usually quantified using a confidence interval), rather than focusing primarily on the rejection of the null hypothesis of no treatment effect (like that used by Meng-Si Luo et al. in their TSA) [6].

    Our approach was consistent with the recent Cochrane recommendations, as we did not draw binary interpretation of the effect estimate as “significant” or “non-significant”. Instead, we presented our results using a confidence interval and assessed heterogeneity using the I2 statistic, then used the GRADE methodology to rate the certainty of evidence. In applying GRADE, we judged imprecision as a reason for rating down the certainty of evidence for all the clinically important outcomes. [6, 7] If we were to analyze the certainty of evidence for the change in PaCO2 using the GRADE methodology, we would have similarly rated down the certainly of evidence for imprecision (and further down due to significant heterogeneity resulting in a low certainty evidence). Indeed, our meta-analysis found a mean reduction of - 4.98 mmHg in the carbonic anhydrase inhibitors group (95% CI -9.66 to -0.3; I2 = 95%). The confidence interval includes both values indicating a clinically significant benefit and values indicating no effect [8].

    One of the main reasons Cochrane adopted the above recommendation is to “support the decision maker and end user by providing the best and latest evidence, but that interpretation of that evidence should be left to the user to make within their own context.” We are happy our approach was consistent with that recommendation given the aim of our review is to support clinicians providing care to critically ill patients.

    Notes

    Abbreviations

    CAI: 

    Carbonic anhydrase inhibitor

    GRADE: 

    Grading of Recommendations Assessment, Development and Evaluation

    RCT: 

    Randomized clinical trial

    TSA: 

    Trial sequential analysis

    Declarations

    Acknowledgements

    Not application.

    Funding

    Not applicable.

    Availability of data and materials

    Not applicable.

    Authors’ contributions

    M-SL was responsible for the conception of the letter and wrote the manuscript. H-ZL and G-JH conceived and wrote this manuscript. LW was responsible for the conception of the letter and revised the manuscript. All authors had read and approved this final manuscript.

    Ethics approval and consent to participate

    Not applicable.

    Consent for publication

    Not applicable.

    Competing interests

    The authors declare that they have no competing interests.

    Publisher’s Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

    Authors’ Affiliations

    (1)
    Department of Anesthesiology, Zhongshan Hospital of Traditional Chinese Medicine, Affiliated to Guangzhou University of Chinese Medicine, 3 Kangxin Road, Zhongshan, 528400, Guangdong, China
    (2)
    Department of Orthopaedics, The Fifth Affiliated Hospital of Sun Yat-Sen University, 52 Meihua East Road, Xiangzhou District, Zhuhai, 519000, Guangdong, China
    (3)
    Department of Otorhinolaryngology, The Second Affiliated Hospital, School of Medicine, Zhejiang University, 88 Jiefang Road, Hangzhou, 310009, Zhejiang, China

    References

    1. Pogue J, Yusuf S. Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet. 1998;351:47–52.View ArticleGoogle Scholar
    2. Brok J, Thorlund K, Gluud C, et al. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J Clin Epidemiol. 2008;61:763–9.View ArticleGoogle Scholar
    3. Imberger G, Thorlund K, Gluud C, et al. False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review. BMJ Open. 2016;6:e011890.View ArticleGoogle Scholar
    4. Jakobsen JC, Wetterslev J, Winkel P, et al. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.View ArticleGoogle Scholar
    5. Tanios BY, Omran MO, Noujeim C, et al. Carbonic anhydrase inhibitors in patients with respiratory failure and metabolic alkalosis: a systematic review and meta-analysis of randomized controlled trials. Crit Care. 2018;22:275.View ArticleGoogle Scholar
    6. Agendas and Minutes: Cochrane Scientific Committee available from Cochrane.org https://methods.cochrane.org/sites/default/files/public/uploads/tsa_expert_panel_guidance_and_recommendation_final.pdf.
    7. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.Google Scholar
    8. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011;64(12):1283–93.Google Scholar

    Copyright

    © The Author(s). 2019

    Advertisement