- Letter
- Open access
- Published:
Trial sequential analysis suggested the potential overestimated effect of carbonic anhydrase inhibitor for respiratory failure and metabolic alkalosis
Critical Care volume 23, Article number: 138 (2019)
Meta-analyses of randomized controlled trials (RCTs) used to be considered as the optimum evidence to guide clinical practices. Generally, a high-quality meta-analysis with conclusive information should meet the minimum requirements of a well-conducted RCT, which includes prospective protocol development, limitation of bias, and adequate sample size [1]. Conversely, meta-analyses based on limited RCTs may trigger the potential overestimation of the authentic intervention effect owing to weak statistical power [1]. More interestingly, increasing studies indicated that pooled results with false positive were frequently existed in published meta-analyses including many Cochrane ones [2, 3].
Trial sequential analysis (TSA) was introduced to monitor potential random error, false positive, and false negative in meta-analyses of RCTs [3]. Moreover, it was recommended that TSA should be performed to assess the “imprecision” of outcomes of interest in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) [4]. A recent meta-analysis indicated that carbonic anhydrase inhibitor (CAI) may have a positive effect on respiratory failure and metabolic alkalosis [5]. Considering that limited trials with small information size included in the study, we assumed that the effect of CAI for respiratory failure and metabolic alkalosis may be overestimated. Subsequently, we performed TSA for one of outcomes (i.e. PaCO2) with the most included RCTs to estimate whether the evidence is enough reliable and credible. TSA in Fig. 1 showed that the cumulative Z-curve did not cross the trial sequential monitoring boundary for benefit and the required information size boundary, which suggested that the current evidence (the positive effect of CAI on PaCO2) was inconclusive. In addition, TSA on PaCO2 showed that the required information size (347 patients) is not reached due to weak statistical power. So, the effects of CAI therapy for patients with respiratory failure and metabolic alkalosis may very likely be overrated.
Collectively, for meta-analyses of RCTs with limited information size, TSA is a good choice to monitor the potential overestimation of the overall pooled effect. Furthermore, it is worthwhile to further discussion whether TSA should be routinely performed in meta-analyses of RCTs.
Authors' response
We thank Meng-Si Luo et al. for their interest in our work, and for highlighting one the limitations of the literature in this field, namely the limited number of published randomized controlled trials (RCTS). In our conclusion, we do acknowledge this important limitation and the need for larger, well designed, and RCTs addressing clinically important outcomes such as mortality, duration of hospital stay, and duration of mechanical ventilation. [5]
Meng-Si Luo et al. propose the use of Trial Sequential Analysis (TSA) to assess the ‘imprecision’ of outcomes. In a recent expert panel consensus statement, the Cochrane scientific committee recommended against the use of sequential methods for the main analysis, or to draw main conclusions. The statement encourages authors to interpret evidence based on the estimated magnitude of the effect of intervention and its uncertainty (usually quantified using a confidence interval), rather than focusing primarily on the rejection of the null hypothesis of no treatment effect (like that used by Meng-Si Luo et al. in their TSA) [6].
Our approach was consistent with the recent Cochrane recommendations, as we did not draw binary interpretation of the effect estimate as “significant” or “non-significant”. Instead, we presented our results using a confidence interval and assessed heterogeneity using the I2 statistic, then used the GRADE methodology to rate the certainty of evidence. In applying GRADE, we judged imprecision as a reason for rating down the certainty of evidence for all the clinically important outcomes. [6, 7] If we were to analyze the certainty of evidence for the change in PaCO2 using the GRADE methodology, we would have similarly rated down the certainly of evidence for imprecision (and further down due to significant heterogeneity resulting in a low certainty evidence). Indeed, our meta-analysis found a mean reduction of - 4.98 mmHg in the carbonic anhydrase inhibitors group (95% CI -9.66 to -0.3; I2 = 95%). The confidence interval includes both values indicating a clinically significant benefit and values indicating no effect [8].
One of the main reasons Cochrane adopted the above recommendation is to “support the decision maker and end user by providing the best and latest evidence, but that interpretation of that evidence should be left to the user to make within their own context.” We are happy our approach was consistent with that recommendation given the aim of our review is to support clinicians providing care to critically ill patients.
Abbreviations
- CAI:
-
Carbonic anhydrase inhibitor
- GRADE:
-
Grading of Recommendations Assessment, Development and Evaluation
- RCT:
-
Randomized clinical trial
- TSA:
-
Trial sequential analysis
References
Pogue J, Yusuf S. Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet. 1998;351:47–52.
Brok J, Thorlund K, Gluud C, et al. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J Clin Epidemiol. 2008;61:763–9.
Imberger G, Thorlund K, Gluud C, et al. False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review. BMJ Open. 2016;6:e011890.
Jakobsen JC, Wetterslev J, Winkel P, et al. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.
Tanios BY, Omran MO, Noujeim C, et al. Carbonic anhydrase inhibitors in patients with respiratory failure and metabolic alkalosis: a systematic review and meta-analysis of randomized controlled trials. Crit Care. 2018;22:275.
Agendas and Minutes: Cochrane Scientific Committee available from Cochrane.org https://methods.cochrane.org/sites/default/files/public/uploads/tsa_expert_panel_guidance_and_recommendation_final.pdf.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011;64(12):1283–93.
Acknowledgements
Not application.
Funding
Not applicable.
Availability of data and materials
Not applicable.
Author information
Authors and Affiliations
Contributions
M-SL was responsible for the conception of the letter and wrote the manuscript. H-ZL and G-JH conceived and wrote this manuscript. LW was responsible for the conception of the letter and revised the manuscript. All authors had read and approved this final manuscript.
Corresponding authors
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional information
This comment refers to the article available at https://doi.org/10.1186/s13054-018-2207-6
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Luo, MS., Li, HZ., Huang, GJ. et al. Trial sequential analysis suggested the potential overestimated effect of carbonic anhydrase inhibitor for respiratory failure and metabolic alkalosis. Crit Care 23, 138 (2019). https://doi.org/10.1186/s13054-019-2384-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13054-019-2384-y