Trial sequential analysis suggested the potential overestimated effect of carbonic anhydrase inhibitor for respiratory failure and metabolic alkalosis

Meta-analyses of randomized controlled trials (RCTs) used to be considered as the optimum evidence to guide clinical practices. Generally, a high-quality meta-analysis with conclusive information should meet the minimum requirements of a well-conducted RCT, which includes prospective protocol development, limitation of bias, and adequate sample size [1]. Conversely, meta-analyses based on limited RCTs may trigger the potential overestimation of the authentic intervention effect owing to weak statistical power [1]. More interestingly, increasing studies indicated that pooled results with false positive were frequently existed in published meta-analyses including many Cochrane ones [2, 3].

Trial sequential analysis (TSA) was introduced to monitor potential random error, false positive, and false negative in meta-analyses of RCTs [3]. Moreover, it was recommended that TSA should be performed to assess the “imprecision” of outcomes of interest in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) [4]. A recent meta-analysis indicated that carbonic anhydrase inhibitor (CAI) may have a positive effect on respiratory failure and metabolic alkalosis [5]. Considering that limited trials with small information size included in the study, we assumed that the effect of CAI for respiratory failure and metabolic alkalosis may be overestimated. Subsequently, we performed TSA for one of outcomes (i.e. PaCO₂) with the most included RCTs to estimate whether the evidence is enough reliable and credible. TSA in Fig. 1 showed that the cumulative Z-curve did not cross the trial sequential monitoring boundary for benefit and the required information size boundary, which suggested that the current evidence (the positive effect of CAI on PaCO₂) was inconclusive. In addition, TSA on PaCO₂ showed that the required information size (347 patients) is not reached due to weak statistical power. So, the effects of CAI therapy for patients with respiratory failure and metabolic alkalosis may very likely be overrated.

Fig. 1

Trial sequential analysis (TSA) of 5 trials testing the effect of CAI therapy on PaCO₂ in patients with respiratory failure and metabolic alkalosis. TSA of 5 trials (black square fill icons) showing that the line of cumulative Z-curve crossed the conventional boundary for benefit, but not the lines of the trial sequential monitoring boundary for benefit and required information size (RIS), which establish inconclusive evidence and suggest that further trials are needed. (X-axis, number of patients; Y-axis, cumulative Z-score; horizontal green dotted lines, conventional boundaries for benefit or harm; sloping full red lines with black square fill icons, trial sequential monitoring boundaries for benefit or harm; full blue line with black square fill icons, Z-curve; vertical red full line, required information size boundary)

Collectively, for meta-analyses of RCTs with limited information size, TSA is a good choice to monitor the potential overestimation of the overall pooled effect. Furthermore, it is worthwhile to further discussion whether TSA should be routinely performed in meta-analyses of RCTs.

Authors' response

• Bassem Y. Tanios,
• Pierre K. Bou-Khalil,
• Samir S. Mallat &
• Elie A. Akl 

We thank Meng-Si Luo et al. for their interest in our work, and for highlighting one the limitations of the literature in this field, namely the limited number of published randomized controlled trials (RCTS). In our conclusion, we do acknowledge this important limitation and the need for larger, well designed, and RCTs addressing clinically important outcomes such as mortality, duration of hospital stay, and duration of mechanical ventilation. [5]

Meng-Si Luo et al. propose the use of Trial Sequential Analysis (TSA) to assess the ‘imprecision’ of outcomes. In a recent expert panel consensus statement, the Cochrane scientific committee recommended against the use of sequential methods for the main analysis, or to draw main conclusions. The statement encourages authors to interpret evidence based on the estimated magnitude of the effect of intervention and its uncertainty (usually quantified using a confidence interval), rather than focusing primarily on the rejection of the null hypothesis of no treatment effect (like that used by Meng-Si Luo et al. in their TSA) [6].

Our approach was consistent with the recent Cochrane recommendations, as we did not draw binary interpretation of the effect estimate as “significant” or “non-significant”. Instead, we presented our results using a confidence interval and assessed heterogeneity using the I2 statistic, then used the GRADE methodology to rate the certainty of evidence. In applying GRADE, we judged imprecision as a reason for rating down the certainty of evidence for all the clinically important outcomes. [6, 7] If we were to analyze the certainty of evidence for the change in PaCO2 using the GRADE methodology, we would have similarly rated down the certainly of evidence for imprecision (and further down due to significant heterogeneity resulting in a low certainty evidence). Indeed, our meta-analysis found a mean reduction of - 4.98 mmHg in the carbonic anhydrase inhibitors group (95% CI -9.66 to -0.3; I2 = 95%). The confidence interval includes both values indicating a clinically significant benefit and values indicating no effect [8].

One of the main reasons Cochrane adopted the above recommendation is to “support the decision maker and end user by providing the best and latest evidence, but that interpretation of that evidence should be left to the user to make within their own context.” We are happy our approach was consistent with that recommendation given the aim of our review is to support clinicians providing care to critically ill patients.