Fallaces sunt rerum species et hominum spes fallunt (appearances are deceptive and betray the hopes of men)

We read with interest the study by Van den Akker and colleagues [1] on the association between mechanical ventilation, and renal failure. We consider it is a very interesting study, but feel some comments are needed.

The core of a meta-analysis is the combination of results across different studies. A major concern about meta-analysis is the range of the studies included in the sample because the trials included may introduce large inhomogeneity. The challenge consists in deciding which set of studies is ‘combinable’ [2].

In the meta-analysis by Van den Akker and colleagues [1] the risk of bias is very high due to sampling errors of the included studies. The authors selected trials that are completely different from each other: different diseases, different clinical severity, different therapeutic treatments. Besides this, the authors neglected the results of the statistical analysis, which showed great heterogeneity within the sample. To evaluate this heterogeneity the authors used Cochran’s Q statistic, and the I² statistic. Both the Q test and I² statistic of all included patients were indicative of high heterogeneity [3].

When the P-value (heterogeneity test) is significant, two approaches are possible: first, avoid summarizing the results and look for reasons for the heterogeneity; second, utilize ‘the random effects model’. Van den Akker and colleagues chose the latter approach, failing to check whether the heterogeneity was due to differences in the clinical studies, methodological issues, and/or publication bias. As stated by DerSimonian and Laird [4], the ‘random effects model is not a cure for difficulty in generalising the results of meta-analysis to the real-world situation’. From our point of view, the methodological errors of the meta-analysis of Van den Akker and colleagues produce misleading results.