Quality of reporting is not the same as quality of methodology
Alvaro Sanabria, Department of Surgery-Pontificia Universidad Javeriana. Bogota, Colombia
2 October 2005
Respected Editor
Critical Care
I read the interesting article of Delaney et al.1 and I would like to make some comments.
First, authors made an evaluation of the quality of “reporting” the meta-analysis and not about the quality of “methodology” of those studies, which are different concepts. It is clear that reporting of a study, be or not a meta-analysis should suggest a flaw in methodology, but this assumption has not been proved. 2 The OQAQ instrument evaluates reporting of the study3, so authors’ conclusion should be limited to this area.
Second, the OQAQ instrument has terms that can be very variable or hard to interpret as “reasonably comprehensive” or “avoided”. If authors want to find a relation between bad quality of reporting and bad quality of the methodology, they must repeat the search strategy for the clinical question stated on the original meta-analysis, find the articles not included, include them in a quality evaluation, make a re-analysis of results and find a systematic difference in conclusions. I couldn’t find this strategy in Delaney’s report.
Besides, I would like to see if the journal where they were published has any affect on the quality of reporting. I just use the tables provided by authors and found a significant statistical difference between those systematic reviews supported by the Cochrane Collaboration and those supported by other institutions. (p<0.001, Kruskall-Wallis test)
Finally, I share the worry of authors for the quality of meta-analysis and the common practice of some researchers to name meta-analysis, studies that are not.4 However, they must be careful when they want to conclude that all critical care meta-analysis published has a bad methodological quality, as is stated on the article. As Aristotle said, the wise man has to distinguish between things that look like from those that really are.
Alvaro Sanabria MD, MSc
General and Trauma Surgeon
Department of Surgery
Pontificia Universidad Javeriana-Hospital Universitario San Ignacio
Bogota, Colombia
References
1. Delaney A, Bagshaw S, Ferland A et al. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 2005; 9:R575-R582.
2. Huwiler-Muntener K, Juni P, Junker C et al. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002; 287:2801-2804.
3. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991; 44:1271-1278.
4. Ioannidis JP, Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qual Improv 1999; 25:462-469.
Competing interests
I have been reviewer of systematic reviews from the Cochrane Collaboration
Critical Care
Quality of reporting is not the same as quality of methodology
2 October 2005
Respected Editor
Critical Care
I read the interesting article of Delaney et al.1 and I would like to make some comments.
First, authors made an evaluation of the quality of “reporting” the meta-analysis and not about the quality of “methodology” of those studies, which are different concepts. It is clear that reporting of a study, be or not a meta-analysis should suggest a flaw in methodology, but this assumption has not been proved. 2 The OQAQ instrument evaluates reporting of the study3, so authors’ conclusion should be limited to this area.
Second, the OQAQ instrument has terms that can be very variable or hard to interpret as “reasonably comprehensive” or “avoided”. If authors want to find a relation between bad quality of reporting and bad quality of the methodology, they must repeat the search strategy for the clinical question stated on the original meta-analysis, find the articles not included, include them in a quality evaluation, make a re-analysis of results and find a systematic difference in conclusions. I couldn’t find this strategy in Delaney’s report.
Besides, I would like to see if the journal where they were published has any affect on the quality of reporting. I just use the tables provided by authors and found a significant statistical difference between those systematic reviews supported by the Cochrane Collaboration and those supported by other institutions. (p<0.001, Kruskall-Wallis test)
Finally, I share the worry of authors for the quality of meta-analysis and the common practice of some researchers to name meta-analysis, studies that are not.4 However, they must be careful when they want to conclude that all critical care meta-analysis published has a bad methodological quality, as is stated on the article. As Aristotle said, the wise man has to distinguish between things that look like from those that really are.
Alvaro Sanabria MD, MSc
General and Trauma Surgeon
Department of Surgery
Pontificia Universidad Javeriana-Hospital Universitario San Ignacio
Bogota, Colombia
References
1. Delaney A, Bagshaw S, Ferland A et al. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 2005; 9:R575-R582.
2. Huwiler-Muntener K, Juni P, Junker C et al. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002; 287:2801-2804.
3. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991; 44:1271-1278.
4. Ioannidis JP, Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qual Improv 1999; 25:462-469.
Competing interests
I have been reviewer of systematic reviews from the Cochrane Collaboration