Rigorous scoping review of randomized trials in pediatric critical care highlights need for a rigorous rethink

Abstract

The randomized controlled trial (RCT) remains the highest-ranked study design when grading recommendations for clinical practice. In the previous issue of Critical Care, Duffett and colleagues published a scoping review of RCTs in pediatric critical care medicine and identified some serious gaps in the body of research underlying the field. Relatively few published RCTs were identified, and they were mostly small and potentially susceptible to bias. High patient heterogeneity, relatively low prevalence of specific disorders such as acute respiratory distress syndrome or septic shock, along with relatively low mortality rates, all make it difficult to improve this situation without the collaboration of pediatric critical care research networks internationally. Designing a robust RCT that can impact clinical practice has always been challenging. First, one must assess current clinical practice and disease prevalence, refine definitions and measurements, and pilot-test the intervention to be studied. The first step, however, is to rigorously assess what has already been done. This step will be facilitated by the now available, innovative, online, searchable repository of RCTs in pediatric critical care on the Evidence in Pediatric Critical Care website.

The practice of triaging critically ill infants, children, and adolescents into intensive care units staffed by clinicians trained to care for them is widespread. In the previous issue of Critical Care, Duffett and colleagues performed a scoping review of randomized controlled trials (RCTs) in pediatric critical care [1] to systematically assess the body of research in the field. They cast a wide net to identify all English and non-English language publications, and employed nine pairs of reviewers to evaluate each publication and then independently and in duplicate to extract the data and assess the risk of bias. After screening more than 7,500 studies, they included 248 RCTs randomly assigning more than 27,000 children from 31 countries. Comparatively, the authors note that this is a small fraction of the number of RCTs in adult critical care [2]; neonatology has at least that many systematic reviews of RCTs [3]. Unfortunately, 88% of the RCTs they found had a risk of systematic bias potentially threatening study validity [1]. Major obstacles plaguing pediatric critical care clinical research have been discussed before and include high patient heterogeneity, low prevalence of any single disorder, relatively low mortality rates, and high practice variability [4–6]. Given the numerous obstacles to performing high-quality RCTs in critically ill children, the rigorous review by Duffett and colleagues [1] highlights the need to rigorously rethink the RCT process.

Ideally, RCTs should enroll a representative sample of the population in which the results will be applied. The great majority of the trials (82%) Duffett and colleagues [1] identified were performed in a single center; the median number of centers participating in multicenter studies was only five. Sample sizes were generally small; 50% of RCTs randomly assigned fewer than 50 children. Most trials focused on intermediate or surrogate outcomes of unclear clinical importance. Only 43% of the trials reported a target sample size; 32% of these trials were stopped early before recruiting the target. To improve the ability to recruit a larger, more representative sample of children, pediatric critical care national networks have formed in the US, UK, Canada, and Australia-New Zealand and in other regions. The low frequency of disorders such as acute lung injury makes it essential that these international networks work together [7].

The only situation worse than not having RCT level evidence is to initiate an RCT, expose children to the untested intervention(s), and then abandon the study midway or not publish the findings (or both). Sadly, Shamliyan and Kane recently reported that of 2,290 intervention studies involving children identified in ClinicalTrials.gov, only 71% were completed and only 30% of these studies were ever published [8]. Bias in trial reporting may also be a problem in pediatric critical care RCTs [1]. The statistical significance of the outcome could be assessed in only approximately half of the RCTs, and of the 50% of these trials that reported significance, 93% of the time the results favored the experimental intervention [1]. Poor recruitment may underlie some of the abandoned studies. An adaptive research design using a Bayesian statistical approach, with the a priori estimate extrapolated from trials carried out in adults, has been proposed as one way to gain statistical power with fewer children enrolled [9].

Finally, RCTs are not always the optimal way to answer a clinical question. Aside from their extraordinary expense, they can mislead. If trial enrollment drags on too long, the intervention can become obsolete or the epidemiology can change. Protocol adherence suffers when hundreds of centers are involved, each enrolling few patients. Most RCTs also exclude many patients in whom the intervention would eventually be used. It has been argued that rigorous epidemiology studies can be equally important in creating a deeper understanding of the problem under study, often informing clinical practice [10, 11]. The Canadian Critical Care Trials Group programmatic approach to research lays out in detail the hard work leading up to a successful RCT [12]. Step 1 consists of systematically reviewing what is currently known and assessing current practice. Step 2 consists of assessing disease prevalence, identifying risk factors, and evaluating study definitions, metrics, and outcomes. Step 3 consists of assessing RCT feasibility by performing an observational study across potential enrollment sites and piloting the intervention. After all of these steps are completed, it is likely fewer pediatric critical care RCTs will be performed, but those that are will be of higher quality and have more potential impact.

To facilitate public access to the trials they identified, Duffett and colleagues [1] built an innovative online searchable repository [13]. Their spectacular website includes an interactive world map showing, by country, the number of RCTs, the number of children randomly assigned, and the median number of children per RCT. It is easily browsed by indication, intervention, and patient type, thus facilitating literature review and future meta-analyses. The work of Duffett and colleagues [1] is essential and will facilitate the initial steps in any program of research leading up to a successful randomized trial [12], that is, Step 1: to assess what has already been done.