Protocol-directed care in the ICU: making a future generation of intensivists less knowledgeable?

Expanded abstract Citation Prasad M, Holmboe ES, Lipner RS, Hess BJ, Christie JD, Bellamy SL, Rubenfeld GD, Kahn JM. Clinical Protocols and Trainee Knowledge About Mechanical Ventilation. JAMA. 2011; 306(9):935-941. PubMed PMID: 21900133 This is available on http://www.pubmed.gov Background Clinical protocols are associated with improved patient outcomes; however, they may negatively affect medical education by removing trainees from clinical decision making. Methods Objective: To study the relationship between critical care training with mechanical ventilation protocols and subsequent knowledge about ventilator management. Design: A retrospective cohort equivalence study linking a national survey of mechanical ventilation protocol availability with knowledge about mechanical ventilation. Exposure to protocols was defined as high intensity if an intensive care unit had 2 or more protocols for at least 3 years and as low intensity if 0 or 1 protocol. Setting: Accredited US pulmonary and critical care fellowship programs. Subjects: First-time examinees of the American Board of Internal Medicine (ABIM) Critical Care Medicine Certification Examination in 2008 and 2009. Intervention: N/A Outcomes: Knowledge, measured by performance on examination questions specific to mechanical ventilation management, calculated as a mechanical ventilation score using item response theory. The score is standardized to a mean (SD) of 500 (100), and a clinically important difference is defined as 25. Variables included in adjusted analyses were birth country, residency training country, and overall first-attempt score on the ABIM Internal Medicine Certification Examination. Results The 90 of 129 programs (70%) responded to the survey. Seventy seven programs (86%) had protocols for ventilation liberation, 66 (73%) for sedation management, and 54 (60%) for lung-protective ventilation at the time of the survey. Eighty eight (98%) of these programs had trainees who completed the ABIM Critical Care Medicine Certification Examination, totaling 553 examinees. Of these 88 programs, 27 (31%) had 0 protocols, 19 (22%) had 1 protocol, 24 (27%) had 2 protocols, and 18 (20%) had 3 protocols for at least 3 years. 42 programs (48%) were classified as high intensity and 46 (52%) as low intensity, with 304 trainees (55%) and 249 trainees (45%), respectively. In bi-variable analysis, no difference in mean scores was observed in high-intensity (497; 95% CI, 486-507) vs low-intensity programs (497; 95% CI, 485-509). Mean difference was 0 (95% CI, -16 to 16), with a positive value indicating a higher score in the high-intensity group. In multivariable analyses, no association of training was observed in a high-intensity program with mechanical ventilation score (adjusted mean difference, -5.36; 95% CI, -20.7 to 10.0). Conclusions Among first-time ABIM Critical Care Medicine Certification Examination examinees, training in a high-intensity ventilator protocol environment compared with a low-intensity environment was not associated with worse performance on examination questions about mechanical ventilation management.


Background
Clinical protocols are associated with improved patient out comes; however, they may negatively aff ect medical edu ca tion by removing trainees from clinical decision making.

Methods
Objective: To study the relationship between critical care training with mechanical ventilation protocols and subsequent knowledge about ventilator management. Design: A retrospective cohort equivalence study linking a national survey of mechanical ventilation protocol availability with knowledge about mechanical ventilation. Exposure to protocols was defi ned as high intensity if an intensive care unit had 2 or more protocols for at least 3 years and as low intensity if 0 or 1 protocol. Setting: Accredited US pulmonary and critical care fellow ship programs. Subjects: First-time examinees of the American Board of Internal Medicine (ABIM) Critical Care Medicine Certifi cation Examination in 2008 and 2009. Intervention: N/A Outcomes: Knowledge, measured by performance on examination questions specifi c to mechanical ventilation management, calculated as a mechanical ventilation score using item response theory. Th e score is standardized to a mean (SD) of 500 (100), and a clinically important diff erence is defi ned as 25. Variables included in adjusted analyses were birth country, residency training country, and overall fi rst-attempt score on the ABIM Internal Medicine Certifi cation Examination.

Conclusions
Among fi rst-time ABIM Critical Care Medicine Certification Examination examinees, training in a high-intensity ventilator protocol environment compared with a low-intensity environment was not associated with worse performance on examination questions about mechanical ventilation management.

Commentary
Evidence-based protocol-directed care has permeated the practice of mechanical ventilation, especially with re spect to ventilation liberation [1], lung-protective strategies for acute lung injury (ALI) [2] and sedation management [3]. Th ese protocols have been endorsed by multiple organizations and societies including the American College of Chest Physicians, American College of Critical Care Medicine and the Surviving Sepsis Campaign. However, there remains a necessary tension between patient care and experiential learning [4]. Th e institution of protocol-directed care in the intensive care unit (ICU) may be detrimental in that it potentially distances physicians from direct patient care and may disallow trainees from thinking beyond an algorithm. Conversely, protocols provide a summary of best practices and minimize practice variation. Th e above study by Prasad et al. aimed to elucidate the eff ects of high vs. low-intensity protocol-directed care on trainee knowledge about mechanical ventilation.
In this study, 88 accredited U.S. pulmonary and critical care medicine training programs were analyzed. Protocols studied included those for sedation management, mechanical ventilation liberation and lung-protective strategies in ALI. Of these programs, 42 (42%) were considered to be high-intensity (i.e., 2-3 protocols for at least 3 years) and 46 (48%) were low-intensity (i.e., 0-1 protocol for at least 3 years). Program characteristics were similar between the 2 groups, with 89% being in the university setting. Trainee characteristics were also similar with 64% of trainees receiving training in the U.S. or Canada. In a bivariable analysis comparing mechanical ventilation exam scores to protocol intensity, results were nearly identical (High-intensity -mean 497; 95% CI, 486-507. Low-intensity -mean 497; 95% CI, 485-509. Mean diff erence, 0; 95% CI, -16 to 16). In a multivariable analysis adjusting for potential confounders determined a priori (birth country, training country), there was no statistical diff erence between the two groups (adjusted mean diff erence, -5.36; 95% CI, -20.7 to 10.0).
Strengths of the study include the fact that mechanical ventilation is fairly ubiquitous in all ICUs and training programs [5] and quality of training is likely to have a signifi cant eff ect on outcomes. One weakness is that the authors were unable to determine by their survey whether or not protocol availability was equal to protocol utilization. Additionally, there may be a large amount of protocol variation. For example, programs that were considered to have a sedation titration protocol did not necessarily have to include a protocol for daily sedation interruption [6]. Th e participating institutions were also largely in the university setting and it is unknown whether or not these results can be generalized to the community setting. Furthermore, the study assessed performance on a standardized exam rather than overall clinical competency, although the authors showed no diff erence between examinees in overall residency program director rating.
Th e results of this study contribute to the ongoing debate regarding the use of protocols in the ICU setting. Despite the results of this study, there have been ongoing concerns that protocol-directed care may promote disinterest, stifl e learning, and remove the physician from direct patient care [7]. A potential approach to overcome this issue is that educators should periodically question trainees on diff erent aspects of the protocol. For example, during bedside rounds, educators could ensure that trainees understand the protocol, determine reasons for failure in an individual patient, and whether protocol was performed under optimal conditions. Rather than simply saying, "Th is patient has failed his spontaneous breathing trial" and moving on, understanding why the patient failed the weaning trial and whether interventions can be instituted to mitigate some of these factors then becomes important. Th us, while mechanical ventilation protocols have been shown to improve outcomes [1][2][3], they cannot replace clinical judgment. Th ey do not, however, necessarily stifl e clinical judgment.

Recommendation
While protocol-directed care remains a source of debate, this study shows that training in a high-intensity mechanical ventilation protocol environment was not associated with worse performance on certifi cation exam mechanical ventilation questions. Further studies are needed to examine whether protocol-directed care or other interventions targeted for patient safety aff ect competency of trainees.