Study design, participants and setting
A total of 17 intensivists at four community hospital intensive care units (ICUs) in the Greater Toronto Area participated in the present prospective interventional study.
Intervention
After training, each physician was equipped with a handheld computing device (Palm M505; Palm Inc., Milpitas, CA, USA) loaded with medical reference material pertinent to the critical care physician. This information included a customized critical care information handbook ('Critical Care'), which was previously developed for use by residents and physicians at our centre (Additional file 1). Commercially available medical reference software was also incorporated, namely PEPID ED (PEPID LLC, Skokie, IL, USA) and MedCalc http://medcalc.med-ia.net/.
The handheld devices were able to receive literature updates on a regular basis, using customized software (IqSync; Infiniq Software, Mississauga, Ontario, Canada), which accessed an internet-based server using either a connection via desktop computer or infrared data transfer to a telephone modem (Fig. 1). New information was sent to the handheld devices and appeared in a file called 'What's New'. These updates, provided every 2–3 weeks, comprised brief reviews of relevant new literature including a short summary, a commentary and the article abstract.
All handheld devices were equipped with backup software that allowed the content to be rapidly restored in the event of a hardware failure (BackupBuddy VFS; Blue Nomad Software, Redwood City, CA, USA). The devices were also equipped with software capable of generating a log of the applications used (AppUsage; Benc Software Production, Slavonski Brod, Croatia).
Between September and November 2002 the handheld devices were distributed to participating physicians, at which time they each received a 1-hour training session on the use of the handheld device and the internet link (Fig. 2). After training, the participants were able to utilize the devices in clinical practice for 12 months. We provided 24-hour support by telephone and e-mail, with a website for independent review.
Outcome measures
Feasibility
Feasibility of the system was assessed by tracking physicians' use of the handheld device and tracking their access of the individual handheld applications during the study period. Physicians who updated their handheld computers at least once a month for 6 months were identified as 'regular users'. A qualitative assessment of the system was achieved through surveys and focus group methodology. Participants completed surveys at baseline to identify their prior familiarity with handheld devices, and at the end of the study period to evaluate subjectively the handheld reference system and the individual handheld applications. Survey data were scored on a 7-point scale, in which 'poor' scored 1 and 'excellent' scored 7. An independent company (The NRC+Picker Group, Markham, Canada) conducted the focus group evaluations at the end of the intervention period, to determine the perceived utility of the information system. Each hospital physician group participated in one focus group meeting.
Information access
Information sources that physicians accessed to make clinical decisions were evaluated during simulated patient care scenarios, completed in the physicians' own ICU utilizing a computerized patient simulator (SimMan; Laerdal Medical Corporation, Wappingers Falls, NY, USA). Each physician completed one scenario before the handheld device was introduced (baseline scenario) and one at the end of the intervention period (final scenario), when the handheld device could be used (Fig. 2). A small pool of five scenarios with equivalent complexity was developed, such that physicians would likely need to access information sources in order to make management decisions. The scenarios involved unusual but important conditions, namely thyroid storm, myasthenia gravis, methanol toxicity, malaria and methemoglobinaemia. They were allocated to study participants in such a way as to avoid participants from the same site receiving the same scenario at the same time point, and to avoid repetition of scenarios among individual participants. Each scenario concluded with the physician writing admission orders for the simulated patient.
During the scenarios we tracked all medical reference sources utilized by the physicians, who were encouraged to use a 'think aloud' process [15]. An audiovisual recording was made of the scenarios for later analysis, and when the handheld was used real-time screen capture was incorporated into the recording (Additional file 2). This allowed us to document which handheld applications were accessed, the time taken to access information and the time taken to complete the scenario. We developed an objective scoring system for the admission orders generated at each scenario. The admission orders were assigned a score (range 0–100) by a critical care physician (SM) and critical care pharmacist (LB), who were blinded as to whether the physician used the handheld device. The scenario-specific scoring system allocated points for all necessary diagnostic and therapeutic interventions, weighted according to relative importance. Negative points were given for potentially harmful orders.
Additional File 2: Quicktime movie (video clip) demonstrating a clinical simulation scenario, using the patient simulator Sim-Man. The physician can be seen accessing the handheld device, and utilization of the various information resources can be tracked. (MOV 1 MB)
Data analysis
Data are presented as median and interquartile range (IQR), and permutation tests were used for comparisons because numbers were small and not normally distributed. The differences between the final and baseline admission order scores and the time to completion of scenarios were calculated for each participant. A two-sample permutation test was used to compare these differences between the group of physicians who chose to use the handheld in the final scenario and those who did not use the device. Admission order scores obtained for each of the five scenarios were compared. Outcomes were considered statistically significant at α < 0.05. The SAS System for Window version 8.2 (SAS Institute, Inc., Cary, NC, USA) was used for all analyses.
Focus groups were recorded, transcribed verbatim and subsequently analyzed. Themes were identified and unique perspectives on key issues noted [16].