Validation of the Critical-Care Pain Observation Tool-Neuro in brain-injured adults in the intensive care unit: a prospective cohort study

Background Pain assessment in brain-injured patients in the intensive care unit (ICU) is challenging and existing scales may not be representative of behavioral reactions expressed by this specific group. This study aimed to validate the French-Canadian and English revised versions of the Critical-Care Pain Observation Tool (CPOT-Neuro) for brain-injured ICU patients. Methods A prospective cohort study was conducted in three Canadian and one American sites. Patients with a traumatic or a non-traumatic brain injury were assessed with the CPOT-Neuro by trained raters (i.e., research staff and ICU nurses) before, during, and after nociceptive procedures (i.e., turning and other) and non-nociceptive procedures (i.e., non-invasive blood pressure, soft touch). Patients who were conscious and delirium-free were asked to provide their self-report of pain intensity (0–10). A first data set was completed for all participants (n = 226), and a second data set (n = 87) was obtained when a change in the level of consciousness (LOC) was observed after study enrollment. Three LOC groups were included: (a) unconscious (Glasgow Coma Scale or GCS 4–8); (b) altered LOC (GCS 9–12); and (c) conscious (GCS 13–15). Results Higher CPOT-Neuro scores were found during nociceptive procedures compared to rest and non-nociceptive procedures in both data sets (p < 0.001). CPOT-Neuro scores were not different across LOC groups. Moderate correlations between CPOT-Neuro and self-reported pain intensity scores were found at rest and during nociceptive procedures (Spearman rho > 0.40 and > 0.60, respectively). CPOT-Neuro cut-off scores ≥ 2 and ≥ 3 were found to adequately classify mild to severe self-reported pain ≥ 1 and moderate to severe self-reported pain ≥ 5, respectively. Interrater reliability of raters’ CPOT-Neuro scores was supported with intraclass correlation coefficients > 0.69. Conclusions The CPOT-Neuro was found to be valid in this multi-site sample of brain-injured ICU patients at various LOC. Implementation studies are necessary to evaluate the tool’s performance in clinical practice. Supplementary Information The online version contains supplementary material available at 10.1186/s13054-021-03561-1.

Despite the ability of the BPS and CPOT to discriminate between nociceptive and non-nociceptive procedures, several issues were identified at the item level. A lower effect size for responsiveness between rest and nociceptive procedures was found for the facial expression item compared to the upper limbs item of the BPS [9]. Also, grimace and muscle rigidity using the CPOT [11] were not frequently observed. However, grimace was the best predictor of self-reported pain intensity in brain-injured ICU patients who were conscious and able to communicate in a reliable manner, i.e., not delirious [17]. Behaviors not included in original versions of the scales such as orbit tightening, eye weeping (tearing), and face flushing were described in this patient population [17][18][19]. Low levels of consciousness (LOC) or high sedation levels, often present in brain-injured ICU patients, were associated with low frequency of behaviors indicative of pain [17][18][19] and low behavioral scale scores [8,9,12,20]. Adaptation of the content of existing scales for brain-injured ICU patients could enhance their applicability and ability to accurately detect pain in this vulnerable population. Although other tools are available for brain-injured patients with disorders of consciousness such as the Nociception Coma Scale [21] and its revised version [22], these were not developed for use in the ICU context and are not applicable to mechanically ventilated patients [23].
The purpose of this study was to validate the use of the French-Canadian and English revised versions of the CPOT for brain-injured ICU patients: the CPOT-Neuro. Specific objectives were to examine the: 1. Association between CPOT-Neuro scores and the reference standard measure of pain (patient selfreporting), and the ability of CPOT-Neuro to detect self-reported pain in brain-injured ICU patients (criterion validation); 2. Ability of CPOT-Neuro scores to discriminate between non-nociceptive and nociceptive procedures, and when feasible, before and after opioid administration (discriminative validation); 3. Agreement of CPOT-Neuro scores between trained research staff and ICU nurses (interrater reliability)

Design
A prospective cohort with repeated-measures design was used for this multicenter validation study. This design allowed the testing of the reliability and the validity of the French-Canadian and English versions of the CPOT-Neuro with the collection of data across several procedures and time points.

Settings
This study was conducted in four neurotraumatology ICU settings across Canada (two sites from Montreal, Quebec and one in Toronto, Ontario) and the USA (one site from Seattle, Washington). One of these settings was a French-speaking one (Montreal, Canada), one was bilingual (Montreal, Canada), and two were Englishspeaking settings (Toronto, Canada and USA). These ICU facilities had similar capacity (22-30 beds) admitting 1000-1500 patients annually. Individualized prescriptions were used for pain management in all sites.

Sample
We sought a heterogeneous population of brain-injured ICU patients as we aimed to validate the CPOT-Neuro to be generalizable for a diverse brain-injured population, rather than specific to patients with a single diagnosis. Accordingly, patients meeting the following inclusion criteria were eligible: (1) 18 years and older; (2) admitted for brain injury to the ICU for less than 4 weeks (e.g., traumatic brain injury [TBI] with or without other trauma, ischemic or hemorrhagic stroke including cerebral aneurysm, cerebral tumor, or brain injury from other causes); and (3) had a score ≥ 4 on the Glasgow Coma Scale (GCS: [24]). To reduce potentially confounding factors and allow for the observation of behavioral reactions, patients were excluded if they: (1) sustained a spinal cord injury affecting motor activity of the four limbs; (2) had cognitive deficits or psychiatric conditions (e.g., psychosis, suicidal ideation) prior to the brain injury; (3) were previously diagnosed with epilepsy, (4) were receiving neuromuscular blocking agents; (5) had a score of − 5 (unarousable) on the Richmond Agitation Sedation Scale (RASS) [25]; and (6) had suspected brain death. According to the regulations in the provinces of Quebec and Ontario, informed written consent form was obtained from all participants (patients or their representative, in case of sudden incapability) of the three Canadian ICU settings. Because this was an observational and noninterventional study, informed written consent was not required according to regulations in the state of Washington. Recruitment and data collection occurred from June 2015 through December 2016.

Procedures
Data collection was performed before and during nonnociceptive (NIBP, soft touch) and nociceptive (turning and others outlined in Table 1) procedures as well as before and after opioid administration. NIBP was measured with cuff inflation (equipment at the bedside) and was used as a non-nociceptive procedure as it was found to be painless in ICU patients including those with a brain injury [10,15,26]. NIBP and all nociceptive procedures were part of ICU standard care. Only soft touch was added as a non-nociceptive procedure during which research staff, an ICU nurse or a family member touched the patient's forearm for one minute [27]. Soft touch was also found to be painless in brain-injured ICU patients [11,13]. These procedures allowed evaluation of the ability of the CPOT-Neuro to discriminate between procedures likely to be painful or not (discriminative validation). Each patient had six-to-ten assessment time points (three to five procedures per patient for two assessment time points per procedure) per data set. All patients had at least one data set when initially enrolled in the study, and some of them had two data sets if their LOC changed during their ICU stay after study enrollment. LOC was evaluated according to their Glasgow Coma Scale (GCS) score: unconscious with GCS ≤ 8, altered LOC with GCS from 9 to 12, and conscious with GCS from 13 to 15 [24]. Patients were assessed for pain with the CPOT-Neuro by trained research staff. Whenever possible for turning and other nociceptive procedures, an available ICU nurse trained for the study was asked to independently complete the CPOT-Neuro to determine interrater reliability. Patients were assessed during one-minute periods or for the duration of the nociceptive procedure, and raters provided a score on each item of the CPOT-Neuro. After the completion of the tool, the research staff asked conscious and non-delirious patients to report the presence/ absence of pain verbally (yes/no) or using signs (e.g., head nodding), and to rate their pain intensity using the 0-10 Faces Pain Thermometer (FPT) visual format [28]. The patient's self-report is considered the reference standard measure in the field of pain and was used to establish criterion validation of the CPOT-Neuro.

CPOT-Neuro
The CPOT-Neuro is an adaptation of the original CPOT. The original tool [3] includes five behavioral items: (a) facial expression, (b) body movements, (c) compliance with ventilator (mechanically ventilated patients) or (d) vocalization (non-intubated patients), and (e) muscle tension. Each item is scored from 0 to 2 for a possible total score from 0 to 8. A higher score reflects more intense behavioral reactions, and a cut-off score ≥ 2 indicates the presence of pain [10,11,23,29].
In the adaptation process, each item and score of the original CPOT were reviewed in accordance with observational data of pain-related behaviors in brain-injured ICU patients [17,18,30]. Evaluations of the behaviors' relevance for pain assessment in this population were also made by 61 ICU nurses, 13 physicians, and 3 physiotherapists [31].
Modifications were made to all items of the tool. Scores of 0 remained unchanged for facial expression, body movements, muscle tension, compliance with the ventilator, and vocalization. For the facial expression item, the score of 1 was modified to only include brow lowering. Brow lowering was identified as a frequent reaction to a painful procedure in brain-injured ICU patients no matter their LOC [17,18], and considered relevant by ICU clinicians [31]. A score of 2 was modified to include at least two contractions in the patient's upper face (e.g., brow lowering + eye tightening) or grimace. Grimace is Nasogastric tube/Dobhoff insertion 3 0 Endotracheal tube removal 2 0 Other common ICU procedures a 9 1 Total 329 108 the strongest predictor of pain intensity in this patient group [17] and rated as highly relevant by ICU clinicians [31]. In regard to body movements, scores of 1 and 2 were also operationalized a bit differently. A score of 1 was modified to include non-purposeful movements such as cautious movements or limb flexion. A score of 2 was related to protection or purposeful movements such as trying to reach or touching the pain site which was rated as highly relevant by ICU clinicians [31]. These descriptions of body movements better reflect what was observed in brain-injured ICU patients [17]. In the ventilator compliance item, only the description of score of 1 was modified for activation of alarms. Coughing was removed as it was rated as irrelevant by ICU clinicians [31]. For vocalization, verbal complaints of pain were added in the score of 2 as it was considered relevant by ICU clinicians for brain-injured patients who are conscious or with an altered LOC [31]. The score of 2 (very tense) for muscle tension was removed as it can be confounded with spasticity as a consequence of brain injury as highlighted by ICU clinicians [31]. Autonomic responses related to tearing and face flushing were newly described in brain-injured ICU patients [17,19]. Autonomic responses were also rated as relevant by clinicians for brain-injured ICU patients who are conscious or with an altered LOC [31]. A score of 1 was assigned in the presence of at least one of these autonomic responses.
The total score of CPOT-Neuro may vary from 0 to 8 and remains consistent with the original CPOT total score. The CPOT-Neuro was initially adapted in French-Canadian from the original CPOT French-Canadian version [32] and then translated into English using a forward-backward translation method. Both the French-Canadian and the English versions of the CPOT-Neuro were validated simultaneously in this study. Description of the CPOT-Neuro can be found in Additional file 1.

Raters and CPOT-Neuro Training
The raters included 11 research staff and 36 ICU nurses. Research staff included 2 nursing researchers, 2 clinical research coordinators, 6 nursing research trainees, and one medical student. ICU nurses were mainly female (89%), held a college (33%) or a university degree (66%), and had an experience in the ICU ranging from 2 to 34 years (median = 8.00 [IQR = 5.25-16.75]). In one Canadian ICU, only research staff completed CPOT-Neuro assessments. In other ICUs, both research staff and ICU nurses completed CPOT-Neuro assessments.
The raters were trained by the PI who is also the author of the CPOT/CPOT-Neuro (CG) employing a standardized training session based on the one used for the original CPOT [33]. The training session lasted 45 min during which each item and scoring of the CPOT-Neuro were explained in detail with the support of illustrations and pictures. Briefly, training sessions were delivered in small groups or individually. Three patient videos were also viewed to practice scoring with the tool and to discuss scores within the group or with individual raters. At the end of the training session, raters were asked to view three other patient videos and to provide their written scores. It was expected that raters would use the CPOT-Neuro consistently resulting in a total score difference of no more than one point which was considered acceptable in previous training with the CPOT [34]. When a difference of two points or more was found, the CPOT-Neuro scoring methods were clarified before moving to the next patient video. More than 73% of raters at each site had appropriate scores for all three patient videos, and fewer than 25% had difference in scores ≥ 2 for a maximum of two patient videos [35]. An overall Intraclass Correlation Coefficient (ICC) of 0.85 (95% CI [0.59-1.00]) with all raters' scores for the three videos was reached. Further details on the training are described in another paper [35].

Participant's self-report
At each time point after the completion of the CPOT-Neuro, conscious patients who were able to self-report their pain and not delirious (see Screening of delirium) were asked the question "Do you have pain?", and answer "yes or no" verbally or with other signs (e.g., head nodding). Then, they were asked to rate their pain level with the 0-10 Faces Pain Thermometer (FPT) visual format. The FPT has been developed and tested in critically ill adults by the PI [28], and used in studies with braininjured ICU patients [10,11,17,18,28]. It consists of a thermometer graded from 0 (no pain) to 10 (worst possible pain), including six faces adapted from Prkachin's work [36]. The scale demonstrated good convergent (r = 0.80-0.86 with a descriptive rating scale: [28]) and discriminative validation when comparing scores at rest, during nociceptive and non-nociceptive procedures [10,11,26,28,30,37].

Socio-demographic and clinical variables
The following were collected for each participant: demographic information (sex and age) and clinical data from the patient's medical chart such as diagnosis, severity of illness (APACHE II score: [38]), neuroanatomical location of lesion (diffuse, frontal, parietotemporal, occipital, subcortical), level of consciousness (GCS score or adapted version) [24,39], level of sedation (RASS score: [25]), and administration of analgesic/sedative agents (i.e., IV infusions, and boluses administered within an hour prior to procedures).

Screening of delirium
Prior to data collection, delirium was screened in all conscious patients who were able to communicate verbally or use signs (e.g., head nodding). Delirium can compromise the reliability of self-reports [23]. Trained research staff used either the CAM-ICU [40] or the ICDSC (Intensive Care Delirium Screening Checklist: [41]) whichever was standard practice in the ICU setting to screen patients for delirium. The CAM-ICU and the ICDSC were found to be the most reliable and valid delirium assessment tools for use in critically ill adults based on a recent systematic review [42].

Data analysis
The sample size for this study was calculated based on the requirements for the evaluation of criterion validation using the reference standard measure of pain (patient self-reporting), and to include brain-injured ICU patients at various LOC which is representative of this population. Using MedCalc and G*Power for sample size estimation, a minimal sample size of 52 conscious patients able to self-report their pain was needed to obtain a moderate correlation of 0.50 as found in previous CPOT validation studies [3,10,11,26,43,44] with 90% power and a significance level of 0.01 based on Bonferroni correction for multiple tests. A sample of 50 patients was required for a Receiver Operating Characteristic (ROC) curve analysis with an AUC of 0.80 and a ratio of negative/positive cases of 0.6 during procedural pain [11,29], with 90% power and a significance level of 0.01. Wilcoxon Signed Ranks test with a minimum sample of 50 braininjured ICU patients in each group of the three LOC with an effect size of 0.50 and a significance level of 0.01 would allow a power of 85%.
SPSS software (version 24.0) was used for data analysis. Descriptive statistics were calculated for sociodemographic and clinical data. Nonparametric tests were used given that variables were not normally distributed as supported by Kolmogorov-Smirnov and Shapiro-Wilk tests with p > 0.05 at all assessment time points, and skewness and kurtosis indices > ± 2 [45] for most time points except during turning and other procedures. Kruskal-Wallis and Mann-Whitney U tests were performed to compare CPOT-Neuro scores across analgesia/sedation regimen, medical diagnosis, and sedation level groups. Criterion validation was estimated using the Spearman's rho correlation coefficient between the patients' self-report of pain intensity and CPOT-Neuro scores. The Receiver Operating Characteristic (ROC) curve analysis was used to evaluate the ability of the CPOT-Neuro to classify patients who self-reported the presence versus absence of pain and different pain intensity levels (i.e., ≥ 1 for all levels of pain and ≥ 5 for moderate to severe levels of pain), and to determine the best cut-off score at these pain intensity levels. Discriminative validation of CPOT-Neuro scores across time points and procedures was examined using the Wilcoxon Signed Ranks test. Finally, interrater reliability between the research staff and ICU nurses was examined using the Intraclass Correlation Coefficient (ICC) (two-way mixed model). Missing data were not replaced. Data analyses were performed by a PhD-prepared health professional who was not involved in data collection.

Sample description
A total of 226 patients participated and had a first data set collected. Of these, 87 had a second data set collected when they experienced a change in their LOC during their ICU stay (see Flow Diagram in Fig. 1). In both data sets, patients had a median age > 50 years, the majority were male (> 65%), White Caucasian (> 80%), and admitted to the ICU following a TBI (> 55%) with a brain injury mostly located in the frontal lobe (Table 2). RASS sedation levels indicated that patients were generally drowsy (− 1) or sedated. Almost half were mechanically ventilated.
Regarding analgesia and sedation for the first data set, more than half of patients did not receive analgesia/ sedation before the nociceptive procedures (turning: 54%, other procedures: 53%), some received continuous analgesia and/or sedation (turning: 41%, other procedures: 42%), and only 5% and 4% of patients had a bolus of analgesia/sedation within one hour before turning and other procedures, respectively. Total CPOT-Neuro scores during turning and other nociceptive procedures were not significantly different across analgesia/sedation regimen (Kruskal Wallis test = 0.35 and 1.91, p = 0.950 and 0.591) and medical diagnosis (TBI versus neuro-medical; Mann-Whitney U test = 4859.00 and 1501.50, p = 0.268 and 0.914). Similar results were obtained for the second data set, where 54% and 44% of patients did not receive analgesia/sedation, 37% and 40% received continuous analgesia and/or sedation, and only 9% and 15% had a bolus of analgesia within one hour before turning and other procedures, respectively. In addition, total CPOT-Neuro scores during turning and other nociceptive procedures were not different across analgesia/sedation regimen (Kruskal Wallis test = 0.87 and 6.25, p = 0.832 and 0.100) and medical diagnosis (Mann-Whitney U test = 809.50 and 36.50, p = 0.943 and 0.502).
A total of 95 conscious patients could self-report pain during turning either during the first or the second data set, 18 screened positive for delirium as per the CAM-ICU or ICDSC and were excluded from criterion validation analyses. Similarly, 39 could self-report during other nociceptive procedures and nine were excluded for being positive on delirium screening. Therefore, criterion validation analyses included the self-report of 77 patients during turning, and 30 during other nociceptive procedures. All self-reports were independent data.

Description of CPOT-Neuro scores
During the first data set, total CPOT-Neuro scores were low (median = 0) when patients were at rest, post-administration of an opioid, and during non-nociceptive procedures such as NIBP and soft touch (Table 3). At these time points, patients expressed mainly neutral behaviors such as relaxed facial expression, absence of autonomic responses and body movements, relaxed muscles and compliance with the ventilator or normal vocalization. Higher total CPOT-Neuro scores were observed prior to opioid administration, during turning and other nociceptive procedures. Similar results were obtained for the sub-group of patients with a second data set (Table 4).

Criterion validation
From those who were conscious, 52% could communicate but 19% of them screened positive for delirium and could not provide a reliable self-report of pain. CPOT-Neuro scores were compared with the pain intensity (Table 5) reported by these conscious patients who were negative on delirium screening. At rest pre-turning, 27 patients reported the presence of pain and 40 negated pain. Of these 67 patients, 56 could provide a score on the 0-10 FPT with a median pain intensity < 1. During turning, 51 patients self-reported the presence of pain compared to 26 who denied pain. Of these 77 patients, 62 could provide a numeric score with a median pain intensity > 3. At rest before other nociceptive procedures, 24 patients reported the presence of pain and 6 negated pain. From these 30 patients, 24 could provide a numeric score with a median pain intensity of 1. During other nociceptive procedures, 24 patients reported the presence of pain versus 6 who negated pain. Twenty-five patients could provide a numeric score with a median pain intensity of 4. Before the administration of an opioid dose, 8 patients reported the presence of pain while 4 denied pain (preemptive analgesia). Post-opioid administration, 5 patients reported pain and 4 did not. The median pain intensity decreased by one point post-opioid administration. Similar proportions of patients reported the presence of pain before (38%) and during NIBP (50%) as well as before (48%) and during soft touch (44%). Medians of pain intensity were 0 for most time points of non-nociceptive procedures, and was 1 during NIBP.

Ability of the CPOT-Neuro to classify patients with or without pain and at different levels of pain intensity during turning and other nociceptive procedures
The ROC curve analysis indicated an Area Under the Curve (AUC) of 0.76 (p < 0.001; 95% CI 0.65-0.87) for the presence versus absence of pain (yes/no) during turning (n = 77). Sensitivity was 77% and specificity was 69% corresponding to a CPOT-Neuro cut-off score ≥ 2. The same cut-off score was obtained for the presence of pain during other nociceptive procedures (n = 30) with an AUC of 0.84 (p = 0.011; 95% CI 0.63-1.00), and both sensitivity and specificity were 83%. ROC curve analyses were performed with pain intensity scores ≥ 1 and ≥ 5 used as reference criteria during turning and other nociceptive procedures and are displayed in Fig. 2. The ability of the CPOT-neuro to classify patients at these two pain intensity criteria varied from 76 to 95% for both turning and other procedures. The best CPOT-Neuro cut-off score that maximized sensitivity and specificity was ≥ 2 with all levels of pain intensity criterion ≥ 1, and was ≥ 3 with moderate to severe pain intensity criterion ≥ 5, respectively.

Discriminative validation
For the first data set, total CPOT-Neuro scores were significantly higher during turning compared to rest (Wilcoxon Signed Ranks test = − 11.17, p < 0.001), NIBP (Wilcoxon Signed Ranks test = − 9.93, p < 0.001) and soft touch (Wilcoxon Signed Ranks test = − 10.92, p < 0.001). After opioid administration, total CPOT-Neuro scores were significantly lower compared to before administration (Wilcoxon Signed Ranks test = − 2.97, p = 0.003). Total scores were similar during turning and other nociceptive procedures (Wilcoxon Signed Ranks test = − 0.42, p = 0.674). Total CPOT-Neuro scores were not significantly different across LOC, with medians of 3 in all 3 groups during turning and other nociceptive procedures (Kruskal Wallis test = 0.03 and 0.27, p = 0.986 and 0.265). Similarly, total CPOT-Neuro scores were not statistically different across sedation levels. Medians were 3 during turning and other nociceptive procedures for most groups except for a median of 2 in the awake group during turning (Kruskal Wallis test = 5.06 and 0.95, p = 0.080 and 0.624).
For the second data set, similar results were found, except for opioid administration, which could be assessed for only 11 patients (Wilcoxon Signed Ranks test = − 1.72, p = 0.086). Again, total CPOT-Neuro scores were not different across LOC, with medians of 3 for all 3 groups during turning (n = 81) (Kruskal Wallis test = 0.99, p = 0.611), and medians of 2.5 and 3 during other nociceptive procedures (n = 27) (Mann-Whitney U test = 77.50, p = 0.697) in the altered LOC (n = 10) and conscious group (n = 17), respectively. In addition, total CPOT-Neuro scores during turning were not statistically different across sedation levels, with medians of 2 in the awake group (n = 18), 3 in the sedated group (n = 50), and 4 in the agitated group (n = 13) (Kruskal Wallis test = 3.53, p = 0.171). Similar results were found during other nociceptive procedures, with medians of 2 in the sedated group (n = 15), 3 in the awake group (n = 6), and 3.5 in the agitated group (n = 6) (Kruskal Wallis test = 0.37, p = 0.833).

Discussion
This is the first validation study of the French-Canadian and English versions of the CPOT-Neuro. The findings supported the validity and the interrater reliability of the tool. This validation was conducted among a heterogeneous sample of brain-injured ICU patients with various LOC and RASS scores ranging from − 4 to + 3. More than half of our study sample were conscious patients and 44% presented an altered LOC or unconsciousness.
Moderate positive correlations [46] were found between CPOT-Neuro and self-reported pain intensity scores at rest and during nociceptive procedures supporting criterion validation of the tool in conscious and non-delirious patients. Similar findings were found with the original version of the CPOT in medical, surgical, and trauma ICU patients [23]. Very few studies included brain-injured ICU patients able to self-report. In an Italian study of 300 observations from 50 brain-injured ICU patients, a lower correlation (Spearman rho = 0.38) was found between the original CPOT scores and self-reports of pain intensity at rest and during mobilization [14]. Similarly to the Italian study [14], a Spearman rho correlation of 0.44 between CPOT scores and self-reported pain intensity was found during 190 observations at rest and during turning of 66 trauma and neurosurgical ICU patients (50% with a brain injury) in a previous Canadian study [15]. In both of these studies, non-dependent data (i.e., multiple self-reports from the same patients) were used to calculate the coefficients which may inflate the correlation values.
The CPOT-Neuro could adequately classify patients who reported the presence of pain and various levels of pain intensity during nociceptive procedures. Overall, the performance of the CPOT-Neuro was better than reported for the original version of the CPOT in a previous study with brain-injured ICU patients (AUC = 0.72) [11]. The CPOT-Neuro cut-off score was higher when the reference criterion was set to moderate to severe pain intensity (≥ 5). In previous studies with the original version of the CPOT, cut-off scores varied from 2 to 3 to establish the presence of pain in the ICU population [23]. A cut-off score matching moderate to severe pain may be more useful in the ICU pain management decision making process to determine when opioids are most required. Replication studies with the CPOT-Neuro are necessary to confirm the cut-off score to be used. The CPOT-Neuro was able to discriminate between nociceptive and non-nociceptive procedures as well as prior to and post-administration of an opioid dose which is clinically relevant. Indeed, we expect from a pain tool to discriminate between painful and non-painful conditions and to detect decreases in pain scores following opioid administration. Regarding the CPOT-Neuro items, it is worth mentioning that the frequency of a score of 2 for facial expression was higher in the CPOT-Neuro (36-49%) compared to original version of the CPOT (12-22%) during nociceptive procedures [10,11]. Chookalayia and colleagues [47] highlighted issues with the items of body movements and muscle tension in the original version of the CPOT showing that they could not discriminate between the presence and the absence of pain in agitated ICU patients (n = 15). The modifications made in the development process of the CPOT-Neuro allowed a better representation of behavioral reactions exhibited by brain-injured ICU patients [17].
The CPOT-Neuro scores were similar across LOC groups and sedation levels during nociceptive procedures. This is a strength of this revised tool compared to the use of the original version of the CPOT in braininjured ICU patients. Indeed, previous studies reported low CPOT scores (< 2) during nociceptive procedures in unconscious or deeply sedated brain-injured ICU patients [16,20,26] or with severe brain injury [12]. It is worth mentioning that nociception and pain are distinct but related concepts. Nociception is the neural process of encoding noxious stimuli, and pain is an unpleasant sensory and emotional personal experience associated with, or resembling that associated with, tissue damage [48]. Further, consciousness or memory/recall of events is necessary for the perception of pain. Previous studies report some individuals in a coma following a traumatic brain injury or a cardiac arrest could recall feeling pain when they were unconscious [49,50]. We used nociceptive procedures known to be painful as reported by conscious ICU patients to validate the CPOT-Neuro in those with an altered LOC and unconsciousness [23]. Therefore, the behaviors included in the CPOT-Neuro can detect nociception which is likely to lead to pain. The International Association for the Study on Pain has acknowledged that the inability to communicate does not negate the possibility that an individual experiences pain [48]. In alignment with this statement and the evidence in the field, it appears reasonable to identify the CPOT-Neuro as a pain assessment tool.
Interrater reliability results were acceptable and met standards for reliability coefficients [51]. ICC was higher (> 0.80) between raters at the time of the training when exposure to the CPOT-Neuro was fresh. Only trained ICU nurses used the CPOT-Neuro for this validation study and for enrolled patients. Some of them could not use it shortly after their training and on a regular basis during the study period. Brief booster sessions of 15 min should be planned when enrollment becomes slow to review the use of the tool in order to maintain rating skills [52]. Indeed, many nurses who used the CPOT-Neuro in Canadian sites expressed their desire to get more training and exposure to the tool in their daily practice as they evaluated the tool to be easy and quick to use as well as clinically relevant [35].
In this study, we collected both CPOT-Neuro and self-reported pain scores for research purposes. The patient's self-report of pain remains the reference standard measure of pain in those able to communicate in a reliable manner, and should be obtained whenever possible in clinical practice [53,54]. Therefore, alternative pain assessment methods such as the CPOT and CPOT-Neuro should only be used when the patient's self-report is unobtainable. Systematic pain assessments should be done on a regular basis using the most appropriate pain assessment method based on the patient's ability to communicate [53,54].

Limitations
Several limitations must be addressed. Although we aimed for a heterogeneous sample of brain-injured ICU patients, those unconscious or agitated were less represented (< 20%) and further validation in these groups is required. It was not possible to blind raters to procedures leading to possible bias when scoring with the CPOT-Neuro during nociceptive procedures. However, the evaluation of interrater reliability allowed us to minimize this bias, and the data analyst was not involved in data collection. It was challenging to collect pain self-reports as more than half of conscious and awake patients had delirium or possible cognitive deficits related to their brain injury. Delirium can affect the capacity of an individual to communicate in a reliable manner, and self-reports of delirious patients were weakly (< 0.25) and non-significantly correlated to behavioral pain scores (with CPOT and BPS) in a previous study [55]. Therefore, the screening of delirium is important for the evaluation of criterion validation [23]. In this study, either the CAM-ICU or ICDSC was used for delirium screening; however, it was not measurable in a large proportion of patients due to deep sedation and/or coma [42]. Finally, because the CPOT-Neuro is based on the observation of behavioral reactions, it cannot be used in unresponsive patients (i.e., GCS of 3 or RASS of − 5).

Conclusions
The French-Canadian and English versions of the CPOT-Neuro were found to be valid and reliable in this multisite sample of brain-injured ICU patients at various LOC and levels of sedation. Indeed, the performance of the CPOT-Neuro appeared superior to the CPOT based on previous validation studies in brain-injured ICU patients. The feasibility of its use for validation purposes was described in a separate paper [35], but implementation studies are required to describe its feasibility in ICU daily practice.
involved in the study implementation, and data collection. MBe was involved in the tool development, data interpretation, and participated in writing this manuscript. MBo analyzed and interpreted the data and participated in writing this manuscript. CD was involved in the study implementation, data collection, and data interpretation. CG designed the study, oversaw the recruitment and data collection, analyzed and interpreted the data, and was the main writer of this manuscript. CS was involved in data collection, data analysis and interpretation. DS was involved in the design of the study, and data interpretation. FB was involved in the study implementation as local investigator of one site, oversaw data collection, and participated in data interpretation. KAP was involved in the design of the study, and data interpretation. LR was involved in the study implementation as local investigator of one site, oversaw recruitment and data collection, and participated in data interpretation. MC was involved in the design of the study, and data interpretation. MRL was involved in study implementation and data collection, data analysis, and interpretation. RM was involved in the study implementation, oversaw data collection, and participated in data interpretation. VW was involved in the study implementation, data collection, and data interpretation. All authors read and approved the final manuscript.

Funding
This study was supported by funding from the Canadian Institutes of Health Research (CIHR Funding #119486) and Fonds de Recherche du Québec en Santé (FRQS Funding #25094). Funding from the Réseau de recherche en interventions en sciences infirmières du Québec (RRISIQ) was also obtained to help cover open access fees. The principal investigator (CG) and co-author (MB) hold a research career award from FRQS. The funding agencies were not involved in the design of the study, collection, analysis, and interpretation of data and in writing the manuscript.

Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due to ethics guidelines but are available from the corresponding author on reasonable request.

Declarations Ethics approval and consent to participate
This study was conducted following approval from the ethics committee of each study site. Informed written consent was obtained from all participants or their representative in Canadian sites (Research Ethics Board MUHC-15-994; Research Ethics Board Sunnybrook 173-2015; Research Ethics Committee CIUSSS NIM-2015-1164). A waiver of consent was obtained in the American site (Human Subjects Division application #47430).

Consent for publication
Not applicable.