Comparison of different pain scoring systems in critically ill patients in a general ICU
© Ahlers et al.; licensee BioMed Central Ltd. 2008
Received: 19 November 2007
Accepted: 16 February 2008
Published: 16 February 2008
Pain in critically ill patients in the intensive care unit (ICU) is common. However, pain assessment in critically ill patients often is complicated because these patients are unable to communicate effectively. Therefore, we designed a study (a) to determine the inter-rater reliability of the Numerical Rating Scale (NRS) and the Behavioral Pain Scale (BPS), (b) to compare pain scores of different observers and the patient, and (c) to compare NRS, BPS, and the Visual Analog Scale (VAS) for measuring pain in patients in the ICU.
We performed a prospective observational study in 113 non-paralyzed critically ill patients. The attending nurses, two researchers, and the patient (when possible) obtained 371 independent observation series of NRS, BPS, and VAS. Data analyses were performed on the sample size of patients (n = 113).
Inter-rater reliability of the NRS and BPS proved to be adequate (kappa = 0.71 and 0.67, respectively). The level of agreement within one scale point between NRS rated by the patient and NRS scored by attending nurses was 73%. However, high patient scores (NRS ≥4) were underestimated by nurses (patients 33% versus nurses 18%). In responsive patients, a high correlation between NRS and VAS was found (rs = 0.84, P < 0.001). In ventilated patients, a moderate positive correlation was found between the NRS and the BPS (rs = 0.55, P < 0.001). However, whereas 6% of the observations were NRS of greater than or equal to 4, BPS scores were all very low (median 3.0, range 3.0 to 5.0).
The different scales show a high reliability, but observer-based evaluation often underestimates the pain, particularly in the case of high NRS values (≥4) rated by the patient. Therefore, whenever this is possible, ICU patients should rate their pain. In unresponsive patients, primarily the attending nurse involved in daily care should score the patient's pain. In ventilated patients, the BPS should be used only in conjunction with the NRS nurse to measure pain levels in the absence of painful stimuli.
Pain is a frequently experienced problem in critically ill patients in the intensive care unit (ICU) . Pain may increase morbidity and mortality and may decrease the comfort of patients and health-related quality of life. The adequate use of analgesics and sedatives therefore may decrease morbidity and mortality . Measurement of pain in ICU patients, however, may be complicated by decreased consciousness, severity of illness, mechanical ventilation, and the use of sedatives in these patients, particularly when high doses of sedatives are administered [3, 4]. Although self-report is still the 'gold standard' in pain measurement according to the guidelines of the International Association for the Study of Pain , one segment of ICU patients is unable to communicate effectively. In these cases, the gold standard (that is, the pain intensity reported by the patient) is not possible or is potentially unreliable. This is also a common problem in, for example, neonates and children, who are not able to report pain in a reliable manner .
Therefore, pain assessment in the ICU remains a challenge for clinicians and researchers. There is no specific neurobiological parameter for the evaluation of pain, nor does an objective quantification of pain intensity or relief exist . Various pain scales are available, but it remains unclear whether they can be applied reliably in the diverse patient population of the ICU, where patients not only may be mechanically ventilated but also are subject to repeated painful procedures. Therefore, it is of interest to define which score should be used for which patient (for example, ventilated, responsive, or unresponsive) and by which health care worker, in case the patient cannot communicate. These results can be used to implement a systematic evaluation of pain in all ICU patients. While, to date, the use of scoring systems for pain severity and sedation depth and the implementation of protocols increase with a more patient-oriented regime for analgesia and sedation, a trend is observed away from a hypnosis-based approach and toward an analgesia-based approach. Although these changes may improve pain and sedation practice, further efforts are needed for widespread implementation of pain scoring systems and analgesia protocols [8, 9]. Of the available pain scales, the Numerical Rating Scale (NRS) (1 to 10) and the Visual Analog Scale (VAS) (1 to 100) have been validated for acute pain only and not in mechanically ventilated patients in the ICU . The Behavioral Pain Scale (BPS) was developed specifically for measuring the severity of pain in sedated, mechanically ventilated, unresponsive patients , but this pain scale still is not generally accepted for routine use. Another question in pain management in the ICU is which health care worker (nurse, physician, and so on) should rate pain in case the patient cannot communicate verbally. While the attending nurse is involved in close daily care of the patient, the physician seems to have a more distant relation to the patient. Therefore, we designed a study (a) to determine the inter-rater reliability of the NRS and BPS, (2) to compare pain scores of different observers and the patient, and (c) to compare NRS, VAS, and BPS for measuring pain in ventilated and non-ventilated patients in the ICU.
Materials and methods
A prospective observational study was performed in a 30-bed surgical/medical ICU in a teaching hospital in Nieuwegein, The Netherlands. The medical ethical committee of St. Antonius Hospital approved the study protocol and waived the need for informed consent because the observational study design and pain measurement are part of the standard care.
All patients in the ICU who were at least 18 years old were included between 27 June and 4 August 2005. Patients who received neuromuscular blocking medications or muscle-paralyzing drugs continuously, who were unconscious after resuscitation, who were quadriplegic, who suffered from a critical illness (poly)neuropathy, or who had an epidural catheter were excluded. Paralysis, whether caused by a pre-existing condition or by medication, makes the BPS unreliable.
Pain measurement instruments
The Behavioral Pain Scale 
Fully bent with finger flexion
Compliance with ventilation
Coughing but tolerating ventilation for most of the time
Unable to control ventilation
Depth of sedation
The Ramsay Scale (RS) was used to assess the sedation level . The RS is a scale from 1 to 6, with higher levels indicating increased degrees of sedation, and considers the following levels: (1) patient anxious, agitated, and restless; (2) patient cooperative, orientated, and tranquil; (3) patient drowsy or asleep and responds easily to commands; (4) patient asleep and gives a brisk response to a light glabellar tap; (5) patient asleep and gives a sluggish response to a light glabellar tap; and (6) patient asleep and gives no response to a light glabellar tap .
Standard pain medication in the intensive care unit
All patients received pain medication according to the local standard protocol, consisting of 1 gram of acetaminophen rectally three times daily and 10 mg of morphine subcutaneously four times daily or 30 to 50 mg of morphine per day using a continuous intravenous infusion, if required.
Pain and sedation scales performed by researcher, nurse, and patient
Behavioral Pain Scale (if ventilated)
Numerical Rating Scale
Visual Analog Scale
Patient (whenever possible)
Training pain measurement instruments
For adequate use of the BPS, NRS, VAS, and RS, the two researchers attended a 4-hour training session (conducted by a trained ICU nurse), during which the BPS and RS were explained with examples of patients who were in the ICU at the time of the training. When the inter-rater reliability was acceptable according to a quadratic weighted Cohen's kappa of greater than 0.6, the researchers were allowed to score patients for the study .
Data were analyzed with the statistical software S-Plus® version 6.2 (Insightful Corporation, Seattle, WA, USA). To correct for the different numbers of measurements per patient, one observation per patient was randomly selected. All statistical analyses were performed using this independent sample, while all measurements were plotted in the figures for better illustration. Kappa coefficients with quadratic weights were used to reflect agreement for ordinal scales (NRS, BPS, and RS) between the independent researchers. Weighted kappa penalizes disagreement in terms of their seriousness . Theoretically, the value of kappa can range from 0 (disagreement) to 1 (perfect agreement). A value larger than 0.6 was regarded as satisfactory . The 95% confidence intervals for kappa coefficients were calculated. Spearman non-parametric rank correlation coefficients (rs) were used to measure the degree of correlation for two ordinal variables. The null hypothesis that the correlation coefficient is zero was tested. A P value of less than 0.05 was considered statistically significant.
Patient characteristics and data
Baseline patient characteristics
Number of patients
66 ± 15a
Male gender, number
Body mass index, kg/m2
26.6 ± 4.7a
Diagnostic categories, number
Mechanical ventilation, number
Median Ramsay Scale score (range)
Inter-rater reliability of the NRS, BPS, and Ramsay Scale
Agreement within 1 scale point
Volume-controlled ventilated patients
Pressure-support ventilated patients
NRS patient versus NRS nurse or NRS researcher
NRS patient and VAS patient
NRS nurse and BPS researcher
NRS researcher and Ramsay Scale researcher
The correlation between the NRS researcher and the RS researcher was low (rs = 0.28, P = 0.078, n = 40). The correlation was weak in both cardiothoracic patients and non-cardiothoracic patients (rs = 0.24, P = 191, n = 31 patients versus rs = 0.04, P = 0.9, n = 13 patients).
In our study, we found that the inter-rater reliability for the NRS and BPS was good, which proves that it is possible to train medical personnel to use these scales in a reliable way in ICU patients. However, although the different pain scales show a high reliability, an important finding of our study is that especially unacceptably high patient scores (NRS ≥4) were underestimated by both the nurses and the researchers as 33% of the NRS patient values were equal to or greater than 4 compared with 18% for the nurses. As it is known that the patients may underestimate pain by themselves (caused by factors like culture and the environment [22, 23]), the risk of underestimation seems to be an important issue when scoring pain in ICU patients. The underestimation of patients' pain scores by the nurses is already supported in the literature. However, the finding that underestimation occurs in especially high patient scores was never reported. Therefore, it is of utmost importance to use restrictive sedation protocols aiming at cooperative sedation levels instead of unconscious levels , allowing for response to questions about pain evaluation and thereby reducing observed-based pain evaluations and allowing for self-report of pain.
The correlation between the NRS and the BPS in our study is in accordance with the study of Payen and colleagues , which showed that the BPS is reliable for measuring the severity of pain in sedated and ventilated patients. Also, Aissaoui and colleagues  concluded that the BPS is valid and reliable for measuring intervention pain in non-communicative ICU patients. In our study, however, in 57 ventilated patients of which 13 were communicative, only 5% of the observations (n = 151 observations of 57 patients) were NRS of 0 (no pain), whereas on the BPS, a remarkable 68% of the observations were BPS of 3 (no pain). In addition, although 6% of our observations were NRS of greater than or equal to 4 (unacceptable pain according to ), the BPS scores were all low with a median value of 3.0 (range 3.0 to 5.0), which is the lowest possible value of the BPS in a scale with a maximum value of 12. Also, in the study of Payen and colleagues , a high non-response on the BPS was found in assessments completed at rest (BPS score of 3 in 88% to 97% of the observations) and 82% of the observations at rest and during interventions were clustered around BPS scores of 3 to 6.
The high non-response on the BPS can be explained by the short time of observation. During 1 minute of observation, the patient may seem pain-free (BPS of 3). However, using the NRS, a higher score may be rated as the nurse tends to include more background information of the patient (for example, the pain levels from the last hours while caring for the patient). So the BPS reflects the objective visible behavior at one specific time point, whereas the NRS represents a global impression of pain, including several contextual factors during a longer time period. It seems, therefore, that the BPS should be used only in conjunction with the NRS nurse to measure pain levels the ICU.
Various studies concluded that, compared with the NRS, the VAS is not an adequate tool in patients with decreased consciousness [25, 26]. This appears to be related to the lack of ability for abstraction and comprehension and provides difficulties in patients who are injured to the upper limbs. In our study, the correlation between the NRS and VAS estimated by the patient is strong (rs = 0.84, P < 0.001, n = 75), suggesting that the VAS is also an adequate tool for measuring pain in about two thirds of the patients in our ICU. However, the VAS could be used in only 75 of 113 patients, in particular in patients with intact comprehension and abstraction, when recovering from critical illness, and just before leaving the ICU following cardiac surgery. Therefore, it is unknown whether this finding can be extrapolated to other ICUs.
The correlation between the NRS score and RS score was low (rs = 0.28, P = 0.078, n = 40). So the degree of pain intensity does not seem to depend on the level of sedation. This is important because these two scores should be able to distinguish between the level of analgesia and the level of sedation. Whereas high RS levels (deep sedation) may be expected in more severely ill patients experiencing more pain, patients with low RS levels (light sedation), in contrast, have more ability to show painful behaviors, resulting in the absence of a significant correlation.
This study had several limitations. In the present study, we collected pain scores of ICU patients at rest and without painful stimuli. In the ideal study design in which different pain scoring systems in the ICU are compared, the pain scores in the absence and presence of an unavoidable painful stimulus should be tested in order to be able to study the sensitivity to change for each pain scale. In further studies, therefore, basal pain scores should be obtained together with intervention pain scores in order to evaluate and judge pain scales for different purposes (for example, at rest and during painful interventions).
Furthermore, the patients included in this study are characterized by a high percentage (73%) of post-cardiothoracic surgery patients and a 50% rate of mechanical ventilation, which was partly due to pain measurements before leaving the ICU, so extrapolation of the results to other ICUs may be limited. In addition, in our ICU, sedation levels are aimed at cooperative levels comparable to those of Kress and colleagues  (2000) and Brook and colleagues  (1999) whenever possible. Both of these characteristics of our ICU may have resulted in the relatively high percentage of responsive patients (66%) who were able to report pain using NRS and VAS. This should still be considered when the results of our study are extrapolated to other ICUs. On the other hand, in our study, there were no significant differences when the results were divided between 'cardiothoracic surgery' patients and 'non-cardiothoracic surgery' patients.
The different scales show a high reliability, but observer-based evaluation often underestimates the pain, particularly in the case of high NRS values (≥4) rated by the patient. Therefore, whenever this is possible, ICU patients should rate their pain. In unresponsive patients, primarily the attending nurse involved in daily care should score the patient's pain. The BPS should be used only in conjunction with the NRS nurse to measure pain levels in the absence of painful stimuli.
Especially unacceptably high patients' scores (Numerical Rating Scale [NRS] ≥4) are underestimated by nurses.
Whenever possible, intensive care unit patients should rate their pain, which is the 'gold standard' in pain measurement.
The Behavioral Pain Scale should be used only in conjunction with the NRS nurse to measure pain levels in the absence of painful stimuli.
Behavioral Pain Scale
- ICU :
intensive care unit
Numerical Rating Scale
Spearman non-parametric rank correlation coefficient
- RS :
Visual Analog Scale.
The authors thank the staff and nurses of the ICU of St. Antonius Hospital for their contribution to this study, Andrea Warman for her contribution during the practical work, and Monique van Dijk for critically reading the manuscript.
- Li DT, Puntillo K: A pilot study on coexisting symptoms in intensive care patients. Appl Nurs Res 2006, 19: 216-219. 10.1016/j.apnr.2006.01.003View ArticlePubMedGoogle Scholar
- Walder B, Tramer MR: Analgesia and sedation in critically ill patients. Swiss Med Wkly 2004, 134: 333-346.PubMedGoogle Scholar
- Kollef MH, Levy NT, Ahrens TS, Schaiff R, Prentice D, Sherman G: The use of continuous i.v. sedation is associated with prolongation of mechanical ventilation. Chest 1998, 114: 541-548. 10.1378/chest.114.2.541View ArticlePubMedGoogle Scholar
- Kress JP, Pohlman AS, O'Connor MF, Hall JB: Daily interruption of sedative infusions in critically ill patients undergoing mechanical ventilation. N Engl J Med 2000, 342: 1471-1477. 10.1056/NEJM200005183422002View ArticlePubMedGoogle Scholar
- Merskey H, Bogduk N: Classification of Chronic Pain: Descriptions of Chronic Pain Syndromes and Definitions of Pain Terms. 2nd edition. Seattle: IASP Press; 1994.Google Scholar
- van Dijk M, de Boer JB, Koot HM, Tibboel D, Passchier J, Duivenvoorden HJ: The reliability and validity of the COMFORT scale as a postoperative pain instrument in 0 to 3-year-old infants. Pain 2000, 84: 367-377. 10.1016/S0304-3959(99)00239-0View ArticlePubMedGoogle Scholar
- Dimopoulou I: Endocrine and metabolic disturbances in critically ill patients: to intervene or not? Eur J Intern Med 2005, 16: 67-68. 10.1016/j.ejim.2004.10.015View ArticlePubMedGoogle Scholar
- Martin J, Franck M, Sigel S, Weiss M, Spies CD: Changes in sedation management in German intensive care units between 2002 and 2006: a national follow up survey. Crit Care 2007, 11: R124. 10.1186/cc6189PubMed CentralView ArticlePubMedGoogle Scholar
- Regan K, Boyd O: Sedation practice: is it time to wake up and embrace change? Crit Care 2008, 12: 102. 10.1186/cc6203PubMed CentralView ArticlePubMedGoogle Scholar
- Aissaoui Y, Zeggwagh AA, Zekraoui A, Abidi K, Abouqal R: Validation of a behavioral pain scale in critically ill, sedated, and mechanically ventilated patients. Anesth Analg 2005, 101: 1470-1476. 10.1213/01.ANE.0000182331.68722.FFView ArticlePubMedGoogle Scholar
- Payen JF, Bru O, Bosson JL, Lagrasta A, Novel E, Deschaux I, Lavagne P, Jacquot C: Assessing pain in critically ill sedated patients by using a behavioral pain scale. Crit Care Med 2001, 29: 2258-2263. 10.1097/00003246-200112000-00004View ArticlePubMedGoogle Scholar
- Chanques G, Jaber S, Barbotte E, Violet S, Sebbane M, Perrigault PF, Mann C, Lefrant JY, Eledjam JJ: Impact of systematic evaluation of pain and agitation in an intensive care unit. Crit Care Med 2006, 34: 1691-1699. 10.1097/01.CCM.0000218416.62457.56View ArticlePubMedGoogle Scholar
- Jensen MP, Karoly P, Braver S: The measurement of clinical pain intensity: a comparison of six methods. Pain 1986, 27: 117-126. 10.1016/0304-3959(86)90228-9View ArticlePubMedGoogle Scholar
- Kremer E, Atkinson JH, Ignelzi RJ: Measurement of pain: patient preference does not confound pain measurement. Pain 1981, 10: 241-248. 10.1016/0304-3959(81)90199-8View ArticlePubMedGoogle Scholar
- Hamill-Ruth RJ, Marohn ML: Evaluation of pain in the critically ill patient. Crit Care Clin 1999, 15: 35-54. v–vi 10.1016/S0749-0704(05)70038-5View ArticlePubMedGoogle Scholar
- Huskisson EC: Measurement of pain. Lancet 1974, 2: 1127-1131. 10.1016/S0140-6736(74)90884-8View ArticlePubMedGoogle Scholar
- Pilowsky I, Kaufman A: An experimental study of atypical phantom pain. Br J Psychiatry 1965, 111: 1185-1187.View ArticlePubMedGoogle Scholar
- Ramsay MA, Savege TM, Simpson BR, Goodwin R: Controlled sedation with alphaxalone-alphadolone. Br Med J 1974, 2: 656-659.PubMed CentralView ArticlePubMedGoogle Scholar
- Lerch C, Park GR: Sedation and analgesia. Br Med Bull 1999, 55: 76-95. 10.1258/0007142991902303View ArticlePubMedGoogle Scholar
- Fleiss J: The measurement of interrater agreement. In Statistical Methods for Rates and Proportions. 2nd edition. New York: John Wiley and Sons; 1981:212-235.Google Scholar
- Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977, 33: 159-174. 10.2307/2529310View ArticlePubMedGoogle Scholar
- Turk DC, Okifuji A: Assessment of patients' reporting of pain: an integrated perspective. Lancet 1999, 353: 1784-1788. 10.1016/S0140-6736(99)01309-4View ArticlePubMedGoogle Scholar
- Vigano A, Bruera E, Suarez-Almazor ME: Age, pain intensity, and opioid dose in patients with advanced cancer. Cancer 1998, 83: 1244-1250. 10.1002/(SICI)1097-0142(19980915)83:6<1244::AID-CNCR26>3.0.CO;2-4View ArticlePubMedGoogle Scholar
- Collins SL, Moore RA, McQuay HJ: The visual analogue pain intensity scale: what is moderate pain in millimetres? Pain 1997, 72: 95-97. 10.1016/S0304-3959(97)00005-5View ArticlePubMedGoogle Scholar
- Berthier F, Potel G, Leconte P, Touze MD, Baron D: Comparative study of methods of measuring acute pain intensity in an ED. Am J Emerg Med 1998, 16: 132-136. 10.1016/S0735-6757(98)90029-8View ArticlePubMedGoogle Scholar
- Park WY, Thompson JS, Lee KK: Effect of epidural anesthesia and analgesia on perioperative outcome: a randomized, controlled Veterans Affairs cooperative study. Ann Surg 2001, 234: 560-569. discussion 569–571 10.1097/00000658-200110000-00015View ArticlePubMedGoogle Scholar
- Brook AD, Ahrens TS, Schaiff R, Prentice D, Sherman G, Shannon W, Kollef MH: Effect of a nursing-implemented sedation protocol on the duration of mechanical ventilation. Crit Care Med 1999, 27: 2609-2615. 10.1097/00003246-199912000-00001View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.