Research | Open | Published:
Case mix, outcome and length of stay for admissions to adult, general critical care units in England, Wales and Northern Ireland: the Intensive Care National Audit & Research Centre Case Mix Programme Database
Critical Carevolume 9, Article number: S1 (2004)
The present paper describes the methods of data collection and validation employed in the Intensive Care National Audit & Research Centre Case Mix Programme (CMP), a national comparative audit of outcome for adult, critical care admissions. The paper also describes the case mix, outcome and activity of the admissions in the Case Mix Programme Database (CMPD).
The CMP collects data on consecutive admissions to adult, general critical care units in England, Wales and Northern Ireland. Explicit steps are taken to ensure the accuracy of the data, including use of a dataset specification, of initial and refresher training courses, and of local and central validation of submitted data for incomplete, illogical and inconsistent values. Criteria for evaluating clinical databases developed by the Directory of Clinical Databases were applied to the CMPD. The case mix, outcome and activity for all admissions were briefly summarised.
The mean quality level achieved by the CMPD for the 10 Directory of Clinical Databases criteria was 3.4 (on a scale of 1 = worst to 4 = best). The CMPD contained validated data on 129,647 admissions to 128 units. The median age was 63 years, and 59% were male. The mean Acute Physiology and Chronic Health Evaluation II score was 16.5. Mortality was 20.3% in the CMP unit and was 30.8% at ultimate discharge from hospital. Nonsurvivors stayed longer in intensive care than did survivors (median 2.0 days versus 1.7 days in the CMP unit) but had a shorter total hospital length of stay (9 days versus 16 days). Results for the CMPD were comparable with results from other published reports of UK critical care admissions.
The CMP uses rigorous methods to ensure data are complete, valid and reliable. The CMP scores well against published criteria for high-quality clinical databases.
This artcle was previously published in this journal as: Critical Care 8:R99-R111, see: http://ccforum.com/content/8/2/R99
High-quality clinical databases are of value in comparative audit, clinical practice, in managing services and in evaluating health technologies [1, 2]. The use of inappropriate, unrepresentative or poor-quality data can, however, lead to inaccurate conclusions. The Directory of Clinical Databases (DoCDat) was established to inform researchers and clinicians of what clinical databases exist and to provide an independent assessment of their scope and quality . This information is provided through a website . An expert group was convened to develop a quality assessment instrument for clinical databases. The resulting instrument (Fig. 1) consists of 10 items, four relating to coverage and six relating to reliability and validity of the data. Each item is rated on a scale of 1 to 4, with Level 1 representing the least rigorous methods and Level 4 representing the most rigorous. The instrument was shown to have good face and content validity, to have no floor/ceiling effects and to be acceptable to database custodians .
The Intensive Care National Audit & Research Centre (ICNARC) is an independent charity (Registered Charity Number 1039417) established in 1994. ICNARC coordinates a national, comparative audit of patient outcomes from adult, general critical care units in England, Wales and Northern Ireland: the Case Mix Programme (CMP) . After extensive local and central validation, data from the CMP are pooled into the Case Mix Programme Database (CMPD). Data collection has been underway for nearly 8 years and yet baseline statistics from the CMPD have never been formally published. These statistics provide a valuable resource to clinicians working in UK critical care units and to those wishing to make international comparisons of critical care.
The objectives for this paper were to describe how the CMPD performs against the DoCDat criteria for a high-quality clinical database, and to describe the case mix, outcome and activity for patients admitted to adult, general critical care units.
Materials and methods
Participation in the CMP
The CMP recruits from adult, general critical care units. Adult, general critical care units are defined as intensive care units (ICUs), combined ICU/high dependency units (HDUs) and combined general care/coronary care units admitting mixed medical/surgical patients predominantly aged older than 16 years. The Audit Commission survey of 1998  found a total of 328 ICUs or combined ICU/HDUs in England and Wales (excluding neonatal and paediatric units) containing 2076 beds. Of these, 229 units with 1456 beds (70%) would be eligible to participate in the CMP, with the remainder representing specialist (e.g. neurological or cardiothoracic) ICUs or units admitting either only medical or only surgical patients. In addition, the survey found 238 stand-alone HDUs (1236 beds) offering an intermediate location between the ward and the ICU. Participation in the CMP is entirely voluntary, although both the Department of Health and the National Health Service Executive have recommended that all units should take part [7, 8].
Data collection, validation and reporting
CMP data are recorded prospectively and abstracted onto standard forms by trained data collectors according to precise rules and definitions. Abstraction is usually performed retrospectively by chart review. It is thought to take around 10–20 min to abstract the data for one admission, depending on how much intervention the patient has received. A comprehensive dataset specification (the ICNARC Case Mix Programme Dataset Specification)  and individual data collection manuals are made available to all data collectors and software developers. Data collectors from each unit are trained prior to commencing data collection at a 2-day training course. One consultant, one nurse and one audit clerk from each new unit are initially trained to ensure a wide knowledge of the data to be collected in the unit. Retraining of existing staff or training of new staff is also available. Training courses are held at least four times per year.
Precise figures on the background of data collectors are not available. However, each unit must register one data collector as a point of contact for ICNARC. Analysis of the job titles of the 187 staff members for which these data are available shows the following split: 117 (62.6%) audit staff (e.g. audit clerk, information officer, data coordinator), 33 (17.6%) nursing staff (e.g. staff nurse, audit nurse), 23 (12.3%) clerical staff (e.g. secretary, administrative coordinator), six (3.2%) joint audit and clerical staff (e.g. audit and administration manager), three (1.6%) consultant anaesthetists and five other staff (audit clerk/nursing auxiliary, clinical effectiveness coordinator, clinical effectiveness facilitator, ICU technician and research assistant).
Data are collected on consecutive admissions to each participating critical care unit and are submitted to ICNARC in cycles of 6 months. Data are validated locally according to the ICNARC Case Mix Programme Dataset Specification and undergo extensive central validation for completeness, illogicalities and inconsistencies, with data validation reports returned to the units for correction or confirmation. The validation process is repeated until all queries have been dealt with, and the data are then incorporated into the CMPD.
Units receive comparative data analysis reports on each cycle (6 months) of data, from which they can identify their own unit's data compared with all other participating units. Clinicians and managers can also interrogate the CMPD directly by submitting requests for analyses to ICNARC. Reports from these ad hoc analyses are published online .
The ICNARC Coding Method
Information on the reason(s) for admission to the critical care unit is recorded in the CMPD using a standard coding method, the ICNARC Coding Method (ICM) . The ICM is a five-tiered, hierarchically structured method for coding conditions in critical care, developed specifically for the CMP. The five tiers that form the ICM code are: the type of condition (a condition that required surgery or not), the body system, the anatomical site, the pathological/physiological process and the condition necessitating admission. The coding for bacterial pneumonia is shown as an example in Fig. 2.
It is frequently of interest to study patient characteristics and the outcomes of admissions to intensive care with specific conditions. There are two ways in which admissions with specific conditions can be identified in the CMPD. The ICM codes may be used to identify admissions by the primary or secondary reason for admission (coded according to the ICM on information available at admission and during the first 24 hours in the unit), or by the ultimate primary reason for admission (coded according to the ICM on information available after the first 24 hours, at discharge from the unit or following autopsy). Admissions can be identified at any tier of the code; for example, all conditions affecting the gastrointestinal system or all conditions categorised as tumour or malignancy. The second method involves admissions being grouped by physiological definitions; for example, the international definitions for severe sepsis where patients have to meet the SIRS criteria based on their values for temperature, heart rate, respiratory rate, PaCO2 and white blood cell count .
Data collected for the CMP take the form of patient identifiers, demographics, case mix, outcome and activity for admissions to each critical care unit, as defined in the following. A schematic diagram of the timing of data collection for the CMP is presented in Fig. 3. All admissions are followed-up for the entire length of their hospital stay, both within the hospital housing the CMP unit and to their ultimate discharge from hospital. Raw data are collected for all variables rather than categorised, derived or aggregated data or scores.
Individual admissions are identified by an admission number and an alphanumeric unit code; individual identifiers such as name and address (with the exception of the postcode) are not recorded. Records are therefore reversibly anonymised and can only be de-anonymised by the unit that submitted them. A legal agreement is made between ICNARC and the participating units ensuring that the identity of the source of all data (of the hospital, of the unit, of the staff and of the patient) shall remain confidential.
Data are collected on date of birth, gender and postcode. The postcode allows linkage to other databases (e.g. census data for deprivation scoring).
Sufficient raw physiological data are collected to enable calculation of the Acute Physiology and Chronic Health Evaluation (APACHE) II and APACHE III scores and hospital mortality probabilities [13, 14], the Simplified Acute Physiology Score (SAPS) II and associated mortality probability , and the Mortality Probability Model (MPM) II probabilities . Both the lowest and highest recorded values during the first 24 hours in the CMP unit are collected. Raw physiology data are submitted to ICNARC and all scores and probabilities are calculated centrally using standard algorithms to avoid any bias that may be introduced by allowing different units to use slightly different methods of calculating scores and probabilities.
Data are collected on the source of admission to the CMP unit and the location immediately prior to the source of admission. For admissions for whom either of these locations is theatre and recovery in the hospital housing the CMP unit, data are collected on the type of surgery using the classification of the National Confidential Enquiry into Perioperative Deaths. Emergency surgery is defined as immediate surgery, where resuscitation is simultaneous with surgical treatment; urgent surgery is defined as surgery as soon as possible after resuscitation; scheduled surgery is defined as early surgery but not immediately life-saving; and elective surgery is defined as surgery at a time to suit both patient and surgeon.
Survival data (alive/dead) are recorded at discharge from the CMP unit and from the hospital housing the CMP unit. For discharges directly transferred to another critical care unit (in either the same or another hospital) or transferred to another hospital, survival data (alive/dead) at ultimate discharge from a critical care unit and from hospital are also recorded.
The length of stay in the CMP unit is calculated (in fractions of days) from the dates and times of admission to and discharge from the CMP unit. The length of stay in hospital is calculated (in whole days) from the dates of admission and of discharge. For admissions directly transferred from/to another Critical care unit (in either the same hospital or another Hospital) or from/to another hospital, the total length of stay in a critical care unit/hospital is also calculated in whole days.
Readmissions to the CMP unit within the same hospital stay are identified from the postcode, date of birth and gender, and are confirmed by the participating units.
Performance of the CMPD against the DoCDat criteria
The CMPD was rated on a scale of 1 to 4 for each of the 10 DoCDat criteria for coverage and accuracy of a clinical database (Fig. 1). The rating process was performed by DoCDat, independent of the authors.
The case mix, outcome and activity were described for all admissions recorded in the CMPD. The case mix was described by the age at admission, by gender, by APACHE II Acute Physiology Score and hospital mortality probability, by surgical status and by reason for admission to the CMP unit.
The APACHE II Acute Physiology Score is constructed from weights assigned to the most deranged values of 12 physiological variables recorded during the first 24 hours following admission to a critical care unit . The APACHE II score additionally encompasses weights for age and for specific conditions in the past medical history. A hospital mortality probability is constructed from the APACHE II score together with a diagnostic category based on the reason for admission to the critical care unit, and from the surgical status (elective patients versus emergency and nonsurgical patients). Surgical admissions are defined as those whose source of admission was theatre and recovery, or whose location immediately prior to the source of admission was theatre and recovery if their source of admission was recovery only, the X-ray department, the endoscopy suite, a computed tomography scanner or similar, or Accident & Emergency. All other admissions are considered nonsurgical. Surgical admissions were further classified by the National Confidential Enquiry into Perioperative Deaths categories, with elective and scheduled surgery combined into a single category, and urgent and emergency surgery also combined.
Coefficients and diagnostic categories were taken from the UK APACHE II model , which is better calibrated to UK critical care admissions. The diagnostic categories are defined by a body system and a precipitating factor as in the original (US) APACHE II model . However, more combinations of the body system and the precipitating factor are given a coefficient in the UK model as the original model was limited by small samples in some categories. Reasons for admission collected using the ICM are mapped to APACHE II diagnostic categories for the purpose of calculating the APACHE II hospital mortality probability.
The outcome was described by mortality at critical care unit discharge and at hospital discharge, both from the CMP unit and ultimately. Activity was described by CMP unit and total critical care and hospital lengths of stay.
Admissions aged younger than 16 years, staying less than 8 hours in the CMP unit or admitted for primary burns or following coronary artery bypass graft were excluded from the calculation of APACHE II scores. Also excluded were readmissions within the same hospital stay, direct transfers in from other critical care units and admissions missing all 12 physiology variables. These patients were not excluded from any other analyses.
All analyses were performed using the statistical package Stata 8.0 (Stata Corporation, College Station, TX, USA).
Performance of the CMPD against the DoCDat criteria
A summary of the performance of the CMPD against the DoCDat criteria is shown in Fig. 4 with the median and interquartile ranges from all 154 databases in DoCDat for comparison. The mean level achieved by the CMPD across all criteria was 3.4. The CMPD exceeded the DoCDat median for five categories and equalled it in the other five categories. The CMPD never performed worse than the median. Detailed scoring of each criterion is described in the following.
Representative of country (Level 3)
At present, 180 adult, general critical care units in England, Wales and Northern Ireland are participating in the CMP. This includes 75% (159/213) of all National Health Service units in England, 56% (9/16) in Wales and 73% (8/11) in Northern Ireland (denominator values taken from the Directory of Critical Care ), plus four non-National Health Service units. The median size of units in the CMP is 7 (range 3–22). This compares with median values of 5.3 for ICUs and of 6 for combined ICU/HDUs in the Audit Commission survey . This survey was carried out in 1998 and there has been a considerable increase in critical care bed provision over the past 5 years, so it is reasonable to conclude that the units in the CMP are typical of the country.
Completeness of recruitment (Level 4)
Units participating in the CMP recruit consecutive admissions.
Variables included (Level 3)
The CMPD contains all appropriate variables except for long-term outcome (i.e. beyond ultimate hospital discharge). These include all major known confounders in the form of raw physiology data for APACHE II/APACHE III, for SAPS II and for MPM II.
Completeness of variables (Level 3)
When examined by DoCDat, 84% of all variables in the CMPD were found to be at least 95% complete.
Collection of raw data (Level 4)
All continuous data in the CMPD are collected as raw data.
Explicit definitions (Level 4)/explicit rules (Level 4)
The CMPD has a comprehensive dataset specification for all variables, developed with wide consultation of appropriate parties. The CMPD has a detailed data collection manual provided to all units. Data collection training and retraining are provided.
Reliability of coding (Level 2)
The reliability of data collection in the CMPD is not universally tested and, consequently, this can be considered one of the weakest areas of the CMPD. However, the ICM has been tested and found to have good inter-rater reliability  even though coding the reason for admission is one of the most subjective parts of data collection. Units are encouraged by ICNARC to perform voluntary assessments of reliability for each 6-month cycle by re-collecting a sample of admissions randomly selected by ICNARC. Two or three such reliability assessments are typically performed each year.
Independence of observations (Level 4)
The outcome variables in the CMPD (survival at unit and at hospital discharge) are objective and do not require independent observation.
Data validation (Level 3)
The validation process in the CMPD includes logic, range and consistency checks, although data are not validated against an independent, external source.
The CMPD at the time of analysis contained validated data for 129,647 admissions to 128 adult, general critical care units. The numbers of admissions meeting the exclusion criteria for APACHE II are presented in Table 1. These admissions were excluded from the calculation of APACHE II scores and probabilities only. Measures of case mix, outcome and activity are presented in Table 2.
The median age at admission to the CMP unit was 63 years, and 59% of admissions were male. The mean Acute Physiology Score was 12.5, and the mean APACHE II score was 16.5. Overall, 55% of admissions were nonsurgical, with 26% admitted following elective/scheduled surgery and 19% admitted following emergency/urgent surgery.
The overall mortality was 20.3% in the CMP unit and was 21.5% in any critical care unit. Mortality in the hospital housing the CMP unit was 28.6%. The ultimate hospital mortality was 30.8%.
The median (interquartile range) length of stay was 1.7 (0.8–4.4) days, 2 (1–5) days, 12 (5–25) days and 14 (7–29) days in the CMP unit, in any critical care unit, in the hospital housing the CMP unit and in any hospital, respectively. Survivors had shorter critical care stays but longer hospital stays (Table 2).
The top 10 conditions reported as the primary reason for admission to the CMP unit (from 2211 different ICM codes or partial codes in the CMPD) are shown in Fig. 5. The most common reason for admission was surgery for aortic or iliac dissection or aneurysm (5.7% of all admissions with a primary reason specified), although bacterial pneumonia and pneumonia with no organism isolated were the second and third most common and, when combined, accounted for 6.3% of admissions.
Comparison with other published sources
A number of studies have reported demographics, physiology and outcomes of UK critical care admissions from multicentre databases [20–25]. The results of these studies are presented in Table 3, alongside the equivalent values from the CMPD. The same results are reported for a number of non-UK critical care databases [14, 22, 26–33] in Table 4, including studies from North & South America, Europe and Japan.
The CMPD performs well against the criteria for clinical databases defined by DoCDat, and can be considered a high-quality clinical database. The summary statistics presented for the case mix, outcome and activity of admissions in the CMPD are therefore representative of the country and are accurate.
The authors would encourage any persons considering organising a similar database to pay close attention to the DoCDat criteria and to consider carefully how to address these important issues to ensure their database is representative and accurate.
Determining scores for some elements of the DoCDat evaluation is necessarily subjective (e.g. deciding what constitutes 'good evidence' rather than 'some evidence' that the database is representative of the population, or whether the 'major known confounders' have been included). However, the scores presented for the CMPD were determined by DoCDat and not by the authors.
Particular strengths of the CMPD include its wide coverage, making it highly representative of the population, and explicit definitions for all variables and data collection rules. Collection of raw data enables risk adjustment models to be derived using standard algorithms across all units, allowing for better comparability of risk-adjusted outcomes between units. The main weakness identified by the DoCDat criteria is in the reliability of data collection. While there is no reason to believe that the reliability should be poor, only small-scale reliability studies in individual units have been carried out. The size of the CMP makes formal assessment of reliability across the entire programme a resource-intensive, mammoth task.
Lack of clear instruction in the timing of data collection  and the definition of variables  have been shown to be sources of interobserver variability in the collection of APACHE II data. The CMP uses data collection training, the data collection manual and a precise dataset specification to minimise this variability. Training in data definitions has been shown effective in improving the quality of intensive care data [36, 37].
Previous work on the inter-rater reliability of the ICM for coding reasons to admission has shown agreement of 79% for the specific condition and of 88% for the body system . This compares favourably with a reliability study from the US Project IMPACT database , which showed agreement of 52% and 62% for the specific condition and of 71% and 69% for the body system for reasons for admission to two critical care units coded using the Project IMPACT coding system.
High-quality clinical databases provide the opportunity to perform studies of high generalisability on large numbers of patients at comparatively low cost . Data from multi-centre, high-quality clinical databases can be used for many purposes, including comparative audit, aiding clinical practice, informing health-service management and evaluating health technologies . Data from the CMP are used to provide comparative reports to each unit on a 6-monthly basis, and to provide additional ad hoc reports on specific questions as required by the units. In addition, these data have been used to explore the effects of patient gender  and socioeconomic status (Hutchings A, personal communication, 2002), of day and time of admission to critical care , of time of discharge from critical care  and of end-of-life decision-making  on critical care outcomes.
The use of a detailed system to code the reason for admission to the critical care unit enables identification of groups of patients with specific conditions. This can be of interest not only for common conditions, but also for rare conditions where a meaningful sample can only be obtained using a large multicentre database . When reporting the prevalence of different conditions in the CMPD (Fig. 5), it is important to consider potential sources of variability. These may include over-representation or under-representation of units admitting certain types of patients in the CMP, and the level of detail to which certain conditions are defined in the coding method (e.g. aortic aneurysm surgery would not be the most common reason for admission if the conditions of bacterial pneumonia and pneumonia with no organism isolated were considered a single category).
The results from the CMPD are consistent with the results reported from other multicentre databases of UK critical care admissions (Table 3). They are based on more than twice the number of admissions of the other studies combined, and cover a much wider geographical region than any other single database.
Hospital mortality following admission to intensive care varies widely in different countries (range 16–34%; Table 4). This is in contrast to the results from UK databases that were fairly consistent (range 27–33%; Table 3). The hospital mortality in the CMPD lies towards the top end of that observed internationally, with only the studies from Portugal  and Brazil  reporting higher rates. Methods of case mix (risk) adjustment also varied considerably among the international studies, with only two studies reporting APACHE II hospital mortality probabilities [29, 32]. Most other studies reported either APACHE III score or SAPS II probabilities, while the two studies that did not had an emphasis on re-estimating the mortality equation of an established model in a new population  or evaluating the discrimination and calibration of several models . While we have concentrated on the APACHE II model in this paper, as it was the most widely used in the large UK studies, it is important that the CMPD contains sufficient data to be able to calculate a number of different models to facilitate comparison with other studies.
Risk adjustment has its limitations when used to compare critical care unit outcomes. Methods that rely on the worst values of data recorded over the first 24 hours following admission (e.g. APACHE II and APACHE III, SAPS II) are unable to distinguish between a very sick patient admitted to a good unit and a less sick patient whose condition deteriorates over the first 24 hours due to poor management . Other methods (e.g. MPM II) have similar drawbacks due to relying on variables reflecting treatment (e.g. mechanical ventilation, vasoactive drug treatment). Methods based on data at or around the time of admission (e.g. MPM II0) have other limitations in that they assume all admissions take place at the same time point in the continuum of critical illness. In addition, all the methods have various exclusion criteria, and the exclusions applied in practice are even more varied. The 'observed' mortality of a unit may change considerably depending on exactly which exclusion criteria are applied . As this study was largely descriptive, we applied no exclusion criteria except in the calculation of APACHE II scores and probabilities, where standard exclusions were applied (Table 1).
Accurate comparisons between databases, both within the UK and internationally, can be problematic due to differences in methods of data collection and reporting. Even something as superficially straightforward as applying the exclusion criteria for a risk adjustment method can result in varied interpretation . Precise variable definitions and clear reporting of collection methods can assist in identifying these differences to improve interpretation of results.
This paper forms the baseline for a series of articles on specific conditions in critical care, providing essential background on the data collection, data validation and overall case mix, outcome and activity for all critical care admissions to set those for specific conditions in context. Baseline statistics (case mix, outcome, length of stay) on specific conditions in critical care provide useful and practical information for working clinicians.
The CMP uses rigorous methods of data collection and validation to ensure data are complete, valid and reliable, and the CMPD meets the criteria of a high-quality clinical database. Values derived from the CMPD are consistent with those reported from other multicentre intensive care databases in the UK, but are more precise due to the large sample and are more generalisable due to the wide coverage of the CMP. Results from the CMP are representative and accurate, permitting reliable comparisons both nationally and internationally.
Through the Directory of Clinical Databases http://www.docdat.org, criteria on coverage and accuracy now exist for determining the quality of clinical databases
The Case Mix Programme Database performs well against the ten DoCDat criteria
High quality clinical databases, such as the Case Mix Programme Database, can provide accurate information on the case mix, outcome and activity of patients for health-care providers, managers and purchasers
The Case Mix Programme Database now holds data for over 129,000 admissions to 128 adult, general critical care units in England, Wales and Northern Ireland, and is available for analysis through the Intensive Care National Audit & Research Centre
APACHE = Acute Physiology and Chronic Health Evaluation
CMP = Case Mix Programme
CMPD = Case Mix Programme Database
DoCDat = Directory of Clinical Databases
HDU = high dependency unit
ICM = ICNARC Coding Method
ICNARC = Intensive Care National Audit & Research Centre
ICU = intensive care unit
MPM = Mortality Probability Model
SAPS = Simplified Acute Physiology Score.
Black NA: High-quality clinical databases: breaking down barriers. Lancet 1999, 353: 1205-1206. 10.1016/S0140-6736(99)00108-7
Rowan K, Black N: A bottom-up approach to performance indicators through clinician networks. Health Care UK 2000, spring: 42-46.
Black N, Payne M, on behalf of the DoCDat Development Group: Directory of Clinical Databases: improving and promoting their use. Qual Saf Health Care 2003, 12: 348-352. 10.1136/qhc.12.5.348
Directory of Clinical Databases[http://www.docdat.org]
Rowan K: Intensive Care Society has set up a centre for national audit. BMJ 1996, 313: 1007-1008.
Audit Commission: The Provision of Critical Care Services in England and Wales London: Audit Commission 1998.
Department of Health: Comprehensive Critical Care: A Review of Adult Critical Care services London: Department of Health 2000.
NHS Executive: Health Service Circular HSC 2000/017: Modernising Critical Care Services London: NHS Executive 2000.
Intensive Care National Audit & Research Centre (ICNARC): ICNARC Case Mix Programme Dataset Specification, version 2.0 London: ICNARC 1997.
ICNARC – Adhoc Reports[http://www.icnarc.org/rptindex.php]
Young JD, Goldfrad C, Rowan K: Development and testing of a hierarchical method to code the reason for admission to intensive care units: the ICNARC Coding Method. Br J Anaesth 2001, 87: 543-548. 10.1093/bja/87.4.543
Padkin A, Goldfrad C, Brady AR, Young D, Black N, Rowan K: Epidemiology of severe sepsis occurring in the first 24 hrs in intensive care units in England, Wales, and Northern Ireland. Crit Care Med 2003, 31: 2332-2338. 10.1097/01.CCM.0000085141.75513.2B
Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: a severity of disease classification system. Crit Care Med 1985, 13: 818-829.
Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, Sirio CA, Murphy DJ, Lotring T, Damiano A, Harrell FE Jr: The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalised adults. Chest 1991, 100: 1619-1636.
Le Gall JR, Lemeshow S, Saulnier F: A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicentre study. JAMA 1993, 270: 2957-2963. 10.1001/jama.270.24.2957
Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J: Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients. JAMA 1993, 270: 2478-2486. 10.1001/jama.270.20.2478
Rowan K: Outcome comparisons of intensive care units in Great Britain and Ireland using the APACHE II method. DPhil thesis University of Oxford 1992.
CMA Medical Data: Directory of Critical Care Cambridge: CMAMedical Data 2001.
de Keizer NF, Bonsel GJ, Goldfrad C, Rowan KM: The added value that increasing levels of diagnostic information provide in prognostic models to estimate hospital mortality for adult intensive care patients. Intensive Care Med 2000, 26: 577-584. 10.1007/s001340051207
Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP: Intensive Care Society's APACHE II study in Britain and Ireland – I: variations in case mix of adult admissions to general intensive care units and impact on outcome. BMJ 1993, 307: 972-977.
Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP: Intensive Care Society's APACHE II study in Britain and Ireland – II: outcome comparisons of intensive care units after adjustment for case mix by the American APACHE II method. BMJ 1993, 307: 977-981.
Castella X, Artigas A, Bion J, Kari A: A comparison of severity of illness scoring systems for intensive care patients: results of a multicenter, multinational study. The European/North American Severity Study Group. Crit Care Med 1995, 23: 1327-1335. 10.1097/00003246-199508000-00005
Goldhill DR, Sumner A: Outcomes of intensive care patients in a group of British intensive care units. Crit Care Med 1998, 26: 1337-1345. 10.1097/00003246-199808000-00017
Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD: Assessment of the performance of five intensive care scoring models within a large Scottish database. Crit Care Med 2000, 28: 1820-1827. 10.1097/00003246-200006000-00023
Beck DH, Smith GB, Pappachan JV, Millar B: External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study. Intensive Care Med 2003, 29: 249-256.
Glance LG, Dick AW, Osler TM, Mukamel D: Using hierarchical modeling to measure ICU quality. Intensive Care Med 2003, 29: 2223-2229. 10.1007/s00134-003-1959-9
Bastos PG, Sun X, Wagner DP, Knaus WA, Zimmerman JE, The Brazil APACHE III Study Group: Application of the APACHE III prognostic system in Brazilian intensive care units: a prospective multicenter study. Intensive Care Med 1996, 22: 564-570. 10.1007/s001340050131
Moreno R, Apolone G, Miranda DR: Evaluation of the uniformity of fit of general outcome prediction models. Intensive Care Med 1998, 24: 40-47. 10.1007/s001340050513
de Jonge E, Bosman RJ, van der Voort PH, Korsten HH, Scheffer GJ, de Keizer NF: Intensivecaregeneeskunde in Nederland, 1997–2001. I. Patiëntenpopulatie en resultaten ven behandeling. [Intensive care medicine in the Netherlands, 1997–2001. I. Patient population and treatment outcome.]. Ned Tijdschr Geneeskd 2003, 147: 1013-1017.
Valentin A, Jordan B, Lang T, Hiesmayr M, Metnitz PG: Gender-related differences in intensive care: a multiple-center cohort study of therapeutic interventions and outcome in critically ill patients. Crit Care Med 2003, 31: 1901-1907. 10.1097/01.CCM.0000069347.78151.50
Vázquez Mata G, Jiménez Quintana MM, Rivera Fernández R, Bravo M, Aguayo de Hoyos E, Zimmerman J, Wagner D, Knaus W: Objetivación de la gravedad mediante el sistema APACHE-III aplicado en España. [Severity assessment by APACHE III system in Spain.]. Med Clin (Barc) 2001, 117: 446-451.
Moreno R, Morais P: Outcome prediction in intensive care: results of a prospective, multicentre, Portuguese study. Intensive Care Med 1997, 23: 177-186. 10.1007/s001340050313
Sirio CA, Tajimi K, Taenaka N, Ujike Y, Okamoto K, Katsuya H: A cross-cultural comparison of critical care delivery: Japan and the United States. Chest 2002, 121: 539-548. 10.1378/chest.121.2.539
Chen LM, Martin CM, Morrison TL, Sibbald WJ: Interobserver variability in data collection of the APACHE II score in teaching and community hospitals. Crit Care Med 1999, 27: 1999-2004. 10.1097/00003246-199909000-00046
Féry-Lemonnier E, Landais P, Lorait P, Kleinknecht D, Brivet F: Evaluation of severity scoring systems in ICUs – translation, conversion and definition ambiguities as a source of inter-observer variability in Apache II, SAPS and OSF. Intensive Care Med 1995, 21: 356-360. 10.1007/BF01705416
Arts DGT, Bosman RJ, de Jonge E, Joore JCA, de Keizer NF: Training in data definitions improves quality of intensive care data. Crit Care 2003, 7: 179-184. 10.1186/cc1886
Polderman KH, Jorna EM, Girbes AR: Inter-observer variability in APACHE II scoring: effect of strict guidelines and training. Intensive Care Med 2001, 27: 1365-1369. 10.1007/s001340101012
Cook SF, Visscher WA, Hobbs CL, Williams RL, Project IMPACTClinical Implementation Committee: Project IMPACT: results from a pilot validity study of a new observational database. Crit Care Med 2002, 30: 2765-2770. 10.1097/00003246-200212000-00024
Black N: Developing high quality clinical databases. BMJ 1997, 315: 381-382.
Raine R, Goldfrad C, Rowan K, Black N: Influence of patient gender on admission to intensive care. J Epidemiol Community Health 2002, 56: 418-423. 10.1136/jech.56.6.418
Wunsch H, Mapstone J, Brady T, Hanks R, Rowan K: Hospital mortality associated with day and time of admission to intensive care units. Intensive Care Med 2004, in press.
Goldfrad C, Rowan K: Consequences of discharges from intensive care at night. Lancet 2000, 355: 1138-1142. 10.1016/S0140-6736(00)02062-6
Wunsch H, Brady A, Harrison D, Harvey S, Rowan K: End of life decisions: withdrawal of active treatment in intensive care units in the United Kingdom [abstract]. Intensive Care Med 2003,29(Suppl 1):S14.
Boyd O, Grounds RM: Physiological scoring systems and audit. Lancet 1993, 341: 1573-1574. 10.1016/0140-6736(93)90706-M
Wunsch H, Brady AR, Rowan K: Impact of exclusion criteria from severity of disease scoring methods on outcomes in intensive care. J Crit Care 2004, in press.
Participants of the Case Mix Programme[http://www.icnarc.org/cmpParticipatingUnits.htm]
This study was supported by ICNARC. The authors wish to thank everyone in the critical care units participating in the CMP  and those responsible for local funding. They acknowledge the Department of Health and the Welsh Health Common Services Authority for the initial, 2-year, pump-priming funds, in 1994, to establish ICNARC. The authors would like to thank the DoCDat team for providing summary data.
All authors are employees of ICNARC. KR was a member of the Directory of Clinical Databases Development Group.