Table 3 Quality assessment checklist of reporting criteria for validation studies of health administrative dataa
From: Validity of administrative data in recording sepsis: a systematic review
Cevasco et al. [19] | Gedeborg et al. [20] | Grijalva et al. [21] | Ibrahim et al. [22] | Lawson et al. [23] | Madsen et al. [24] | Ollendorf et al. [25] | Schneeweiss et al. [28] | Quan et al. [26] | Iwashyna et al. [12] | Ramanathan et al. [27] | Whittaker et al. [29] | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
1. Identify article as study of assessing diagnostic accuracy | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
2. Identify article as study of administrative data | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
3. State disease identification & validation as goals of study | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Methods: participants in validation cohort | ||||||||||||
4. Age | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 |
5. Disease | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
6. Severity | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
7. Location/jurisdiction | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 |
8. Describe recruitment procedure of validation cohort | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
9. Inclusion criteria | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
10. Exclusion criteria | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 |
11. Describe patient sampling (random, consecutive, all, etc.) | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
12. Describe data collection | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
13. Who identified patients and did selection adhere to patient recruitment criteria | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
14. Who collected data | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
15. A priori data collection form | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 |
16. Disease classification | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
17. Split sample (that is, revalidation using a separate cohort) | 0 | 0 | 0 | 0 | 0 | U | 0 | 0 | 0 | 0 | 0 | 0 |
Test methods | ||||||||||||
18. Describe number, training and expertise of persons reading reference standard | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
19. If more than one person reading reference standard, quote measure of consistency (for example, κ) | 1 | 0 | 1 | N/A | N/A | 0 | 0 | N/A | 0 | 0 | 0 | 0 |
20. Blinding of interpreters of reference standard to results of classification by administrative data (for example, chart abstractor blinded to how that chart was coded) | U | 1 | 1 | U | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 |
Statistical methods | ||||||||||||
21. Describe methods of calculating diagnostic accuracy | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 |
Results: participants: | ||||||||||||
22. Report when study done, start/end dates of enrolment | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
23. Describe number of people who satisfied inclusion/exclusion criteria | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
24. Study flow diagram | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
Test results: | ||||||||||||
25. Report distribution of disease severity | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | 1 |
26. Report cross-tabulation of index tests by results of reference standard | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 |
27. Report at least four estimates of diagnostic accuracy | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 |
Diagnostic accuracy measures reported | ||||||||||||
28. Sensitivity | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 |
29. Specificity | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 |
30. PPV | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
31. NPV | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 |
32. Likelihood ratios | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
33. κ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
34. Area under the ROC curve/C-statistic | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
35. Accuracy/agreement | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
36. Other (specify) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
37. Report accuracy for subgroups (for example, age, geography) | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 |
38. If PPV/NPV reported, does the ratio of cases/controls of validation cohort approximate prevalence of condition in the population? | 1 | 1 | N/A | N/A | 1 | N/A | N/A | N/A | N/A | 0 | 0 | N/A |
39. Report 95% CI for each diagnostic measure | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 |
Discussion | ||||||||||||
40. Discuss the applicability of the validation findings | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
Total score | 27 | 25 | 27 | 30 | 28 | 24 | 10 | 22 | 28 | 29 | 24 | 26 |