Book contents
- Frontmatter
- Contents
- List of contributors
- Acknowledgements
- Introduction: Accountability, informed consent and clinician performance information
- Part I Accountability
- Part II Informed consent
- Part III Reporting performance information
- Part introduction
- 13 Is the reporting of an individual surgeon's clinical performance doing more harm than good for patient care?
- 14 Examining the link between publicly reporting healthcare quality and quality improvement
- 15 Hospital and clinician performance data: what it can and cannot tell us
- 16 An ethical analysis of the defensive surgery objection to individual surgeon report cards
- 17 Surgeon report cards and the concept of defensive medicine
- 18 Training, innovation and surgeons' report cards
- 19 Doctors' report cards: a legal perspective
- Index
- References
15 - Hospital and clinician performance data: what it can and cannot tell us
Published online by Cambridge University Press: 08 August 2009
By
Book contents
- Frontmatter
- Contents
- List of contributors
- Acknowledgements
- Introduction: Accountability, informed consent and clinician performance information
- Part I Accountability
- Part II Informed consent
- Part III Reporting performance information
- Part introduction
- 13 Is the reporting of an individual surgeon's clinical performance doing more harm than good for patient care?
- 14 Examining the link between publicly reporting healthcare quality and quality improvement
- 15 Hospital and clinician performance data: what it can and cannot tell us
- 16 An ethical analysis of the defensive surgery objection to individual surgeon report cards
- 17 Surgeon report cards and the concept of defensive medicine
- 18 Training, innovation and surgeons' report cards
- 19 Doctors' report cards: a legal perspective
- Index
- References
Summary
Introduction
The monitoring of healthcare performance has a surprisingly long history, going back to the work of Florence Nightingale, who produced analyses of mortality outcome measures and campaigned for uniform hospital and surgical statistics (Spiegelhalter, 1999). More recently, there has been a renewed focus on monitoring clinical standards in many countries' health services, particularly in the UK in light of high-profile cases like Bristol (The Bristol Royal Infirmary Inquiry, 2001) and Shipman (Baker, 2001; Aylin et al., 2003a) and in the US with the Agency for Healthcare Research and Quality's Patient Safety Initiative (Agency for Healthcare Research and Quality, 2003) and the Institute for Healthcare Improvement's 100 000 Lives Campaign (Institute for Healthcare Improvement).
Monitoring of performance data is not straightforward, with many pitfalls and it is important to get it right. A recent report from the Royal Statistical Society noted that: ‘Performance monitoring done well is broadly productive for those concerned. Done badly, it can be very costly and not merely ineffective but harmful and indeed destructive’ (The Royal Statistical Society Working Party on Performance Monitoring of Public Services, 2003).
This chapter looks at:
Sources of data, both administrative and clinical, and their quality
Statistical issues around performance monitoring
Methods and presentation of data
How to deal with analyses suggesting poor performance
Sources of data
The Bristol Inquiry report concluded that ‘Bristol was awash with data’ (The Bristol Royal Infirmary Inquiry, 2001).
- Type
- Chapter
- Information
- Informed Consent and Clinician AccountabilityThe Ethics of Report Cards on Surgeon Performance, pp. 226 - 242Publisher: Cambridge University PressPrint publication year: 2007
References
100k lives Campaign. Institute for Healthcare Improvement. Available from: URL; http://www.ihi.org/IHI/Programs/Campaign/.
AHRQ's Patient Safety Initiative: Building Foundations, Reducing Risk. ‘Interim report to the Senate Committee on appropriations.’ AHRQ Publication 04-RG005, December 2003. Agency for Healthcare Research and Quality, Rockville, MD. Available from: URL; http://www.ahrq.gov/qual/pscongrpt/.
Aylin, P., Alves, B., Cook, A. et al. Analysis of hospital episode statistics for the Bristol Royal Infirmary inquiry. (1999). Available from: URL: http://www.bristol-inquiry.org.uk/Documents/hes_(Aylin).pdf.
, , et al. (2001). Comparison of UK paediatric cardiac surgical performance by analysis of routinely collected data 1984–96: was Bristol an outlier?The Lancet, 358, 181–7.CrossRefGoogle Scholar
, , and (2003a). Following Shipman: a pilot system for monitoring mortality rates in primary care. The Lancet, 362, 485–91.CrossRefGoogle Scholar
Aylin, P., Best, N., Bottle, A. and Marshall, C. (2003b). Methodological issues in the analysis of performance data. A discussion paper produced for the Office for Information on Healthcare Performance.
, and (2004). Cross sectional survey of multicentre clinical databases in the United Kingdom. British Medical Journal, 328, 1478.CrossRefGoogle ScholarPubMed
The Bristol Royal Infirmary Inquiry (2001). Learning from Bristol: the report of the public inquiry into children's heart surgery at the Bristol Royal Infirmary 1984–1995. Available from: URL; http://www.bristol-inquiry.org.uk/.
, and (1990). How much change in the case mix index is DRG creep?Journal of Health Economics, 9, 411–28.CrossRefGoogle ScholarPubMed
and (1997). Improving the statistical approach to health care provider profiling. Annals of Internal Medicine, 127, 764–8.CrossRefGoogle ScholarPubMed
Commission for Health Improvement (2001). Report on the investigation into heart and lung transplantation at St George's Healthcare NHS Trust. http://www.chi.gov.uk/eng/organisations/london/st_georges/2001/st_georges.pdf.
Department of Health (England) (2003/4). Hospital Episode Statistics (2003/4). Available at: http://www.hesonline.nhs.uk/Ease/servlet/AttachmentRetriever?site_id=1802&file_name=:\efmfiles\1802\A Free Data Jane\Tables-21\Tb 21 03-04.xls&short_name=Tb 21 03-04.xls&u_id=2460.
(2004). Payment by results – new financial flows in the NHS. British Medical Journal, 328, 969–70.CrossRefGoogle ScholarPubMed
, , , , and (1998). Assessment of the reproducibility of clinical coding in routinely collected hospital activity data: a study of two hospitals. Journal of Public Health Medicine, 20, 63–9.CrossRefGoogle ScholarPubMed
Dr Foster Intelligence. Available from URL: http://www.drfoster.co.uk/.
Elliott, P. and Wakefield, J. C. (2000). Bias and confounding in spatial epidemiology. In Spatial Epidemiology – Methods and Applications, ed. , , and . Oxford: Oxford University Press, pp. 68–84.Google Scholar
European System for cardiac operative risk evaluation. Available from: URL; http://www.euroscore.org/.
Fourth National Vascular Database Report. Vascular Surgical Society of Great Britain and Ireland. (2004). Available: http://www.vascularsociety.org.uk/Docs/nvdr2004.pdf.
, , and (2004). Survival after surgery or therapeutic catheterisation for congenital heart disease in children in the United Kingdom: analysis of the central cardiac audit database for 2000–1. British Medical Journal, 328, 611–15.CrossRefGoogle ScholarPubMed
and (1996). League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society, Series A, 159, 385–443.CrossRefGoogle Scholar
, , and (2001). Accessing and using hospital activity data. Journal of Public Health Medicine, 21, 51–6.CrossRefGoogle Scholar
The Healthcare Commission (2005). 2005 Performance Ratings. Available from URL: http://www.healthcarecommission.org.uk/.
HES Data Quality Indicator (DQI) reports for 2003–04. Available from: URL; http://www.hesonline.nhs.uk/Ease/servlet/DynamicPageBuild?siteID=1802&categoryID=405&catName=DQI%20Reports%20for%202003-04.
, , et al. (2002). Performance league tables – letters. British Medical Journal, 324, 542.CrossRefGoogle Scholar
(1986). The exponentially weighted moving average. Journal of Quality Technology, 18, 203–10.CrossRefGoogle Scholar
, , , , and (2002). Consensus-based method for risk adjustment for surgery for congenital heart disease. Journal of Thoracic and Cardiovascular Surgery, 123, 110–18.CrossRefGoogle ScholarPubMed
(2005). Surgery for congenital heart conditions in Oxford. British Medical Journal, 330, 319–20.CrossRefGoogle ScholarPubMed
, , and (2004). Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. The Lancet, 363, 1147–54.CrossRefGoogle ScholarPubMed
, , , and (1997). Monitoring the results of cardiac surgery by variable life-adjusted display. The Lancet, 350, 1128–30.CrossRefGoogle ScholarPubMed
and (1998). League tables of in vitro fertilisation clinics: how confident can we be about the rankings?British Medical Journal, 316, 1701–4.CrossRefGoogle Scholar
, , and (2004). Statistical issues in the prospective monitoring of health outcomes at multiple units. Journal of the Royal Statistical Society, Series A, 167(3), 541–59.CrossRefGoogle Scholar
(1993). Routine data: a resource for clinical audit?Quality Health Care, 2, 104–11.CrossRefGoogle ScholarPubMed
, and (1994). Making routine data adequate to support clinical audit. British Medical Journal, 309, 1246–7.CrossRefGoogle ScholarPubMed
, , and (2001). Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. The Lancet, 357, 463–7.CrossRefGoogle ScholarPubMed
New York State Department of Health (2001). Coronary Artery Bypass Graft Surgery in New York State 1996–1998, New York State Department of Health. Available from: URL: http://www.health.state.ny.us/nysdoh/consumer/heart/1996-98cabg.pdf.
, and (1997). Statistical methods for profiling providers of medical care: issues and applications. Journal of the American Statistical Association, 92, 803–14.CrossRefGoogle Scholar
, and (1989). A method of uniform stratification of risk for evaluating the results of surgery in acquired heart disease. Circulation, 79, I3–I12.Google Scholar
(1998). Half of all doctors are below average. British Medical Journal, 316, 1734–6.CrossRefGoogle ScholarPubMed
, and (1998). Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. British Medical Journal, 316, 1697–700.CrossRefGoogle ScholarPubMed
, and (2005). Potential use of routine databases in health technology assessment. Health Technology Assessment, 9(20). Available from: URL; http://www.hta.nhsweb.nhs.uk/fullmono/mon920.pdf.CrossRefGoogle ScholarPubMed
(1972). Computers in Medicine Series: Hospital Activity Analysis. London: Butterworths.Google Scholar
The Royal Statistical Society Working Party on Performance Monitoring of Public Services (2003). Performance indicators: good, bad and ugly. Available from:URL; http://www.rss.org.uk.
, , , and (1994). Attempt at deriving a formula for setting general practitioner fundholding budgets. British Medical Journal, 309, 1059–64.CrossRefGoogle ScholarPubMed
The Society of Cardiothoracic Surgeons of Great Britain and Ireland Fifth National Adult Cardiac Surgical Database Report 2003. (2004). Dendrite clinical systems. Henley-upon-Thames: The Society of Cardiothoracic Surgeons of Great Britain and Ireland.
(1999). Surgical audit: statistical lessons from Nightingale and Codman. Journal of the Royal Statistical Society, 162, 45–58.CrossRefGoogle Scholar
(2002). Funnel plots for institutional comparison. Quality and Safety in Health Care, 11, 390–1.CrossRefGoogle ScholarPubMed
(2005). Funnel plots for comparing institutional performance. Statistics in Medicine, 24, 1185–202.CrossRefGoogle ScholarPubMed
Spiegelhalter, D., Evans, S. J. W., Aylin, P. and Murray, G. (2000). Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995. Available from: URL; http://www.bristol-inquiry.org.uk/Documents/statistical overview report.pdf.
, , , and (2002). Commissioned analysis of surgical performance using routine data: lessons from the Bristol Inquiry. Journal of the Royal Statistical Society, Series A, 165, (2), 1–31.CrossRefGoogle Scholar
, , et al. (2002). Mortality rates after surgery for congenital heart defects in children and surgeon's performance. The Lancet, 355, 1004–7.CrossRefGoogle Scholar
, , and (2002). Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics, 1(4), 441–52.CrossRefGoogle Scholar
, , and (2002). The Surgical Risk Scale as an improved tool for risk-adjusted analysis in comparative surgical audit. British Journal of Statistics, 89, 763–8.Google ScholarPubMed
, and (1993). A sequential procedure for surveillance of Down's syndrome. Statistics in Medicine, 12, 13–25.Google Scholar
(2004). Risk adjustment for coronary artery bypass graft surgery: an administrative approach versus EuroSCORE. International Journal for Quality in Health Care, 16, 157.CrossRefGoogle ScholarPubMed
and (1999). A study of the average run length characteristics of the national notifiable disease surveillance system. Statistics in Medicine, 18, 3309–19.3.0.CO;2-G>CrossRefGoogle Scholar
, and (1993). Comparison of the quality of patient data collected by hospital and departmental computer systems. Health Trends, 25, 105–8.Google ScholarPubMed