Last month Channel 4 News carried lengthy and uncritical coverage of work by Brian Jarman comparing hospital mortality in seven Western countries between 2004 and 2012. The headline claims were that English “health service patients are 45% more likely to die in hospital than in the US,”which was the leading (and only named) country of the seven being compared.
This was followed by newspaper coverage including claims that “A patient in England was five times as likely to die of pneumonia and twice as likely to die of septicaemia compared to similar patients in the US.”
The basis of these claims was questioned on Twitter and in online articles, and blogs, particularly as neither the data nor the methods were publicly available—it is perhaps notable that the BBC’s website did not cover the story at all. To his credit, Jarman responded with a torrent of robust tweets and provided links to files with some limited details of the methods and results.
Health systems differ
Nevertheless, it is frustratingly difficult to assess the evidence for his conclusions because of lack of information about the data, methods, hospitals, and even countries involved. Jarman seems to have pooled routinely collected, individual level data on hospital discharges from the seven countries, used in-hospital mortality as an outcome, and fitted a common prediction model using age, sex, emergency or elective admission, comorbidity score and diagnosis using the Agency for Healthcare Research and Quality clinical classification system, which is based on ICD-9 codes.
This enabled him to calculate an expected mortality risk for a hospital’s admissions and so obtain a hospital standardised mortality ratio (HSMR) for each hospital.
Criticism has been focused on the comparison with the US. Indeed, in the document provided to Channel 4 News, Jarman acknowledges that the US “has lower life expectancy and higher infant mortality rates” than the UK and “there is a disincentive for poorer people to be admitted” (it is notable that this international HSMR, unlike the UK version, does not adjust for deprivation). In addition, the comparability of coding can be questioned because of the known practice of “up-coding” in the US to increase reimbursement  and possibly different use of terms such as pneumonia and sepsis.
There also seem to be wide international differences in discharge policies before death—a recent study estimated that 78% of deaths in Japan occur in hospital, compared with 56% in England and Wales, 45% in the US, and 34% in the Netherlands. 
This will have an important effect on in-hospital mortality—other countries may rapidly move patients into intermediate care facilities, an option that is not readily available in the UK. Jarman himself observed in 2004 that “In-hospital death rates are 4.9% in the US compared with 9.3% in England, suggesting that people are more likely to die out of hospital in the United States” - a similar finding to his current analysis but with a rather different emphasis. Given that there is also general scepticism about the HSMR methodology, my personal inclination is to take little notice of the overall comparison with the US.
Over-reliance on the media
Jarman is clearly passionate about improving the NHS and has been frustrated at the lack of interest that has been taken in his analyses over the years. This has led him to a personal crusade, sidelining the usual routes of scientific papers and worthy reports by committees and to make direct contacts with the media.
However, just as with the previous Telegraph story about “13,000 needless deaths” (currently subject to an investigation by the Press Complaints Commission after it received two complaints, one from me), he seems to trust the media to report his caveats. They almost invariably fail to do so.
We are entering the era of “big data” and, although you can’t help but be impressed at anyone who does logistic regressions with 21 000 000 observations, this is a fine example of when size is not everything—rather, we need data that are fit for the purpose of comparing what would happen to a similar patient were they admitted to different hospitals worldwide.
And unless big also means open data, it is impossible for outside observers to verify the analysis and interpretation, especially when the stories are trumpeted by media with an apparent vested interest in running down the NHS. This inevitably breeds suspicion and scepticism.
Of course, it would be deluded to deny there are serious problems in aspects of the NHS, or that we could not learn from good evidence of improved outcomes for comparable patients, and the culture of safety that exists in the best hospitals in the US and elsewhere. Channel 4 featured the Mayo hospital in Arizona as an example, and ventures such as Risky Business have been pressing these issues for years.
The recent Berwick report into patient safety proclaimed that “The NHS in England can become the safest health care system in the world.”
But that this would “require unified will, optimism, investment, and change.” If it takes this kind of publicity to bring these issues to increased prominence and contribute towards a cultural shift, then we should not complain.
But this was an exercise in closed data, and I remain sceptical about the specific statistical claims.
 Silverman E, Skinner J. Medicare upcoding and hospital ownership. J Health Econ 2004;23:369-89.
 Broad JB, Gott M, Kim H, Boyd M, Chen H, Connolly MJ. Where do people die? An international comparison of the percentage of deaths occurring in hospital and residential aged care settings in 45 populations, using published and available statistics. Int J Public Health 2013;58:257-67.
Martin McKee, Professor of European Public Health at the London School of Hygeine and Tropical Medicine, adds:
As Spiegelhalter notes, not knowing which countries are included in the analysis presented by Sir Brian Jarman makes it difficult to assess their validity. However, it is instructive to look at what researchers studying hospital standardised mortality rates (HSMRs) in other countries have concluded.
A Danish study  noted that “We have analysed the available data from 2007 through 2011 and cannot reconcile the quite substantial and often sudden changes in HSMRs with changes in quality of care, but believe that they are due to inherent noise in calculating HSMRs, e.g. owing to variable quality of the diagnostic coding."
A Canadian paper  “found a lack of empirical evidence supporting the use of the HSMR in measuring reductions in preventable deaths. We also found that limitations in standardization as well as differences in palliative care coding and place of death make inter-facility comparisons of HSMRs invalid … It should not be viewed as an important indicator of patient safety or quality of care."
In Australia, researchers decided that "Despite its apparent low cost and ease of measurement, the HSMR is currently not "fit for purpose" as a screening tool for detecting low-quality hospitals and should not be used in making interhospital comparisons."
One Dutch paper, noting improvements on hospital mortality, concluded "There can be many reasons … including improved quality of care; however, it may also be due to, for instance, changes in hospital admission and discharge policies" while another, examining the effect of inflation of denominators by increased readmissions found that models that did or did not adjust for this” produced substantially different HSMR outcomes."
These conclusions suggest that rather more thought might have been appropriate before publicising these findings on national television.
 Gerdes LU, Poulstrup A. [Hospital standardised mortality ratios do not with certainty reflect the quality of patient care]. Ugeskr Laeger 2012; 174: 1590-4.
 Penfold RB, Dean S, Flemons W, Moffatt M. Do hospital standardized mortality ratios measure patient safety? HSMRs in the Winnipeg Regional Health Authority. Healthc Pap 2008; 8: 8-24.
 Scott IA, Brand CA, Phelps GE, Barker AL, Cameron PA.Using hospital standardised mortality ratios to assess quality of care--proceed with extreme caution. Med J Aust 2011; 194: 645-8.
 Ploemacher J, Israëls AZ, van der Laan DJ, de Bruin A. [Standardised in-hospital mortality decreasing over time]. Ned Tijdschr Geneeskd. 2013;157(17):A5267.
 van den Bosch WF, Spreeuwenberg P, Wagner C. Variations in hospital standardised mortality ratios (HSMR) as a result of frequent readmissions. BMC Health Serv Res. 2012 Apr 4;12:91. doi: 10.1186/1472-6963-12-91.
This piece first appeared in the British Medical Journal.
Get our weekly email