Skewed NHS stats are big business

Increasingly commercialised NHS data collection is being inappropriately used in ways that could jeopardise hospitals' futures. Is it any wonder staff might feel under pressure to skew the stats?

Ewen Speed
25 March 2014

The BBC took the Nottinghamshire Healthcare Trust (NHT) to task last week amidst claims that 49% of patient reviews about their experience of healthcare on the Patient Opinion and NHS Choices websites had been posted from NHS computers. 

Patient Opinion and NHS Choices took feedback offline until its authorship could be attributed. They partially defended theselves by saying - not entirely unreasonably - that it was established practice for staff helping patients to make feedback, using NHS computers, because those patients were unable (for a number of reasons) to deal with the practicalities of the online feedback system. 

But the headlines were of a system 'open to abuse'. In a supposed new age of patient-centred transparency (following the terrible events of Mid-Staffs) the actions in Nottingham are presented as a betrayal of the principles of trust that exist between patients and professions.

Why might staff feel compelled to juke the figures? The staff member leaving the positive feedback does not benefit directly from the action. Rather, the benefit is at an organisational level, effectively gaming the performance management metrics their organisation is held accountable to. It is not about manipulating the appearance of the trust in terms of the outward facing public website, the focus is much more on the inward facing quality and performance controls that patient experience feedback is tied into.

A curious observer might ask what these actions tell us about the prevailing management structures within the NHS. A critical observer might even question what these performance metrics tell us about issues of transparency and trust between citizens and government.

This is about political practice, not professional practice. Information and data are the new battleground.

Government has promised an information revolution in the NHS. Jeremy Hunt has claimed that publishing surgical survival rates will “save thousands of lives” and “drive up clinical standards”. Similarly, going paperless by 2018 will save the NHS “billions of pounds”.

But there are all sorts of problems with NHS data. These problems extend across qualitative data like patient feedback and the purportedly harder-edged, more scientific numeric quantitative data.

Consider Hospital Standardised Mortality Ratio (HSMR). The HSMR is a simple calculation, based on the number of actual deaths divided by the number of expected deaths, multiplied by 100. If the number of expected deaths corresponds with the number of actual deaths, then the trust has a score of 100. A score of more than 100 means there are deaths than would be (statistically) expected, and under 100 that there are fewer deaths than would be statistically expected.

These were the data that were at the heart of the Mid-Staffs inquiry, and the more recent ‘scandal’ at Leeds General Infirmary NHS Trust, which resulted in the temporary cessation of children’s heart surgery.

Writing in the London Review of Books Paul Taylor outlines the problems with this particular set of statistics.  He highlights how, as with so much NHS data now, the development of HSMRs in the UK was in a commercial context, by academics who set up the Doctor Foster Intelligence Unit to exploit their potential. This company also publishes the ‘Good Hospital Guide’.

There is widespread disagreement about HSMR’s usefulness as an indicator of poor care. Statisticians have suggested they whilst they may be somewhat indicative of poor care (though this is a moot point) they are overly simplistic or inappropriately interpreted. The Professor leading a review into their use in the NHS, recently suggested they were of no value at all.

What we see is a political mechanism being used to identify what are judged to be poorly performing hospitals. But if the performance metric itself is broken, how can we adjudge the extent to which the hospital is broken?

Take another example, the friends and family test (FFT) - "How likely are you to recommend our ward/A&E department/maternity service to friends and family if they needed similar care or treatment?"

This one question is intended to act as a real world barometer of the level of quality and performances across all units within the NHS. Professor Peter Lynn says it makes no sense because it is hypothetical. It assumes there is a choice, without specifying what that alternate choice might actually be. The measure will skew the rankings of hospitals across the country.

The Friends and Family test “does not confirm to basic scientific standards” and “provides no reliable evidence” according to Rachel Reeves, writing in the Health Service Journal. She argues that such is the poor quality of FFT data that publishing it actually breaches the department of Health’s own publication guidelines. Differences across trusts in how the tests are implemented make any cross trust comparison of the data problematic to say the least.

Both HSMRs and the Friends and Family test have been problematised or even dismissed by experts. But both have been implicated in several high level inquiries about professional practice. And they are central to on-going funding and performance management regimes.

It’s been a bad couple of weeks for the government in terms of healthcare data. The Newsnight report adds a new angle to the data debate. The professionals are not to be trusted with our data either. The news about the patient experience data is portrayed as a fundamental betrayal of trust. The heat is deflected from the allegations of hospital data being sold to insurance companies. 

The Newsnight report does nothing to speak to the real concerns this story raises, around the commercialisation of data, the pressures on healthcare, and whether you can always find a suitable data metric to measure care.  

Had enough of ‘alternative facts’? openDemocracy is different Join the conversation: get our weekly email


We encourage anyone to comment, please consult the oD commenting guidelines if you have any questions.
Audio available Bookmark Check Language Close Comments Download Facebook Link Email Newsletter Newsletter Play Print Share Twitter Youtube Search Instagram WhatsApp yourData