Jeremy Hunt’s Way with Statistics

‘Around 6000 people lose their lives every year because we do not have a proper seven-day service in hospitals,’ Jeremy Hunt said on 16 July 2015. ‘You are 15 per cent more likely to die if you are admitted on a Sunday compared to being admitted on a Wednesday.’ A Department of Health statement later clarified that the figures came from an analysis ‘soon to be published in the BMJ’. Nick Freemantle, a professor of epidemiology at UCL, had been invited by Bruce Keogh, the chief medical officer, to update a 2012 analysis of hospital data, apparently on the suggestion of Simon Stevens, the new chief executive of NHS England. The resulting paper wasn’t accepted by the BMJ until 29 July, after Hunt’s speech. When it appeared in September, it contained no reference to the 6000 figure.

After publication, Keogh told the press that the results represented a call to action. The story was picked up by the media across the world. Hunt, locked in an increasingly bitter dispute with junior doctors over working practices and rates of pay, began quoting a different figure from the paper: 11,000 patients are dying unnecessarily each year following a weekend admission.

The paper is, unsurprisingly, controversial. The editor of the BMJ has written to Hunt to take him to task for misrepresenting the paper’s conclusions by asserting that the 11,000 ‘excess’ deaths are avoidable. Outraged junior doctors have queued to register their disapproval in the ‘rapid response’ section of the BMJ, and numerous blogs have picked apart the analysis and the ‘zombie statistic’ of 11,000 preventable deaths.

An odd thing about the paper is that it actually shows hospital patients are less likely to die at the weekend. The risk is associated with being admitted at the weekend. Critics have argued that this is simply because patients admitted at the weekend tend to be sicker. But Freemantle and his colleagues’ analysis is pretty robust. They built a complex statistical model which adjusted for differences in age, sex, ethnicity, whether or not the admission was an emergency, whether or not it was a transfer from another hospital, the diagnosis, number of previous emergency admissions, number of previous complex admissions, co-morbidities, social deprivation, hospital trust, and day of the year. Faced with this level of statistical sophistication, critics tend to fall back on the intuition that if patients aren’t dying at weekends, the problem isn’t with the care at weekends.

No model is perfect, and it’s difficult to use mortality as a measure of the quality of care because very few patients die, and of those who do, most die unavoidably. So the signal detected is generated by a tiny sub-sample of the data. This is a particular issue in Freemantle’s study because the numbers of admissions in the two groups are quite different: there are many fewer admissions at weekends. Some critics have argued that this means the data are susceptible to differences in the way they are recorded at weekends. (Though if there is a bias, it could work the other way: the threshold for being classed as an emergency, for example, is probably lower at the weekend.) There will be effects that aren’t captured in the model: it’s possible there isn’t a weekend effect but an out-of-hours effect, which is more apparent at weekends because a much higher proportion of patients admitted on Saturday or Sunday are admitted at night. All of that said, the model is pretty sound, the effect is strong, and the results are in accordance with other studies from around the world.

You can’t expect politicians to treat statistics in the same way as academics. The distinction between deaths which are ‘excess’ – i.e. which are a measure of the difference between the mortality rates for patients admitted at the weekend and during the week – and deaths which are ‘avoidable’ might seem too subtle for political rhetoric, but it is huge. Part of the excess may well be a consequence of the differences in the way hospitals are organised at weekends, but if we don’t know what it is about the process that causes the problem, we can’t know how to solve it. Hunt has focused on junior doctors’ working practices. There is nothing in the evidence to suggest that changing these will have the required effect. There is more evidence that hiring extra nurses would make a difference, but presumably no way of achieving this without spending money. There is, in fact, no reason to believe that the problem is with hospitals; it could equally be explained by problems with the standards of nursing in care homes after 5 p.m. on Fridays or at GP surgeries on Saturday mornings.