So, what’s my problem? First, we need to be clear what ‘return on investment’ is and why it can be powerful. In brief, it’s a methodology that comes from the economics literature of project appraisal, and is closely related to cost-benefit analysis. It seeks to compare the cost and benefits of alternative actions to see whether the returns are worth the costs of intervening. The key word here is ‘return’. Return on investment methodology moves beyond seeing this simply in terms of financial returns and cost savings to enable comparison between alternatives that provide different sources and types of ‘value’. It does this by monetising them. So, a public health intervention that saves the NHS cash but does little to improve health can be compared to one that doesn’t save the NHS cash, but improves health. The simplest way of doing this is to use the monetary value of a QALY (a quality-adjusted life-year, the standard measure of health outcome which NICE uses to assess value for money in the NHS), which is roughly £20,000 judging by NICE’s decisions on whether the NHS should fund new treatments. Because, as a society, we place a high value on health when estimates of health gain are included in RoI estimates of effective public health interventions, results tend to be very favourable, demonstrating why such interventions can be a wise use of resources.
So far so good. But this is where my concerns come in, because this understanding of RoI is not widely shared. In the public health debate, and particularly when public health interventions are proposed, the ‘return on investment of public health’ is increasingly seen as a synonym for ‘cost-saving’ (either as directly cashable savings or through demand reduction), often, but not exclusively, to the NHS. This ignores the whole point of RoI methodology.
As an illustration of how much impact this can have, I looked at the source studies for The Kings’ Fund and Local Government Association’s infographics on public health RoI. While we were clear about what was included in the ‘the value’ in those infographics we didn’t present the breakdowns in detail. For example, the headline estimated RoI of the Be Active intervention, a large-scale community physical activity intervention in Birmingham, was £23 of ‘value’ for every £1 spent. About 80 per cent of this was the monetised value of health gains, and a much smaller amount was through the impact on health care including demand reduction and small direct health care cost savings. This shouldn’t be a surprise, since the purpose of RoI is to value the overall return, and the purpose of public health interventions is health gain. See our work for South London Health Innovation Network on how the other studies’ RoIs breakdown.
All the studies showed public health is worth doing, and for most of them that is because of the large health gain they lead to. What a surprise! They are interventions which provide high net benefit to society, and are worth paying for. But if we slip into the trap of thinking the ‘RoI of public health’ is the same as cost savings to the NHS or wider system then we will also slip into setting far too high a bar for public health interventions to cross. Despite the clear net benefit to society, not all these interventions will cover their costs through short-term cost savings.
Those of us who advocate for public health also need to be wary: it’s understandable and tempting to take headline RoI figures without looking into what they truly imply and use them to lobby for public health. I do it. But in the long-run this is a mistake. A recent example of this is how the systematic review of the RoI of public health interventions has been misinterpreted by some. The review showed that the median RoI of public health interventions across 52 studies was 14.3:1. This has been widely shared on social media, with many people thinking it shows that for every £1 spent, public health interventions will save the public sector £14 in cash. But that’s not what the findings tell us. We don’t know how that £14 breaks down into cash saving (and if it did, whether savings would fall to the NHS or other sectors) or health or other outcomes of value since the authors didn’t report this information. Most of the studies will – rightly – have included long-term health and wider societal benefits in their ‘returns’, which, after all, is the point of public health. By ‘retelling’ RoI as cost savings we are complicit in setting up public health interventions to fail on the wrong criteria, when it doesn’t deliver that cost saving, but does deliver the health return which is worth paying for.
So, how can we avoid this problem? First, by raising awareness of it, hopefully this blog helps in that respect. Second, by being much clearer about what is in the ‘R’ of RoI and what isn’t when figures are presented and reported in future. There should be more standardisation of inclusion and reporting criteria for return on investment studies in public health. As a starting point for this, RoI studies – and the tools and commentary that refer to them – could routinely and transparently distinguish between three sorts of returns: ‘cashable savings’ that provide public budget holders with a direct financial saving; ‘utilisation reduction’ that reduces the demand pressure on public services but which is not directly cashable as financial savings; and the monetised value of other outputs including health. Public Health England (with the National Institute for Health Research and NICE) could help lead this process and disseminate it across the system.
In conclusion, the bar for public health interventions should not be short-term cost saving to the NHS, it should be a cost-effective use of society’s funds that reflects the value society puts on health and other goals. That’s the bar for NHS treatment and drugs, it should be the same for public health interventions. That’s what RoI and the tools developed above, used and interpreted properly, help to do. Everyone, but especially those with control over what gets funded and prioritised locally and nationally, needs to stop conflating RoI and cost savings. It’s a good sign that Public Health England are increasingly aware of this as a problem. When it comes to RoI, it’s time we got it right.
Great summary. I think the ROI is useful because it broadly tells you, is this worth doing? But if you show it to a finance director, he will think it is nonsense and ask what the cashable savings will be.
I think if you look at NICE's decisions rather than what is quoted on paper you would think the true willingness-to-pay threshold is more like £50,000 - £60,000, not £20,000. But their decisions also depend on the level of certainty around the point estimate of the ICER. And it gets tricky with public health interventions where you have the cross-sector flow problem, for instance drug treatment preventing crime.
If the average production cost of a QALY in the NHS is £13,000 and public health interventions produce QALYs at substantially lower production costs, then you can make the case that it is fair to put this financial value on health because public health interventions are reducing the need for the NHS to produce the same health - but this begs the question, is it the same health? I think there is an artificial line between public health and healthcare with a lot of the most cost effective programmes like vaccinations and immunisations and (arguably) screening happening in the NHS, not being commissioned by local authority public health teams.
There are inconsistencies with valuations of QALYs with some tools using the lower NICE threshold of £20,000, some using UK Treasury value of £60,000, and some going up to £81,000 for criminal justice/drug misuse interventions.
Thank you for this salient reminder about the potential dangers in using these particular tools -- we haven't seen enough discussion and debate in this area.
Our group has modelled the RoI of some of our commissioned interventions using the New Economy CBA tool developed for the New Manchester authority. We undertook this work with one eye firmly on the narratives, both intentional and unintentional, that we'd be creating.
Which programmes, that we invest in, have the best return for our population? Including, but certainly not only in terms of direct financial returns to our health and care system.
Our first effort has, in many ways, raised more questions for us than it has answered, but that is part of the learning process in understanding how to best 'evidence' value and worth and to communicate that message.
One of the most interesting questions that our work has raised, in my view and in terms of telling our story, revolves around who the financial returns and cashable savings accrue to, which the New Economy tool allows you to do. This really makes narrative development around value and worth more challenging, as it exposes the 'silo' like nature of health and care funding. It will be interesting to see how these challenges change as we move towards the new "Dorset Integrated Care System".
Doing the modelling work is only the first step, and is hard enough for most local public health groups, but we still have a long way to go in terms of building the confidence to use and share these types of decision-support tools in the pursuit of improving population health.
In order that change is effectively undertaken all levels of government and other public funded organisations need to demonstrate good management
Independent management accreditation is required to achieve this
Useful blog post.
However, I think part of the post risks causing additional confusion:
"The simplest way of doing this is to use the monetary value of a QALY (a quality-adjusted life-year, the standard measure of health outcome which NICE uses to assess value for money in the NHS), which is roughly £20,000 judging by NICE’s decisions on whether the NHS should fund new treatments. Because, as a society, we place a high value on health when estimates of health gain are included in RoI estimates of effective public health interventions, results tend to be very favourable, demonstrating why such interventions can be a wise use of resources."
This seems to be implying that NICE's cost-effectiveness threshold of £20,000 per QALY represents the value that society places on health. But this is not what the cost-effectiveness threshold represents. It is supposed to represent the marginal production cost per QALY in the NHS, *not* how much society values health.