Yet more performance ratings for the NHS: new STP ratings are narrowly focused and centrally driven

This content relates to the following topics:

In July NHS England published its first 'progress dashboard' for the NHS’s 44 new sustainability and transformation partnerships (STPs) – the groups of NHS organisations and local authorities expected to deliver the NHS’s sustainability and transformation plans (also, confusingly, referred to as STPs). The plans cover all areas of NHS spending in England up to 2021. The plans, and the partnerships set up to deliver them, represent a major shift in approach to NHS reform, prioritising collaboration over competition as the route to improving services and reducing financial deficits.

The new dashboard aims to provide a 'baseline' for tracking improvements made by STPs. But it also offers judgments: each STP is given a summary assessment of 'overall progress', ranging from a top rating of 'outstanding' (category 1) to the lowest of 'needs most improvement' (category 4). Five STPs were rated outstanding and another five were rated as poorest.

Unlike the ratings of individual NHS and social care providers’ performance by the regulator the Care Quality Commission, these new ratings are intended to reflect the collective performance of health and care services in geographical areas. Though the aim of measuring performance across local health systems is a good one, as we recommended in a review for the Department of Health in 2015, there are at least four major problems with these latest ratings.

Firstly, they are too narrow in scope to fairly reflect system-wide performance. STPs have been tasked with improving services right across the health and care system: from primary prevention and community based services to specialised care in hospitals. Yet the dashboard includes only 17 performance indicators, imbalanced towards hospitals. Even the three indicators chosen to assess 'prevention' are defined in relation to inpatient care: emergency admissions, emergency bed days, and delayed transfers from hospital.

General practice, mental health, and cancer care are chosen as other areas to be measured, following the national priorities recently outlined by NHS England. But indicators in these areas are limited too. General practice, for example, includes only extended access to appointments and patients’ experience of making appointments. The lack of indicators on social care and public health is unlikely to help STPs in their ambition to work closely with local government.

Secondly, the summary ratings are too simplistic to be meaningful. Marking an STP 'outstanding' or 'needs most improvement' may be superficially clear and easy to understand. But it conceals the more complex reality of health system performance that lies underneath. In the end, ratings are artefactual constructs based on the choice of indicators, how they are scored and weighted, and the methods used to aggregate them. For example, just five indicators in the new STP ratings (emergency admissions, emergency bed days, delayed discharges, leadership, and finance) carry a 50 per cent weighting in the overall scores. This weighting, of course, reflects political choices and current national priorities – and NHS England states that the rating system will evolve over time. But different indicators with different weightings would produce different results. This has important consequences for local areas, given that the ratings are being used to make decisions about STP funding.

Thirdly, the ratings have potential perverse effects. Though ratings can help improve performance in the areas they cover, past experience in the NHS shows that they can also have multiple unintended consequences, such as distorting local and clinical priorities, gaming, and damaging organisational culture and staff morale.

And fourthly, the ratings add yet another layer to the already fragmented and confused approach to measuring performance in the NHS. The STP dashboard sits alongside ratings for clinical commissioning groups, providers of NHS and social care, and the increasingly forgotten national outcomes frameworks for the NHS, public health, and adult social care. Some indicators in these frameworks align; others do not. The ratings also sit alongside various other financial targets for NHS organisations and constitutional requirements. So which performance measures matter? Paradoxically, the measures that matter to STPs and the public they serve may not be included in the ratings at all, given that STPs’ priorities were originally intended to be developed from the 'bottom up' to fit the local context and needs of the population.

STPs are based on the principle that collective action is needed to improve services and manage limited resources. This logic is good. So too is the principle that STPs’ progress should be measured and reported. But what is this measurement for? Our view is that the priority for national NHS bodies should be to use measurement to support local learning and improvement. This means providing data that are meaningful to STPs and helping them use the data to inform action.

But old habits die hard. The use of summary ratings for top-down performance management has a long history in the NHS. There is a clear danger that the new STP ratings will be used primarily for this purpose too – shifting the balance further towards central control over the NHS in England rather than enhancing local engagement, at a time when the OECD recommends shifting the balance in the opposite direction.

Comments

Roger Steer

Organisation
HAC
Comment date
10 August 2017
Hugh could have mentioned that they also present a misleading picture.
There is no way anyone who takes the trouble to read the STPs of the top 5 "outstanding" candidates could come to the conclusion that plans exist to enter this brave new world.
A lot of water has flowed since the end of 2016 but the ever shifting sands putting the onus of STPs onto Partnerships rather than Plans and on creating ACOs rather than delivering STPs gives the game away.

Steve Benfield

Position
Chief Executive,
Organisation
The Beech Centre - aspirant Institute for OD
Comment date
11 August 2017
The reality remains that such scoring and judgements remain subjective, can drive dysfunctional behaviour and survival strategies and can encourage gaming to protect contingent parts and particularly organisational positions/interests.

Unfortunately, the challenges systemically within the NHS and its various parts remain that it is a complex system. With complex systems, they can't ever be fixed or perfect. However, what is possible with a complex system is to intervene with changes, progress, great OD etc. and behave with it in a way that makes a positive difference. You can of course also behave with the same set of circumstances and have negative difference. So, you can either make it better or make it worse overall - you can't fix it sustainably.

I worked with PCT's when they changed into CCG's. The same change in all but one example had leadership that intervened in a way that made this a positive and sustainable difference and another example in which the leadership made it a negative experience - just because of a difference in energy - leadership is about energy into the system - its not a job title!

So, some STP's will work well because of the leadership and true collaborative behaviours and others will be problematic. If we focused on leadership behaviours and culture as the way to make a judgement on "outstanding" and through to "needs improvement" - then we might see a different response to such initiatives.

Add your comment