A framework for understanding the impact of regulation
We have developed a new framework that outlines eight ways in which regulation can affect provider performance. This framework will help regulators, providers and policy-makers to understand the impact of regulation. It shows that regulation has an impact before, during and after inspection and through interactions between regulators, providers and other key stakeholders.
How do the different types of impact work in practice?
We used this framework to explore the impact of CQC inspection and rating in six areas of England and found examples of each type, although there was more evidence of some types of impact than others.
Table 1: Eight regulatory impact mechanisms
|Impact mechanism||Description of logic/causal chain/process||Example|
|Anticipatory||The regulator sets quality expectations, and providers understand those expectations and seek compliance in advance of any regulatory interaction.||Before their CQC inspection, a trust held discussions to develop its values and behaviours and to encourage a sense of ownership of these among staff.|
|Directive||Providers take actions that they have been directed or guided to take by the regulator. This includes enforcement actions and, at the extreme, may involve formal legal repercussions such as prosecution or cancellation of registration.||CQC inspected and then closed a GP provider. Others in the system said they had been aware of a performance issue but didn’t have the evidence or power to address it.|
|Organisational||Regulatory interaction leads to internal organisational developments, reflection and analysis by providers that are not related to specific CQC directions. This leads to changes in areas such as internal team dynamics, leadership, culture, motivation and whistleblowing.||Motivated by a CQC visit, a mental health trust became much better at setting organisational objectives, giving them a greater focus on ongoing improvement, with staff and managers regularly reflecting on the direction of travel and the resources they need to deliver those objectives.|
|Relational||Results from the nature of relationships between regulatory staff (ie, inspectors) and regulated providers. Informal, soft, influencing actions have an impact on providers.||A mental health provider had regular meetings with CQC, at which the provider highlighted the challenges the organisation was facing and received verbal feedback from the CQC.|
|Informational||The regulator collates intelligence and puts|
information about provider performance into the public domain or shares it with other actors who then use it for decision-making (eg, commissioning, patient choice).
|Relatives of those living in a care home mentioned they had looked at the home’s CQC report online when selecting the provider.|
|Stakeholder||Regulatory actions encourage, mandate or|
influence other stakeholders to take action
or to interact with the regulated provider.
|NHS Improvement and NHS England worked with a trust following their inspection to address quality issues identified in their report.|
|Lateral||Regulatory interactions stimulate inter-organisational interactions, such as providers working with their peers to share learning and undertake improvement work.||An acute provider encouraged its staff to take on inspector roles so that they could learn to better gauge their own performance and prepare for their|
|Systemic||Aggregated findings/information from regulation are used to identify systemic or inter-organisational issues, and to influence stakeholders and wider systems other than the regulated providers themselves.||CQC’s annual State of Care report focused national attention on the challenge of the sustainability of the social care sector.|
For example, there was a lot of evidence of anticipatory impact: the CQC set expectations of quality and providers responded to these in advance of inspection. This had some positive effects (eg, it helped some organisations to prioritise quality issues) and some negative effects (eg, some providers focused a lot of energy on getting through the inspection process rather than on improving the quality of care).
On the other hand, there was less evidence of systemic impact, and some concern that the organisational focus of CQC’s regulatory model was not well suited to an increasingly integrated health and social care system or to provider organisations that work in partnerships, chains or networks.
We also found significant differences in how impact is achieved across the four sectors that we studied. For example, a provider’s improvement capability and the availability of external support for improvement were key determinants of impact, and these were more often present in the acute and mental health sectors than in general practice and social care.
One of the most striking findings from our work was that the relationships between CQC staff and health and social care professionals and managers fundamentally affect the way regulation works and what impact it has, and contributes to the variation in providers’ experiences of inspection.
Overall, providers accepted and generally supported the need for quality regulation within the health and care system and saw the approach introduced by the CQC in 2013 as a significant improvement on the system it replaced.
What does performance data tell us about impact?
We also tried to measure the impact of regulation quantitatively.
First, we examined whether provider performance changed following an inspection by analysing routine data about accident and emergency services, maternity services and general practice prescribing. We found that inspection and rating had small and mixed effects on key performance indicators in these areas. This may suggest CQC had a limited impact in these areas, or it may be that the effects of regulation are difficult to measure with routine data sources. The impact of the CQC is also difficult to isolate from other factors affecting provider performance.
We also explored whether CQC ratings affect where patients seek treatment, by looking at the impact of inspection and rating on service volumes in maternity services. We found little evidence of parents (or their agents) exercising choice in response to ratings – receipt of an ‘inadequate’ rating seems to have little measurable impact on subsequent service volumes.
Finally, we analysed the Intelligent Monitoring (IM) dataset – a large set of routine performance indicators that CQC used to risk assess organisations and to help them decide when to inspect a provider and what to focus on. While the datasets were not intended to predict inspection ratings, we might expect an association between the two. We found that the IM datasets had little or no correlation with the subsequent ratings of general practices or of acute trusts. This highlights the limitations of risk-based regulatory models that use routinely reported performance data to target regulatory interventions.
What are the implications of our findings?
Our qualitative research shows that there are a range of ways that regulators have an impact on providers beyond directing them to make specific changes. These happen before, during and after inspection.
This may be part of the reason why our qualitative findings contrast with the findings from our quantitative research, which attempted to compare performance before and after inspection and found limited evidence of impact.
Inspection and rating have dominated the CQC’s regulatory model, consumed most of its available regulatory resources and may have crowded out other potential regulatory activities that might have more impact.
Furthermore, the inspection model we studied was focused on individual providers. However, as health and social care provision becomes more integrated, this focus will become less tenable and place- or service-based regulatory approaches that cross organisational and sectoral boundaries will become increasingly important.
Now that CQC has completed its first full cycle of inspecting and rating health and social care providers, it is implementing a new strategy for regulation that addresses some of the issues raised in our research. We welcome the greater emphasis that this strategy places on relationship management and the development of systemwide approaches to monitoring quality. We also welcome CQC’s efforts to develop a more insightful system for prioritisation to replace the IM model discussed above.
As CQC works to implement its revised strategy with stakeholders across the health and care system, our research emphasises the need to take a prospective and deliberative approach to designing, piloting and testing regulatory interventions in order to measure their impact in practice. In particular, it emphasises the need for:
- develop and use regulatory interactions other than comprehensive inspection that draw on its intelligence and insight to support providers, foster improvement and prioritise its use of resources
- develop its model in different ways in each sector, depending on factors such as the size and number of organisations being regulated, their capacity to respond to its recommendations, and the other resources available to support improvement
- invest in recruitment and training to create an inspection workforce with the credibility and skills necessary to foster improvement through close relationships, while maintaining consistency and objectivity
- not underestimate the difficulty in developing a more insightful system of prioritisation that draws together both hard and soft intelligence from a wide range of sources, takes into consideration providers’ own ability to accurately and honestly self-evaluate and engages patients, users, providers and commissioners in its development
- consider the range of ways that they can work with CQC staff and processes/tools to improve services before, during and after inspection
- as part of this, to recognise that the impact of CQC’s regulatory processes is co-produced by CQC, the provider and other stakeholders and that they have a responsibility to engage collaboratively in order to maximise impact and improvement
- encourage and support their staff to engage in open, improvement-focused discussions with CQC inspection teams and other stakeholders
CQC, other regulators and commissioners to:
- continue to develop place- or service-based regulatory approaches that cross organisational and sectoral boundaries; to do this, important work which is already under way to align the activities of regulators, commissioners and other improvement-focused organisations must gain pace and depth.
All very interesting, however my campaigning for Supported Accommodation to be regulated continues to fall on 'deaf' ears. The FACTS: Those with Learning Disabilities and 'severe and enduring Mental Illness are placed in into this type of accommodation for many reasons: it's cost effective, often no Statutory Care Plan, 'eligibility' ignored, no Support Worker, and CHARGED for Services they don't receive via Housing Benefit forms, (completed by someone else)? if you have been awarded 'enhanced care' via PIP, you can still be left with no SUPPORT. The CQC have ONE way of 'stamping' out this 'illegal' practice, request to view the LIST of those subject to Section 117, and 'Care Plans' for those PLACED into Supported Accommodation by LAs.
This REPORT based around the CQC, has NOT understood their are thousands in the COMMUNITY left unprotected: (because they don't fit into the box).
The NAO are investigating my complaint, regarding their Auditing of the DWP HB/Subsidy Department, who are responsible for the 'set of instructions' for their 'external auditor' to follow when 'checking' Housing Benefit application, that would identify non 'EXEMPT' CRITERIA claims re HB/Subsidy repayments to a LA, and 'illegal' charges made on the CLIENT via Housing Benefit Forms, and do NOT be surprised the LA have stated 'it is possible' we have NOT identified those subject to Section 117 FREE Aftercare on these forms. This now continues to those subject to a COP Order for 'Property and Finance' who is responsible for his client finances, (who we know is just a figure head).
The above was identified by me two years ago, nobody want to investigate, the PHSO are responsible, even my MP's referral was ignored, so the NAO.
This 'vulnerable' group are completely 'exploited' by the system, even 'Service Charges' applied, but no INVENTORY taken when they took up their tenancy, and they pay out of their Welfare Benefits.
The COST to the PUBLIC PURSE is enormous, the Cost to the TENANT is enormous. The LA are the winners, along with the LANDLORD usually a Housing Association.
The TRUSTS are nowhere to be seen, the 'severely mentally ill' have now become 'invisible' to the GP, the LA and the TRUST. The only service they reserve is a Prescription, and even that is NOT monitored by anybody.
'integration' it is NOT, legal it is NOT, caring it is NOT.
If you only listened to an expert-by-experience you would SAVE yourself millions in DATA Collection from the wrong places.
Thank you so much for sharing the insightful results of this data - the need for quality regulation is vital in all nonprofit sectors and it is encouraging to see this being practised and examined within the health and care system - there's a lot to learn from these studies for nonprofit fundraisers as well.