Marie Polley: The challenges of evaluating common sense

This content relates to the following topics:

Dr Marie Polley, Co-Chair, Social Prescribing Network and Senior Lecturer in Health Sciences and Research, University of Westminster, discusses progress made in developing better ways of researching and evaluating social prescribing.

This presentation was recorded at our conference, Social prescribing: from rhetoric to reality, on 18 May 2017.

Transcript

I feel like a proud parent today. Who knew when Michael and I had an idea to set a network up that fifteen months later there would be 400 people in two rooms taking over the Kings Fund and Twitter trending nationally?  So big up to you all #socialprescribing.  And hello to the room upstairs.

So I wanted to unpack a little bit about where we’re at with evaluation and evidence, because I think it’s the next big challenge in social prescribing. We’ve got everyone talking about it, we’ve got 75% of the CCGs involved at some level.  And now we really need to get serious about how we provide the right data to persuade the right people with influence to embed this.  So I’ve called this the challenges of evaluating common sense, I dedicate this to the team that have just worked on the toolkit and the evidence review which is also in Purdah at the moment and will be out at the end of June.

So a lot of people have said over the last fifteen months ‘Well it’s common sense’ we’ve even heard it today from Helen’s dad, that sounds like a great idea. You would not believe how hard it is to do evaluation on common sense, because it materialises itself in all sorts of ways.  And can I just say, if you’re not in the social prescribing network, then the e-mail is there and we have regional networks across the country now, so just haven’t said anything about the network and that is all I will say.  And Janet couldn’t have set my next slide up better if she had tried.  I found this quote and I just think this is where we’re at in a way.  Making the simple complicated is commonplace, that describes our Health Service, our policymakers, and everything that we’re trying to deal with, and making the complicated simple, that is creativity.  If Sam said that, he would say that is breaking the rules.  I think that’s what social prescribing is actually doing for a person is making a very complicated system a simple process.  It’s simple to say what matters to you and how can we help you and here’s the people that are going to do that, but we’ve gone about it in a very complicated way in the last ten years and more, so I think it’s time to bring that back.

The creativity bit is the innovators and the people that have thought incredibly creatively to get out of this fixed system. And if you talk to Alison McGregor, she talks about the liminal space and actually the free thinkers and the fixed thinkers.  And there’s a lot of people in this room that have worked really hard to come up with creative solutions to making healthcare better for a person.  But I think we may have got to a really interesting position, because there’s a lot of different expectations and these expectations come from all the different professional sectors, they come from the policymakers, they come from the people on the ground.  And having just worked with a team to put an evidence review together and national guidance, there were times we felt like we were in between a rock and a hard place, and I just wonder if commissioners feel like this at times.  Because on one hand we’ve got system wide transformation, it’s going at break neck speed and it’s happening, and you can’t stop it, so we’ve got to go with it.  But for me that creates quite a tension because there is not the depth of evidence and knowledge about a social prescribing issue we would normally have for something that is breaking the rules around influencing policy.

So evidence is a word that means different things to different people, and I think this also sets up a rub in conversations. So if you’ve been trained in a medical/clinical paradigm there are multiple clinical trials that have been systematically reviewed to answer a specific question, that’s your evidence.  If you’re talking to third sector, evidence might be some of the data that they’re putting together from their records.  So it’s really important that when we talk about evidence, we’re really clear on what we’re meaning, so that we can have a shared understanding.  So going forward we’ve now got to work together, because it’s us here and it’s everybody that we know that are going to work together to start to pool our information, our data, to really underpin the evidence for social prescribing.

The first thing I want to do is make you aware of this guidance. So the idea of evaluating complex health interventions, which you would call social prescribing, is not new.  And the MRC put out guidance in 2006 which is actually going to be updated again.  And this guidance, there is lots of reasons why they did it, but one of them was to help people understand how to tackle these highly complex non health sector interventions.  Another one is to make people aware that it is okay to have attention to the social, political, geographical context that interventions take place in.  Now this guidance it’s a meaty bit of reading and it’s designed for the academics that are designing the evaluations to pick the right methods, and approaches according to the situation.  If you’re thinking of doing evaluation, it’s downloadable, just Google the title and it comes up.  Have a look at it, because this is the expectation that any funder will have if the NIHR, the MRC, Wellcome, anybody is going to fund, they have expectations that we’re following the rules set out here.  So as academics, this is what we would be leaning to.

So that’s the first resource. And I think it’s a really helpful resource and it helps you find the path of where do we start, how do we bring in the case studies, what’s the importance of having all the interview data, how do we design a really good clinical trial or controlled trial?  So if we just unpack this a little bit more, there’s two things that are happening concurrently in social prescribing.  If you’re in Janet’s position, you’ve already developed a social prescribing scheme, you’re way on, you’ve got loads of feasibility data, you’ve got everything you need to actually have a really large scale solid controlled trial or big scale evaluation.  But a lot of people as we know 22 out of the 44 STPs are actually starting to develop social prescribing.  So actually developing is different to next stage research.  And developing an intervention requires you to understand what it is that’s going on in the day to day reality of practice.  So it’s great, we all know there’s clinical trial data for physical activity being good for you, but how you get people that are obese, depressed, low self-esteem and confidence, maybe don’t want to go out the house, socially isolated, how you get those people to be more physically active, well that’s the day to day reality of introducing something that might have clinical trial evidence, but not in real life.

So you need to also know how the intervention is working. So that would be what are the active ingredients?  When you know the active ingredients, you can adapt your intervention in different localities but keep the active ingredients.  If you look at the social prescribing network, we put a report out in March last year, and actually in there are the active ingredients of social prescribing.  I get asked a lot ‘What’s the model, tell me how to do it?’  And it’s like ‘No, no, you’ve got to work that out, but I’ll tell you what you need to have in the model’.  And you’re going to talk to everybody in your area, and you’re going to map what exists and you’re going to work together, and you’re going to realise those key ingredients.  So understanding those key ingredients is absolutely essential.

So in this developing stage of evaluation, you would want to think well what’s working in a local context? How do we measure outcomes in practice?  So you either get an external agency or academics to come in and do a discrete evaluation, or you say ‘Well we’re going to collect the data on the ground’.  Well you need training, people need to be trained, they need to know what to measure, how to measure.  You need to know how to put it into a database, how to analyse it and how to report it, so that is not a simple step.  But we’re moving to an outcomes based system so that needs thinking about.  What outcomes have you identified that you hadn’t thought about at first, because social prescribing has loads of unintended and wonderful impacts?  And is the intervention acceptable?  Well social prescribing when it works usually is incredibly acceptable to everybody, but you need to understand that.  What else do you need to adapt going forwards?  And then you get your preliminary data which hopefully says yes now go off and do a bigger evaluation.

If we just look at capturing outcomes, and outcomes measurement is actually the one thing is the thread through the whole of my career that I’ve worked on. I think again the rapid increase in the amount of social prescribing that is occurring, it being named in the high impact actions, has meant that everybody wants everything at the same time now.  And you’re just like my brain is going to explode, because it just isn’t that simple.  And there’s a lot of mixing.  And I’m going to borrow from a conversation that Dan Hopewell and I were having as we were mulling over something.  If you wanted to know for somebody that was depressed, if you gave them drugs, and you wanted to know if the drugs were efficacious, would you see if there was a demand in reduction for Health Service use?  No, you would measure their level of depression.  And just like Janet was saying, you don’t refer someone to a pharmacist and say ‘Give the drugs for free’ when we’re talking about the third sector.

So first of all you need to know as any separate question, is the social prescribing scheme doing what you wanted it to do, is it meeting its aim? Because if it’s not even doing that, don’t measure anything else, don’t waste your time, work out why it’s not doing what it’s meant to be doing, that’s the first thing, and that’s the primary thing.  What’s the wider impact is a separate question.  You can still collect the data at the same time, but you need to know if it’s working first of all.  You might look at health and wellbeing as they do in Rotherham, you might look at housing, work, welfare, there’s all sorts of things you can measure, but you have to link that to the aim of your scheme.  Well then you look at the wider impact and this is why looking at the whole process is very important for social prescribing.  The economic impact we’re hearing is good and we’ve got an evidence summary in this area coming out.  It could be NHS services, it could be work status, prescribing of drugs, volunteering, someone could be more confident, more active, there’s all sorts of things.  Generally there’s a good impact on the carers and the family and very importantly on the healthcare professionals, so it brings more joy into their lives.  So that will keep Helen happy that talked about that this morning.

The third sector. Well the wider impact is there’s a massive increase in demand, but that’s not a demand that’s underpinned fully by the same level of funding, and I’m always very open about the fact that we need more funding into the third sector.  And there might be unintended impacts which are usually positive but you need to understand what they are.  So if we did a really crude overview from my position of sort of being in touch with lots of people around the country and internationally, we know in England 75% of the STPs are involved, these are different degrees.  But what is really interesting is there’s a much better quality of conversation happening between the CCG, the local authority, public health and the third sector.  But to actually get CCG local authority, public health, break down those barriers, because actually they’re all involved in helping a person, so why don’t they talk together?  And often there’s a lot of advocacy that we do to bring people together to actually find out what each other knows first before even setting up a scheme.  But that’s good because hopefully the funding will start to be shared and it will allow schemes to develop.

I think there is quite compelling feasibility data now. I wouldn’t say we’ve got the compelling clinical trial data, but I would say feasibility wise, there’s a huge amount of qualitative data, case reports, interviews, you only needed to listen to Debs this morning, that’s amazing, you know?  I remember the day my life changed.  There’s loads of data like that in lots of organisations.  I think, if we did a qualitative review, we’d identify all the outcomes that could be measured.  And we did some work, and again in the social prescribing report from last year, we detailed categories of outcomes.  It’s the general health, wellbeing, physical activity, physiological stuff, you’ve got social isolation, mental health, wellbeing, confidence, self-esteem areas.  You’ve got actually for the patient they gain control and activation, self-esteem, and very, very importantly a sense of purpose, life is rubbish if you haven’t got a sense of purpose.  But then there’s housing welfare advice, employment, volunteering, there’s loads of outcomes.  Economically £2.3 per pound in the first year, hearing more from other projects, £6, £8, £11, that’s a great return across the whole social prescribing schemes.  But that’s only data that’s collected on a certain group of people and I’m going to come to that in a sec.

We’re seeing on average a reduction in 28% for GP attendance, so we’ve been doing some internal analysis, a 24% fall in A&E attendance. And that’s in areas where social prescribing schemes have been introduced, and often that’s about six months later, a reduction in emergency admissions or unplanned admissions to hospital between 6 and 33%, and overall reductions in referral to secondary care.  But I think it’s really important, if you actually empower a person to get the care that they’ve really not had, there will be a proportion of people where the referral rate goes up, because they’re getting the support they should have been having all of their life.  On balance, it goes down.  This is where we need to work together to improve I would say the state of play.  What is commonly unreported is actually for instance how many people did not engage in a service, how many people dropped out, why, when?  So when we come to review social prescribing reports and papers, it’s really important for us to know actually who is the data based on?  Because we know that when a person engages fully in a social prescribing scheme, and you get the before measurement and the after measurement, it works, and the results are spectacular.  But that’s not everybody.

So for every scheme that there is, there will be people that didn’t engage or dropped out. Sometimes they drop out because life happens, they move, they get a job, blah blah blah, they’ve got better, they’re too busy, but that’s not always.  Because it might mean that the model needs adapting.  We don’t know if we’re fully meeting the needs of the target population, we know we’re meeting the needs of the people that use the services, but actually we want to get the whole target population that need the service.  And if one thing would help the whole field of data going forwards is for people to report that type of data as well.  Generalisability and researchers, can you take data from one study and say that that would reflect another population that used it, or who does it actually work for?  We can’t say that these services are generalisable, we can say they work in the area they work.  So we need to sort this aspect out, because we can’t design good controlled trials going forwards which is the next thing we really need to do, unless we have the full picture.  And it’s not about saying it’s not working, it’s about saying it is really working for these people, who isn’t it working as well for and can we make it better?  Ultimately a balanced picture is what will help a funder to commit their money.

If we put the picture out at the moment this is going to say well that is quite biased, but what about this and what about that? So to help us go forward with this, I know I’ve seen Tim Joss from Aesop here today, so I wanted to say there’s a really good evaluation framework that is done by Aesop and Public Health England, this gives you a feel, particularly if you’re in the third sector, what do you need to think about when you’re approaching evaluation and how do you report it?  So I think a lot of people have the information, it just doesn’t always get reported.  So this again, you can Google this, you can download it and have a read.  So the next step.  So people that have heard me talk over the last fifteen months, I always talk about shuffling forwards together.  Because none of us can go forward on our own now, we’re in it together, we’re all tied in, and we can’t do anything without each other, and that means it takes a little bit longer.  Because if you’ve tried to get twelve people’s diaries together for a meeting you will know that you need about three months just to do that to make a decision.

So we have a toolkit for implementing social prescribing’s national guidance, and led the project and we’ve co-produced it, all the major social prescribing projects have been involved, lots of people involved, and an evidence summary. They’re out after purdah is lifted and they will be a great way for setting the scene for people that are in that middle part of the spectrum of thinking how to go forwards.  I think we really need to capture the national picture now and we need some read codes on the system, we need to know nationally how many people are engaging in social prescribing.  We do need a common outcomes framework, I’m not sure how easy that is going to be, but we need I think to agree a few outcomes that everybody could measure so that we can again build a very strong picture.  And I think we need to work out how to value economically all aspects of social prescribing, very good at working out the health stuff, but actually really agree the value to be put on the different parts of the social prescribing schemes.

I think there’s another discussion to have on what is the right way to capture controlled trial data? It’s very hard to do a controlled trial when the intervention you’re researching is actually concurrently being rolled out nationally, that is a real sticking point for research methodology.  And we need to secure funding for everybody, Network needs some, third sector needs some, everybody needs some.  So I just want to say thank you to the people up here who have been involved in information on the slides and I look forward to having a glass of wine with you later.  Thank you.

Comments

Add your comment