Huw Prosser Evans, Clinical Informatics lead for the Primary Care Patient Safety Research (PISA) Group, speaks at the Digital Health and Care Congress 2015 at The King's Fund. Huw talks about the 'PISA' method of discovering patterns in incident reports which can help to identify if things go wrong and why they have done so. He discusses the technical challenges faced and the future for the method.
More from the 2015 Digital Health and Care Congress
Every time a mistake is made in the NHS an Incident Report should be filled out, and every month 100,000 Incident Reports are completed. Some of these are things such as letters going to the wrong department, leading to a delay in out-patient appointment. Others are about patients having medications that they are allergic to, with obvious severe consequences.
This is complex, confidential, clinical data. Data that needs interpretation, almost like a chest x-ray needs to be interpreted by a doctor, and we need to find better solutions.
People say to me ‘you don’t work with big data, big data is people like Facebook or Twitter’, but the definition of ‘big data’ is not just large data sets, it’s data that’s large but has also outstripped conventional means of analysis, and I definitely think we’re at that point now in the NHS.
But I want to move away from talking about big data to talk about little data, because I’m sure you’d all agree here that if harm was to come to one baby that would be one baby too many, but in our analysis of 270,000 Incident Reports we found 99 cases of babies receiving a TB vaccination when it was not safe for them to do so, and this could have had potentially severe consequences. We are looking for stories in the data, patterns, signals. How do we find those 99 babies? We have ample data, although we can always do with more.
We need to apply clinical knowledge, clinical interpretation in order to provide us with the information. These are free-text responses written by clinicians on the ground, they often need interpretations to the situation so that they can be better understood.
So in order to do this our research group developed something called the PISA Method. The Incident Reports are read by clinicians who then classify exactly what has happened, such as the wrong vaccine, the wrong dose of medication. They look at what’s contributed to that, such as locum staff or being out of hours, and finally the outcome that happened to that patient. We find new concepts and reiterate our framework and this works quite well. It really is granular enough that we can learn from it, but generalisable enough that we can peck out the patterns in a big data set.
But with it comes numerous technical challenges. The coding framework iterates almost daily. This was also an international collaboration. We had researchers from across the world working on this and keeping my colleagues in Australia up to date when I’m asleep in bed is quite tricky. What we need to do is think about how we can move this forward.
We’ve got over 50,000 patient safety incidents that have been coded by a trained clinician now, of which at least 20 per cent have been double coded. We’ve also got our accurate framework which changes very rarely now and we’ve started to develop extra appendices to these for things such as dentistry and other primary care specialties, and what we need to do now is start thinking about automating this process.
I don’t ever see a world where a computer will be able to completely interpret an Incident Report. I said before, there’s too much interpretation, much like I don’t ever see a computer reading a chest x-ray. But what we can do is create algorithms to peck out the important cases, to look for those cases that we need to focus on that I need to pass on to my clinician colleagues sooner, to find those 99 babies and to find that needle in the haystack that eludes us.
Thank you very much for listening to me this morning. It’s been an absolute pleasure.
Voluntary reporting of incidents is always a challenge because there are so many variables affecting reporting and the quality of analysis. Who has the time to analyze 100,000 incidents per month? More important in my view is to perform prospective focused studies, opportunities for which may be identified by data mining of large numbers of incidents. The learning comes from focused studies. Willing to discuss at your discretion.