Charlie Harp is CEO of Clinical Architecture of Carmel, IN.
Tell me about yourself and the company.
I’m a developer by training. I’ve been building systems in healthcare for about 35 years. Back in 2007, after working for a bunch of different companies, I started Clinical Architecture to focus on the plumbing of healthcare, such as semantic normalization, data quality, and gaining insights by looking at patient data.
The industry has more technical pipes available to exchange data, but have we equally advanced terminology and semantic issues?
In the last few years, people have become a little bit more sophisticated in how they share data. USCDI has driven some folks, through the Cures Act, to at least try to share more data. The guidelines we have are still a little fuzzy in terms of being more guidelines than rules. We have made some progress, but we are still dealing with people that might have access to the data through something like data exchange. I think TEFCA is going to continue this, but I still think there’s a lot of hesitancy to accept that data when you get it.
The last time we talked, you said that providers don’t trust each other’s data, and that one provider doesn’t have much incentive to clean up their own data that they have already used for someone else’s benefit. Has that situation improved?
A little bit. We started doing a data quality survey last February. People generally did not think very much of their own data quality. Most of them — depending upon the domain, whether it’s drugs or labs — had some level of confidence, but they didn’t have high confidence in the quality of that data. The only thing they had less confidence in was the quality of other people’s data, which I thought was interesting.
The problem we have in healthcare today is that we gather information as a byproduct of the process of providing care. Providers rely heavily on their notes to go from one patient encounter to another. They fill in the structured data because they have to.
We have this illusion on the analytics side of healthcare that the structured data is of high quality. When we go to share the data, most of these systems — whether it’s Epic, Cerner, Meditech, or whatever — are still using dictionaries that were developed for that EMR, with code systems that are specific to the EMR. They still have to be normalized on the outbound.
The challenge with people sharing data out, especially if it’s a regulatory requirement, is that it’s a “letter of the law” as opposed to “the spirit of the law” type of engagement. The data is leaving, and people tend not to care as much about the data that’s leaving as they do about the data that’s coming in. The problem with the data coming in is that, to the people who sent it, it was data leaving, so it wasn’t as important to them.
Do those clinicians who don’t trust their own data at least have confidence in the subset that they need to treat patients, or do they create their own notes or records?
It’s a combination of the time famine that providers have. They don’t have a lot of time. A handful are aware and plugged into what’s happening with health informatics and interoperability, but a lot of them in the trenches are just focused on how to provide the best care while complying with the things they are being asked to do by their organization. A lot of them, at least the ones that I talk to, tend to still rely heavily on their unstructured notes to go from encounter to encounter.
A few years ago, we looked at the structured data and did inferencing to find patients who were undocumented diabetics, patients who had no mention of it in the structured medical record. We looked for other indicators, like the fact that they had a hemoglobin A1C that was out of whack, or they were taking something like insulin or metformin. In six months, we found 3,600 undocumented diabetics. When the folks we were working with presented that finding to providers, the feedback was pretty universal — I know those patients are diabetic, that’s why I gave them metformin.
The problem is that there’s a disconnect between the provider, who is legitimately just trying to take care of people, and the unintended consequence of not having the structured data in the system. That means that your quality measures are out of whack, your patient management software is not scheduling foot exams. There’s still a disconnect between why you put in the structured data in the first place and all the downstream systems that consume that. Analytics, machine learning, and AI, all these things that we want to embrace and leverage in healthcare, depend on the structured data being there and being correct. We are pretty far off from that, unfortunately.
Does AI offer opportunities to structure that data using free text notes or audio recordings of encounters?
We have done a lot with NLP and also evaluated what’s going on with large language models. The problem in healthcare is that when it comes to data, it always falls back to trust.
If I could wave my magic wand and fix healthcare, I think what I would change is the way that we collect the data, so that we are collecting structured data without turning the provider into a terminologist to make that work. The problem we have is that providers don’t want to go to a list and pick something. They want to be able to articulate something in a way that is natural and organic for them, and then get it back in a way that’s natural or organic. We’ve had two worlds, one where you create a note and the other where you put things into a list.
I think the real answer is finding a way where the provider gets what they want. They say something in a way that’s granular and organic. We capture in a way that preserves the granularity of that information in high resolution, and can leverage that from an analytics perspective. When the provider wants to see the data, we can deliver it in a way that’s organic to them instead of them looking at a list and reconciling things. We’re a little bit off from that.
The problem with what we are doing now is that we are trying to find an easy way out. We’re saying, let’s just take the note and use NLP, a large language model, or something else to read the note and turn the note into something structured. You can do that, and we have had some success when it comes to high-certainty type data like pulling ejection fraction out of a procedure note or looking across a complete patient record and coming back with a sense of the patient’s diabetes because I found all these references that I can correlate to that. But you still run into the problem of, how can I trust that?
When you look at all the things that are happening in the industry now with AI, large language models, and NLP, there’s a lot of talk about transparency. In the past, when people have tried to do things in healthcare with these types of approaches using NLP or AI, it hasn’t been successful. The machine works great 60% of the time, and then 40% of the time it does something horrifically wrong. That comes back to this idea of trust. If you are taking care of somebody and their life is in your hands and the machine just happens to pick that day to follow the wrong probabilistic pathway, that’s challenging in healthcare.
Thinking back to providers not trusting their own data, is that a vague impression or is it measurable? What could they measure to assess or improve data quality?
When I’m working with clients, I sometimes ask them this question, so I’ll ask you. When it comes to healthcare data for an individual patient, who is responsible for the quality, accuracy, and integrity of that one patient’s data, regardless of where it is?
Some would say the patient, although that’s not a reasonable expectation for all types of data.
The problem is that patients aren’t really trusted to do that. You can fill out a form, hand it to somebody, and they’ll type it in, but rarely is a patient trusted to own and articulate, “Here’s my health situation.” It usually has to be vetted by some kind of clinical person.
That’s fine, but it goes back to this root problem that nobody is responsible. There is no data steward for an individual patient’s health record. When you talk about how you trust the data, the fact that I can take one patient and look across multiple venues of care and see different information. They don’t really trust each other and where their data is coming from. They don’t know whether that ICD-10 code was added for billing purposes or added for clinical purposes.
The problem we have in healthcare is that we don’t have a mechanism that allows us to objectively and quantitatively look at the data and say that the quality is good or bad. We are working with other organizations to do this taxonomy for healthcare data quality, because I think that we should be able to look at patient data in an abstract way and say, is the quality of this data good? Is there duplicate stuff? Is there old stuff? Is there stuff that’s clinically impossible? Are there things in the medical record that contradict themselves?
How can we automate the evaluation of that semantic interoperability so that you don’t need a sweatshop full of clinical people looking at 5 million patient records? How do you build something that can objectively, with some type of deterministic AI, evaluate an individual patient and any data that comes in for them to say, yes, this all makes sense. It looks real, and I just noticed that there’s no mention of this patient’s diabetes, whether you’re looking at unstructured notes and pulling it out.
At the very least, you should pull the data, check it against the integrity of the rest of the medical record, and say, yes, the fact that the note says they are diabetic resonates with the fact that they’ve got a funky fasting blood sugar and they’re taking these three medications that are indicated for diabetes. Let’s go ahead and suggest that they add diabetes to their official structured medical record so that we can take advantage of that. All these things that only look at the structured medical record and retain the evidence of where that came from. Those are some things that we could do to improve the level of trust and the reliability of the data.
My big fear is that we start to roll out some of these more sophisticated things that could be beneficial, but because the data quality is bad, we fumble the results early on and these things fail, and because we applied them before the data quality was ready, people don’t have confidence. You only have one opportunity to be credible. You come in with this new technology and say, “This is going to save lives. This is going to do great things.” But because the data that we are feeding it is bad, it is very possible and probable that the results of what it comes up with will be likewise bad. We will flip the bozo bit, as they used to say, on that thing. Then later, when we fix the data quality, we say, “No, we tried that and it didn’t work.” But maybe it would work if we fed good quality data.
What is the oversight structure and mechanism of reviewing the longitudinal patient record from multiple providers and identifying missing or conflicting data? Then, going back to the data source and either asking them to fix their problem or perhaps excluding their data as being unreliable?
The first place is the pipes. Look at what’s happening with TEFCA and QHINs. Let’s say the QHINs turn on their pipes and people start streaming data from Point A to Point B for every patient. The first thing we need to do is, somewhere in that pipe, we need to have something that looks at the message. Is the message right? Does the data look fundamentally correct? Not clinically correct, but is a valid code in the value set? Let’s say it’s an RXNorm code. Does the name match what RxNorm says that code stands for? So the first thing you do is evaluate someone in the network to determine whether they are a good data provider.
If they’re not a good data provider, you can’t really remediate data quality in flight. You have to go back to the source and say, you’re not a good data provider. This is what our taxonomy is focused on. By identifying the nature of the quality failure, you can go back and say, you’re putting the decimal place in the wrong spot on your lab results. You are not using a valid RXNorm code set. Your maps are bad. Whatever the feedback is.
The first thing we need to do is to make sure that the people that are sharing data from their systems are good members of society who care about the data they are sending out and are making sure that the quality is good. QHINs are going to be in a great position to evaluate the data in flight at a basic level and say, OK, the data that you are sending looks clean, looks good, and has good intrinsic quality. That’s the first step, because that’s where you stop bad data from getting out.
We also need to do a better job of knowing where data’s coming from originally so that we can stop duplication. We worked with a partner who gave me a bunch of data to evaluate, data that was coming from a bunch of different sources. In a couple of million records, there were about 750,000 duplicates, the exact same lab result done at the exact same time. Because of the way the data was shared in some of these older formats, you had no idea that that was the same data. It just looked like this patient had 64 lab results on the same day at the same time.
That’s the other thing, if we want to trust data, we need to know where it originally came from, especially as we start sharing data across an entire network of participants.
The last thing is you need is a way where we are landing it or looking at the data in our own system, saying, does it look right for every condition that I have? Do I have a treatment for every drug that’s in their profile? Do I know why they are taking that drug? This goes back to what you are talking about when it comes to oversight. Within any repository of patient data, perhaps a large IDN doing analytics or population health on your patients, we need to have mechanisms that can identify issues in the patient. Data can alert a human operator. Let’s call them a data steward. The data steward can inform the systems that they are connected with on how to remediate the data.
There needs to be oversight. The trick is, how do we have enough automation in place so that instead of a human looking at 5 million patients, automation is looking at 5 million patients for things that are a concern, and streamlining the resolution of those things? Because it’s easy for a human to be presented with something and say, “Yeah, that looks right,” as opposed to humans poring over data looking for something. That’s why when we do semantic normalization, our software does like 85% of the work, where it tries to search for the right target and it suggests the target. A human can take two seconds to look at the target and say, “That’s right.” We need to get to the same place when it comes to patient data.
It’s one of those things where the idea of having people whose job it is to review issues that come up with patient data and resolve it at a patient level might seem a little daunting, but the problem is, that’s the only way we can fix it. You have to fix it at the atomic level to have the entire ecosystem be of high quality. There’s no way to do it at a macro level. You have to do it at an individual patient level.
What factors will be important to the company and the industry in the next few years?
For us to use artificial intelligence and some of these other things that we are coming up with in a meaningful way, we are going to have to move away from pre-coordinated terminologies as how we collect data for patients. We’re going to store patient information in a much more granular graph style, so that both software and people can make better use of it. Right now, everything we do with the terminologies and practices that we use today create these big pixels of information that limit our ability to do sophisticated reasoning over that data, whether it’s for research purposes or for decision support purposes. We’re going to have to dial up the resolution to get to where we want to be in terms of software providing meaningful assistance to people that are providing care.
FYI, that whole wedgie thing is actually brutal to read about, and it led to some extremely serious injuries for…