Going to ask again about HealWell - they are on an acquisition tear and seem to be very AI-focused. Has…
HIStalk Interviews Charlie Harp, CEO, Clinical Architecture
Charlie Harp is CEO of Clinical Architecture of Carmel, IN.
Tell me about yourself and the company.
I’m a developer by training. I’ve been building systems in healthcare for about 35 years. Back in 2007, after working for a bunch of different companies, I started Clinical Architecture to focus on the plumbing of healthcare, such as semantic normalization, data quality, and gaining insights by looking at patient data.
The industry has more technical pipes available to exchange data, but have we equally advanced terminology and semantic issues?
In the last few years, people have become a little bit more sophisticated in how they share data. USCDI has driven some folks, through the Cures Act, to at least try to share more data. The guidelines we have are still a little fuzzy in terms of being more guidelines than rules. We have made some progress, but we are still dealing with people that might have access to the data through something like data exchange. I think TEFCA is going to continue this, but I still think there’s a lot of hesitancy to accept that data when you get it.
The last time we talked, you said that providers don’t trust each other’s data, and that one provider doesn’t have much incentive to clean up their own data that they have already used for someone else’s benefit. Has that situation improved?
A little bit. We started doing a data quality survey last February. People generally did not think very much of their own data quality. Most of them — depending upon the domain, whether it’s drugs or labs — had some level of confidence, but they didn’t have high confidence in the quality of that data. The only thing they had less confidence in was the quality of other people’s data, which I thought was interesting.
The problem we have in healthcare today is that we gather information as a byproduct of the process of providing care. Providers rely heavily on their notes to go from one patient encounter to another. They fill in the structured data because they have to.
We have this illusion on the analytics side of healthcare that the structured data is of high quality. When we go to share the data, most of these systems — whether it’s Epic, Cerner, Meditech, or whatever — are still using dictionaries that were developed for that EMR, with code systems that are specific to the EMR. They still have to be normalized on the outbound.
The challenge with people sharing data out, especially if it’s a regulatory requirement, is that it’s a “letter of the law” as opposed to “the spirit of the law” type of engagement. The data is leaving, and people tend not to care as much about the data that’s leaving as they do about the data that’s coming in. The problem with the data coming in is that, to the people who sent it, it was data leaving, so it wasn’t as important to them.
Do those clinicians who don’t trust their own data at least have confidence in the subset that they need to treat patients, or do they create their own notes or records?
It’s a combination of the time famine that providers have. They don’t have a lot of time. A handful are aware and plugged into what’s happening with health informatics and interoperability, but a lot of them in the trenches are just focused on how to provide the best care while complying with the things they are being asked to do by their organization. A lot of them, at least the ones that I talk to, tend to still rely heavily on their unstructured notes to go from encounter to encounter.
A few years ago, we looked at the structured data and did inferencing to find patients who were undocumented diabetics, patients who had no mention of it in the structured medical record. We looked for other indicators, like the fact that they had a hemoglobin A1C that was out of whack, or they were taking something like insulin or metformin. In six months, we found 3,600 undocumented diabetics. When the folks we were working with presented that finding to providers, the feedback was pretty universal — I know those patients are diabetic, that’s why I gave them metformin.
The problem is that there’s a disconnect between the provider, who is legitimately just trying to take care of people, and the unintended consequence of not having the structured data in the system. That means that your quality measures are out of whack, your patient management software is not scheduling foot exams. There’s still a disconnect between why you put in the structured data in the first place and all the downstream systems that consume that. Analytics, machine learning, and AI, all these things that we want to embrace and leverage in healthcare, depend on the structured data being there and being correct. We are pretty far off from that, unfortunately.
Does AI offer opportunities to structure that data using free text notes or audio recordings of encounters?
We have done a lot with NLP and also evaluated what’s going on with large language models. The problem in healthcare is that when it comes to data, it always falls back to trust.
If I could wave my magic wand and fix healthcare, I think what I would change is the way that we collect the data, so that we are collecting structured data without turning the provider into a terminologist to make that work. The problem we have is that providers don’t want to go to a list and pick something. They want to be able to articulate something in a way that is natural and organic for them, and then get it back in a way that’s natural or organic. We’ve had two worlds, one where you create a note and the other where you put things into a list.
I think the real answer is finding a way where the provider gets what they want. They say something in a way that’s granular and organic. We capture in a way that preserves the granularity of that information in high resolution, and can leverage that from an analytics perspective. When the provider wants to see the data, we can deliver it in a way that’s organic to them instead of them looking at a list and reconciling things. We’re a little bit off from that.
The problem with what we are doing now is that we are trying to find an easy way out. We’re saying, let’s just take the note and use NLP, a large language model, or something else to read the note and turn the note into something structured. You can do that, and we have had some success when it comes to high-certainty type data like pulling ejection fraction out of a procedure note or looking across a complete patient record and coming back with a sense of the patient’s diabetes because I found all these references that I can correlate to that. But you still run into the problem of, how can I trust that?
When you look at all the things that are happening in the industry now with AI, large language models, and NLP, there’s a lot of talk about transparency. In the past, when people have tried to do things in healthcare with these types of approaches using NLP or AI, it hasn’t been successful. The machine works great 60% of the time, and then 40% of the time it does something horrifically wrong. That comes back to this idea of trust. If you are taking care of somebody and their life is in your hands and the machine just happens to pick that day to follow the wrong probabilistic pathway, that’s challenging in healthcare.
Thinking back to providers not trusting their own data, is that a vague impression or is it measurable? What could they measure to assess or improve data quality?
When I’m working with clients, I sometimes ask them this question, so I’ll ask you. When it comes to healthcare data for an individual patient, who is responsible for the quality, accuracy, and integrity of that one patient’s data, regardless of where it is?
Some would say the patient, although that’s not a reasonable expectation for all types of data.
The problem is that patients aren’t really trusted to do that. You can fill out a form, hand it to somebody, and they’ll type it in, but rarely is a patient trusted to own and articulate, “Here’s my health situation.” It usually has to be vetted by some kind of clinical person.
That’s fine, but it goes back to this root problem that nobody is responsible. There is no data steward for an individual patient’s health record. When you talk about how you trust the data, the fact that I can take one patient and look across multiple venues of care and see different information. They don’t really trust each other and where their data is coming from. They don’t know whether that ICD-10 code was added for billing purposes or added for clinical purposes.
The problem we have in healthcare is that we don’t have a mechanism that allows us to objectively and quantitatively look at the data and say that the quality is good or bad. We are working with other organizations to do this taxonomy for healthcare data quality, because I think that we should be able to look at patient data in an abstract way and say, is the quality of this data good? Is there duplicate stuff? Is there old stuff? Is there stuff that’s clinically impossible? Are there things in the medical record that contradict themselves?
How can we automate the evaluation of that semantic interoperability so that you don’t need a sweatshop full of clinical people looking at 5 million patient records? How do you build something that can objectively, with some type of deterministic AI, evaluate an individual patient and any data that comes in for them to say, yes, this all makes sense. It looks real, and I just noticed that there’s no mention of this patient’s diabetes, whether you’re looking at unstructured notes and pulling it out.
At the very least, you should pull the data, check it against the integrity of the rest of the medical record, and say, yes, the fact that the note says they are diabetic resonates with the fact that they’ve got a funky fasting blood sugar and they’re taking these three medications that are indicated for diabetes. Let’s go ahead and suggest that they add diabetes to their official structured medical record so that we can take advantage of that. All these things that only look at the structured medical record and retain the evidence of where that came from. Those are some things that we could do to improve the level of trust and the reliability of the data.
My big fear is that we start to roll out some of these more sophisticated things that could be beneficial, but because the data quality is bad, we fumble the results early on and these things fail, and because we applied them before the data quality was ready, people don’t have confidence. You only have one opportunity to be credible. You come in with this new technology and say, “This is going to save lives. This is going to do great things.” But because the data that we are feeding it is bad, it is very possible and probable that the results of what it comes up with will be likewise bad. We will flip the bozo bit, as they used to say, on that thing. Then later, when we fix the data quality, we say, “No, we tried that and it didn’t work.” But maybe it would work if we fed good quality data.
What is the oversight structure and mechanism of reviewing the longitudinal patient record from multiple providers and identifying missing or conflicting data? Then, going back to the data source and either asking them to fix their problem or perhaps excluding their data as being unreliable?
The first place is the pipes. Look at what’s happening with TEFCA and QHINs. Let’s say the QHINs turn on their pipes and people start streaming data from Point A to Point B for every patient. The first thing we need to do is, somewhere in that pipe, we need to have something that looks at the message. Is the message right? Does the data look fundamentally correct? Not clinically correct, but is a valid code in the value set? Let’s say it’s an RXNorm code. Does the name match what RxNorm says that code stands for? So the first thing you do is evaluate someone in the network to determine whether they are a good data provider.
If they’re not a good data provider, you can’t really remediate data quality in flight. You have to go back to the source and say, you’re not a good data provider. This is what our taxonomy is focused on. By identifying the nature of the quality failure, you can go back and say, you’re putting the decimal place in the wrong spot on your lab results. You are not using a valid RXNorm code set. Your maps are bad. Whatever the feedback is.
The first thing we need to do is to make sure that the people that are sharing data from their systems are good members of society who care about the data they are sending out and are making sure that the quality is good. QHINs are going to be in a great position to evaluate the data in flight at a basic level and say, OK, the data that you are sending looks clean, looks good, and has good intrinsic quality. That’s the first step, because that’s where you stop bad data from getting out.
We also need to do a better job of knowing where data’s coming from originally so that we can stop duplication. We worked with a partner who gave me a bunch of data to evaluate, data that was coming from a bunch of different sources. In a couple of million records, there were about 750,000 duplicates, the exact same lab result done at the exact same time. Because of the way the data was shared in some of these older formats, you had no idea that that was the same data. It just looked like this patient had 64 lab results on the same day at the same time.
That’s the other thing, if we want to trust data, we need to know where it originally came from, especially as we start sharing data across an entire network of participants.
The last thing is you need is a way where we are landing it or looking at the data in our own system, saying, does it look right for every condition that I have? Do I have a treatment for every drug that’s in their profile? Do I know why they are taking that drug? This goes back to what you are talking about when it comes to oversight. Within any repository of patient data, perhaps a large IDN doing analytics or population health on your patients, we need to have mechanisms that can identify issues in the patient. Data can alert a human operator. Let’s call them a data steward. The data steward can inform the systems that they are connected with on how to remediate the data.
There needs to be oversight. The trick is, how do we have enough automation in place so that instead of a human looking at 5 million patients, automation is looking at 5 million patients for things that are a concern, and streamlining the resolution of those things? Because it’s easy for a human to be presented with something and say, “Yeah, that looks right,” as opposed to humans poring over data looking for something. That’s why when we do semantic normalization, our software does like 85% of the work, where it tries to search for the right target and it suggests the target. A human can take two seconds to look at the target and say, “That’s right.” We need to get to the same place when it comes to patient data.
It’s one of those things where the idea of having people whose job it is to review issues that come up with patient data and resolve it at a patient level might seem a little daunting, but the problem is, that’s the only way we can fix it. You have to fix it at the atomic level to have the entire ecosystem be of high quality. There’s no way to do it at a macro level. You have to do it at an individual patient level.
What factors will be important to the company and the industry in the next few years?
For us to use artificial intelligence and some of these other things that we are coming up with in a meaningful way, we are going to have to move away from pre-coordinated terminologies as how we collect data for patients. We’re going to store patient information in a much more granular graph style, so that both software and people can make better use of it. Right now, everything we do with the terminologies and practices that we use today create these big pixels of information that limit our ability to do sophisticated reasoning over that data, whether it’s for research purposes or for decision support purposes. We’re going to have to dial up the resolution to get to where we want to be in terms of software providing meaningful assistance to people that are providing care.
Great article Charlie,
Thanks for illuminating many of the issues around clinical interoperability.
I spent half a decade slinging data between multiple EHR systems (in a lab) and we came up with a set of phrases for what happens when the data exchanges: Refraction, Reflection, Distortion, and Zombie records.
You mentioned the data that is duplicate, but what happens when the data is brought in and distorted — it is the same data but is no longer recognizable as its original source data. An intolerance that comes in as a full blown allergy, or an immunization that has an invalid CVX code (leading zero’s missing). So in that case, it is the same, but isn’t.
Zombieism and Reflection are cousins: Reflection is when your data is consumed by someone else, then presented back to you, if it isn’t exactly the same it can easily be incorporated and clutter the patients record. Zombieism is when something that should be corrected is reflected back and is reintroduced into the record.
Had a trauma event a couple years ago and had just reconciled my a couple days prior. Four health records, on the same platform, and the ED doc started asking me about my allergies, medications, conditions — and they were all wrong — I had to tell him that we had just reconciled the record and I was on no meds, had no medication allergies, and we corrected my blood type, yet again. A victim of Reflection, Refraction, and Zombieism.
Charlie, if you were to get hospital systems to look at three specific data corruption taxonomies, which three would you ask them to address?
Brody,
It is timely that you ask this question, as I just finished a webinar where I propose a potential taxonomy for healthcare data quality issues. If you are interested, it can be found here (https://www.youtube.com/watch?v=JO6Jt1kIdqA).
I think the dimensions you describe are common and I like your optics-based analogies (Zombie notwithstanding) and you can add another one, hallucination, where you get incorrect data based on a patient identity issue (I like extending a good metaphor).
To your question, in an environment where we are encouraged to share information, it is inevitable that in the process of ingesting and normalizing the data we risk the ‘whisper down the lane’ effect of duplicating the data, transforming the data and losing the data’s provenance. These are issues that can affect original data, but you find them mostly in exchange scenarios.
As to the three qualitative dimensions I would suggest they address, it depends if they are the sender or the receiver.
If they are receiver of data, I would suggest:
1) They retain the original data records with source information for anything they receive and make sure it is available for clinician review.
2) Ensure that any code mapping performed has a semantic component and is not just relying on the codes or translations. Another form of corruption is when the code that someone mapped was not a good match for the original information. (I have seen CA-125 lab results mapped to Calcium…)
3) Verify that you have not already received the data by examining source, dates and other key values. (Another reason the original data is important)
If they are the sender, I would suggest:
1) Make sure that all data is mapped to non-retired codes in the standard code system recommended/required by the standard you are using (USCDI, US Core, OMOP, etc).
2) Make sure to include the provenance of the data (source facility, source application, date, time, author).
3) Consider whether you should share information that was not sourced by your facility. In other words, share your data on the patient, not information you received from someone else. This is a little controversial but
in a world of exchanges we should be able to get each sources original information from each source… right?
I would also remind any sender of data that sharing the data is not just something that is done to avoid penalties, but rather to help others provide the best care for their patients. I sometimes think this gets lost in the chaos of providing healthcare. We need to share the information in a way we would like to receive it. Like Mahatma Gandhi said “If you want to change the world, start with yourself” (that’s right – I played the Gandhi card).
Thanks for your thoughtful comment.
This article really hit the mark for me and my work history! Lot’s of comments:
“We have this illusion on the analytics side of healthcare that the structured data is of high quality.”
Well, I never had that illusion, or was quickly disabused of that notion. My universal rule is, no one cares about data quality until someone tries to use it. Once you get real data consumption, you can move the needle on data quality, but not until then.
“People generally did not think very much of their own data quality […] only thing they had less confidence in was the quality of other people’s data…”
Medical personnel optimize for their time and the patient’s care. A clinician is tempted to dismiss 3rd party data if any of it is seen to be problematic. Why? Because the clinician can re-generate that data locally, using Lab tests, exams, and the like. At least they have control over that and can even understand local failure modes. Usually, they have little or no control over the 3rd party data source, and frequently no insight either.
“…people tend not to care as much about the data that’s leaving…”
Yeah, that’s gonna happen. And there’s really nothing you can do to change that. It’s not just selfishness either. Scope of practice, responsibilities, and the effort required all play into this dynamic.
“…maybe it would work if we fed [it] good quality data.”
An intriguing notion. There’s a lot of corporate psychology and politics in playing the “it failed” card during meetings.
“They want to be able to articulate something in a way that is natural and organic…”
See, I tend to think that this is going to be nearly impossible to achieve. Current AI systems do not truly understand the meanings of words, and this essentially mandates that they must. The same is true of neural nets, frame-based knowledge representation systems, and all the rest. Clinicians have an incredible ability to represent the same information in myriad ways. Computers can only play in this space by mandating a single knowledge representation system.
A couple of additional points.
“You have to go back to the source and say, you’re not a good data provider.”
I used to think that the silver bullet for data quality was having dedicated data quality resources. Analysts who actually did this for a living, and entire org structures to support them. And then we got them…
My experience was enlightening, to say the least. First off, they helped, there’s no question of that. However, the EMR was generating so much data, that the data quality analysts needed to be pointed in the right direction. Without a focus they didn’t have enough direction and their impact was negligible. The DQ analysts were aware of this and so they actually required direction for most of their work (understand, I’ve got an outsider view of their work, so this is my impression only).
What this meant was, in practice, you still needed a data consumer. Without real data consumption, even the DQ analysts wouldn’t get involved. It literally wasn’t worth their time. You can’t make a business case to expend valuable analytical DQ time on data that doesn’t get secondary use. And by definition, everything out of direct patient care is a secondary use.
End the end, the physician is right at the nexus of data generation. I understand they get frustrated by the administrative overhead of managing this, but what is the solution? We can ask Diagnostic Imaging to own their generated data, the Lab to own theirs, and a handful of others. However we cannot pull the attending physician out of the loop. It just does not work.
That leaves us with workflow optimization, and taking full advantage of EHR features that are helpful in this.
Brian,
I think that is what I was trying to get to on the data quality side with my Taxonomy reference. Charlie has laid out a first and second tier taxonomy, but I think we should be focused on the third tier: the actual errors that are generated within the Healthcare Data Quality Taxonomy (HDQT). Once you have this, then you can begin to put checks in place that will seek out these defects. That said, the last thing I would want to do is “alert” the clinician that he has these defects — unless it really was valuable.
I could see putting a check on the free text portion of an allergy entry to see if there are ‘medications’ listed in the free text. But other than something like that I wouldn’t want to alert the user and increase the alert overload. However, I could see that the system could self monitor at some level and prevent some of this bad data entry.
Also, not sure where HDQT is in its lifecycle — it certainly isn’t a Google term yet — which means if we are going to adopt it, we need some papers and articles on it. I think we need a Bezier like taxonomy that gets to the third or even fourth level. And, once you start hanging third and fourth order anomalies on the taxonomy the first/second order items will change.
I appreciate your insights, Brody. You seem to be looking at the HDQT from a meta level and I support that approach.
“End the end”, LOL, did I really write that?!
Great article, and really insightful responses.
But one issue no one touched on that contributes to all this ‘variation’ in terminology is Medical School training. If a doc graduates from Duke and another from Stanford they are exposed to core medical definitions and terminologies that are consistent, but as they move into specialties there can be significant variations, all dependent on a particular professors preferences.
This is ironic since it is usually the research/academic MD that complains most about ‘bad’ clinical data documentation, yet it they who train those clinicians.