Curbside Consult with Dr. Jayne 9/30/24

A care delivery organization recently asked me to work on an AI project. They are looking at ways to incorporate generative AI into the clinical process, but didn’t want to use an off-the-shelf solution due to concerns around cost and clinical quality. Instead, they set out to create their own solution, which I suspect was in part a way to justify the recent creation of an innovation team, which hadn’t yet produced anything mind-blowing despite being a significant cost to the organization.

Although I sympathize with their desire to have a system where they can work behind the scenes and ensure the validity of the data being used and the outcomes, I could have told them months ago that they would spend way more money taking the do-it-yourself approach instead of working with someone who already had expertise in this area.

Their solution is pretty far along in the development process. They have had a single physician who is providing input. They are ready for more physicians to be involved, and because they are a care delivery organization, they assumed that physicians would be clamoring to be part of the project, either in providing clinical scenarios or being beta testers.

However, they didn’t budget compensating those physicians for their time, which given the tone in their physician group, was a significant oversight. Physicians who are already feeling burdened and burned out are less likely to give freely of their time to an organization that they feel is not working in their best interests.

I started my work with them by attending an onsite meeting where the team was strategizing on how to convince providers to be more involved. They asked me to go around to offices and try to convince physicians to participate.

It quickly became clear to me that many of the people on the innovation team had not worked in healthcare. They thought that it would be great to just show up during office hours and try to get people’s attention. I had to do a little explaining about how physicians are so protective of their time that many of them have eliminated the presence of non-essential people in the office during the day – no drug representatives, no lab representatives, no med students, etc. They were surprised by this, so I got to share how COVID really changed this landscape and how once clinicians realized how nice it was to not be interrupted, they weren’t going back.

It turns out that during the development process, no one had been working with physician leaders to talk about the project and to build consensus around its use. I found that pretty remarkable since most organizations have by now learned the value of buy-in.

I asked to meet with physician leaders so I could build an understanding of the physician group’s culture and whether there were people who would be willing to participate and what kind of compensation or reward might be needed. Everyone is motivated a little differently, and some will respond to non-monetary incentives like being the first practice to use a new tool or being bumped up in the line for enhancement requests that they’ve already entered. Others do want to participate in making things better, so I thought we should learn about any existing physician wellness committees where we might find willing participants.

I also suggested that since there is a corporate IT department, it might be interesting to pull search histories on some of their clinical users to determine what kinds of things they might be asking Dr. Google. There was a lengthy conversation about this being a violation of user privacy, which surprised me. How many annual compliance training sessions have I been through that explained that nothing that is done on a company-owned device or on a company network is private? Had I stumbled into an alternate universe where people had no fear of corporate types seeing what they were doing on their work laptops?

Although they agreed in principle that it would be an interesting approach, they said that they would have to take it through various approval processes. It was a non-starter in the short term.

In the meantime, while we were working through that issue as well as working with physician leaders to find clinical testers and potential beta sites, I agreed to create some testing scenarios across various specialties. I drafted some requests to pull diagnosis data from their EHR to better understand what kinds of conditions were being treated. My thoughts there were twofold. First, I wanted to find out the most common conditions for which there might be a need for generative AI around patient-facing communication, clinical documentation in the EHR, or other use cases. Second, I wanted to understand the least common conditions for which users might be seeking additional information, either about other similar conditions or about treatment of a condition once they had narrowed it down.

I was a bit surprised that their in-house lead clinician hadn’t suggested these things, and it became more clear in some of those conversations why the organization wanted to bring in someone a bit more seasoned to assist.

I decided to start building test scripts around the organization’s genetics clinic since physicians are seeing increasing numbers of patients who are asking for full-panel genetic testing to try to understand their cancer risk. These tests can be expensive and are often not covered by insurance. They also test for genes that the average primary care physician doesn’t necessarily encounter on a daily basis, figuring there would be searches about them. I developed a test plan and got ready to kick the tires.

The first test scenario I did was with a condition that I thought would be an easy one since it’s fairly common and testing has been around for more than a decade. I asked the system what the clinical implications were for a patient who was a homozygous carrier of the condition, since the answer should be straightforward about early screening. The answer was anything but straightforward, with the system taking me on a wild ride that ultimately ended in a recommendation to do nothing. I was stunned.

I tried quite a few more test scenarios and the system performed as expected, but I was left with a bad feeling about how to proceed. The engineers who had been following my testing didn’t think that one miss was a big deal, but to me as a clinician, the miss was a very big deal. I knew I would have another topic for my meetings with clinical leaders as we would need to discuss what the organization’s tolerance was for misses and near-misses, and also whether there were ethics committees that we could bring to the table.

I was starting to feel like this project was one of those “on the back of a napkin” efforts that hadn’t been fully fleshed out and would ultimately need more discussions than I was prepared to lead as part of my engagement.

We’ll have to see how this shakes out over the next few months, but it left me wondering how many other organizations are in positions just like this, taking projects forward when they don’t have the right stakeholders at the table or an understanding of the true clinical implications of the technology they’re trying to add to the mix. I suspect we’ll have a lot of uncomfortable conversations, and some folks won’t be happy that this outsider is poking holes in their project. Alas, that’s all in the fun of being a consultant, so I’ll just keep putting one foot in front of the other and try to navigate them in the right direction.

What is your organization’s process for ensuring clinical stakeholders are involved in clinical technology projects? Leave a comment or email me.

Email Dr. Jayne.