Curbside Consult with Dr. Jayne 10/16/23

Generative AI continues to be a hot topic around the virtual physician water cooler. My colleagues have come to expect me to have my finger on the pulse of innovation, even though interacting with these solutions is a small part of my current work in clinical informatics.

OpenAI recently announced that “ChatGPT can now see, hear, and speak,” heralding an opportunity for a “more intuitive type of interface” that allows the user “to have a voice conversation or show ChatGPT what you’re talking about.” The blog details the potential of the technology to impact daily life.

As an aspiring amateur chef, I’m intrigued about the potential to “snap pictures of your fridge and pantry to figure out what’s for dinner (and ask follow up questions for a step by step recipe).” At any given time, my kitchen has a stack of recipes that I find intriguing. It would be great if artificial intelligence could parse them and determine which ones I might be more likely to make, and what ingredients I need to make their creation a reality.

Plus and Enterprise users get first crack at the new features, with voice coming on mobile platforms and images being available on all platforms. As for voice, the goal is for users to be able to have a conversation with the virtual assistant after choosing one of five voice options.

As an early adopter of the Garmin Nuvi back in the dark ages before Google Maps, I miss my former trusty companion “Ken the Aussie” and was hoping that there would be a similarly engaging option available with the new solution, especially since the company states they worked with professional voice actors to create the options. Alas, I didn’t find an internationally-accented voice – the options are named Juniper, Sky, Cove, Ember, and Breeze.

I can’t wait to explore the image options. One of the use cases that OpenAI lists includes “analyze a complex graph for work-related data.” I have some absolutely crazy pictures of whiteboard drawings that I’ve collected over the years and can’t wait to unleash AI technology on those and see what sense it can make of them, if any. New York Times reporter Kevin Roose got a sneak peek at a beta version of the technology recently and shared his results. He noted some issues:

• Taking a picture of the front page of the newspaper, he asked ChatGPT to summarize it. The AI hallucinated, “inventing a statistic about fentanyl-related deaths that wasn’t in the original article.”
• The technology “flopped” when asked to assist with a crossword puzzle.
• It referred to a stuffed dinosaur as a whale.

It was also unable to assist with deciphering a diagram for assembling a piece of IKEA-esque furniture, although I’m less surprised by that than the other issues mentioned. He also noted limitations in how the technology processes images of human faces, although he mentions this is functioning as designed. Developers wanted to avoid it being used for facial recognition or critiquing people’s appearance. As someone who has cared for teens who have been cyberbullied, I’m grateful for the latter.

The New York Times reporter found the voice capabilities to be particularly powerful, referring to it as “Siri on steroids” with a “fluid and natural” voice that has “slight variations in tone and cadence that make it feel less robotic.” He notes that his request to hear the story of the Three Little Pigs “in the character of a total frat bro” was “a sleeper hit.” (The example is available in the article, if you’d like to give it a listen – I agree it was pretty funny.)

Honestly, I can’t wait to ask it to tell me a story about healthcare IT from the perspectives of some of the archetypal personas we see in the industry: the exuberant CEO, the frustrated project manager, the surly end user, and the burned-out clinician. It would probably be more entertaining than some of the talks we saw at conferences like HLTH, hands down.

Earlier in the month, OpenAI had also announced that its Dall-E image generator was being incorporated into ChatGPT. When AI-generated images first came on the scene, there were a lot of concerns about copyright issues, competition with human artists, and the role of AI in the creative process. Now that the technology is becoming more accessible, some of my physician colleagues have also been concerned about the potential for using generative AI to create images that can be passed off as medical records in order to manipulate a physician into providing treatments or medications for which a patient might otherwise be inappropriate.

There was a big discussion among our group about the potential for diversion of controlled substances if patients presented with AI-generated x-rays, CTs scans, or MRIs. I’m seeing an increasing number of physicians paying attention to political happenings, so of course there was concern with the potential to use AI to manipulate upcoming elections. Of course, there are plenty of bad human actors that already have the technology to do those sorts of things, but somehow things just seem scarier to some when automation is involved.

OpenAI isn’t the only company that’s doubling down on chatbot investments. Google recently released improvements that allow Bard to access information from Gmail, Google Docs, and Google Drive accounts through a feature called Bard Extensions. It can also draw from YouTube and Google maps for information. Although those enhancements potentially represent a substantial increase in generative power for sophisticated applications, I’m more interested in straightforward but potentially complex tasks, like helping me parse through the hundreds of emails I receive each day across both my personal and HIStalk accounts and helping me identify which ones might be most intriguing.

Given my consulting work helping organizations streamline their meeting calendars, I’d also love to unleash a chatbot to parse calendar data to help me figure out which meetings should be moved to maximize attendance, which should be kept where they are, and which might be able to be eliminated. Of course, most of the organizations I work with are still devotees of Office 365, so Bard isn’t going to be much help there.

What do you see as the areas with the greatest potential for generative AI, and what do you see as the biggest potential pitfalls? Leave a comment or email me.

Email Dr. Jayne.