Curbside Consult with Dr. Jayne 7/25/22

This weekend marked the first time in a decade that I took a trip without my laptop. It was weird at first because my backpack felt surprisingly light, but it was good to know that I couldn’t slip into the trap of doing work even if I tried.

Now that the workplace is more mobile, and especially when some companies offer unlimited time off, it’s tempting to blur the lines between the work day and the rest of your life. It was a great opportunity to connect with old friends and focus on spending time together as we crammed as much adventure into 48 hours in the Big Apple as possible.

Of course, since we were a group of healthcare IT people, there were plenty of opportunities to discuss where the industry has gone since we all worked together at a big health system. Several in the group have moved into the non-profit space and payer space, which was interesting to learn about compared to others’ experiences in the vendor and consulting spaces. One person’s resume reads like he’s a job hopper, but in reality, he has been in the same job, but his company has just been merged and acquired multiple times. I’m not sure everyone would pick up on that just looking at his employment history.

There was a lot of chatter about the Amazon acquisition of One Medical for nearly $4 billion. Nearly everyone in the conversation was concerned about Amazon having access to patients’ protected health information, even though the company issued a statement that they don’t plan to share health information to advertise or market other Amazon products without clear permission from patients.

Several of us brought up the point that most consumers don’t read terms and conditions, even when there are limited updates to an agreement, so it’s likely that Amazon could obtain such a consent without patients full understanding what they’ve accepted. Most of us are a bit skeptical about how this is going to play out, so we’re going to have to watch it as the deal moves towards the regulatory approval it needs for closure.

Digital health startups were a hot topic throughout the weekend, and one of my traveling companions sent me an article that caught his attention. It’s from the Journal of Medical Internet Research and looks at the “clinical robustness” of companies in that sector. The background for the article notes that “health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry.” That phrase caught my eye as well, since I’m constantly seeing companies that make me wonder what exactly it is that they do and why people, including investors, are so excited about them. Certainly there’s a hype factor for a lot of digital health startups, as well as a great deal of marketing, but the meaty information tends to get pretty thin when you start to dig into some of them. The article looked to assess clinical robustness along with the claims made by companies.

The authors used data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. (It should be noted that three of the authors are employees of Rock Health, although the article states that no individuals involved in investments were part of the analysis.) The authors defined clinical robustness using the sum of the number of regulatory filings and clinical trials completed by each company with each having equal weight. For the companies’ claims, the authors looked at company websites. They looked at 220 digital health companies with an average tenure of 7.7 years.

The average clinical robustness score was 2.5, with a median score of 1. Companies specializing in diagnostics had the highest scores, followed by those focusing on treatment and those specializing in prevention. The authors found that 44% of the companies they looked at had a clinical robustness score of zero. There was no correlation between clinical robustness and the number of publicly made clinical claims or between clinical robustness and total funding.

There were some other interesting statistics in the article:

• More than 1,900 digital health startups have raised more than $2 million in venture capital funding in the US.
• In total, digital health startups have raised more than $77 billion in venture capital funding.
• Multiple studies have shown the need for greater clinical validation.
• Many solutions are not supported by robust clinical evidence.
• Solutions have mixed results on cost savings and cost-effectiveness.

Additionally, the authors noted that there have been examples of misleading claims in the industry, as well as actions by the Federal Trade Commission and state attorneys general. They note the limitations of using publications as a proxy for clinical impact, especially given that there is often a lag in publications. There were also limitations in that they only looked at venture-backed startups in the US that had raised at least $2 million in funding, which may have excluded early-stage companies as well as larger technology companies. They also note the need for deeper research into the effectiveness of solutions.

I found it amusing that the article noted that “all data were stored in Microsoft Excel,” where the calculations were performed. Excel isn’t always the most sophisticated tool, but it’s gotten the job done for me more often than not.

It would be interesting to do a follow-up study to see how many of the startups that the authors looked at are even here in two, five, or 10 years and what their clinical robustness might look like over time. I suspect we may see some significance between funding and clinical strength or between time in the market and clinical outcomes. In the short term, though, it’s often difficult to demonstrate value beyond individual case studies and client profiles. It’s certainly challenging to wade through some of the content on company websites and to truly know how broadly a solution has been deployed or whether clients are happy with it.

What do you think about the idea of measuring clinical robustness for digital health tools? What markers might you use beyond those selected by the authors? Leave a comment or email me.

Email Dr. Jayne.