I agree that to evaluate the quality of an NLP application it is imperative to know its precision and recall scores. In common language, recall measures the engine’s ability to accurately identify the full range of correct answers and precision evaluates the accuracy of the answers.

For example, when CodeRyte processes a note and identifies 10 ICD-9 codes and an expert coder agrees that each of those 10 codes were appropriately supported by the documentation, then the precision would be 100 percent. However, if the expert coder also thinks that the engine should have identified 20 ICD-9 codes, then the recall score would be 50 percent.

As Dr. Scarlat points out, there can be a trade-off between precision and recall. If it were more important that every ICD be right, then an NLP vendor might be willing to have lower recall. On the other hand, if it were more important that no ICD code is missed, then an NLP vendor might be willing to have some incorrect ICD codes in order to make sure that all the correct ones were included.

Thus, the F-Score…

In addition to creating a high-performing coding engine, there was an additional challenge that CodeRyte had to overcome to successfully bring our NLP engine to the market. We had to put “accuracy” in context. In the scenarios above, for example, one would expect that three medical coders would agree that each of the ICDs selected by the engine were correct and, furthermore, would agree on exactly which ICD codes should have also been included. Unfortunately, that is not how it works. There is significant disagreement among coders.

Because CodeRyte believes in transparency, and especially when bringing a disruptive technology like this to the market, we have already published metrics regarding CodeRyte’s performance. The papers, which get to the heart of your questions, were written by our team of engineers and NLP pioneers to dissect automated confidence assessment for compliant accurate coding.

Because those papers were published two years ago and our scores have improved on every metric, I suggest that we set up a meeting during which we can provide you detailed answers to your questions.

1. Can Andy share some metrics regarding CodeRye performance (such as precision, recall and their combined F-statistic) ?
These metrics will provide an insight into CodeRyte capabilities for coding righteously…Minimum under-coding (loss of revenues) and minimum over-coding (auditing, penalties, etc.) Unfortunately as with other binary classifications techniques, a gain in one metric means a loss in the other and vice-versa. Thus the F-statistic (no pun intended).

2. Does the NLP system “learn” in the process ? Does the algorithm performance improve in time?

3. How long it takes to demonstrate a ROI for a medium size hospital ?

Many thanks,
Alex Scarlat MD

I agree that the pushback is almost entirely about the cost, and that it is wrongheaded. It is surprisingly easy to pushback on the structure of ICD-10-CM.

The upshot of it all is that “accurate diagnoses” are impossible with a disease classification, as opposed to disease, coding system. So long as you code the class of diagnoses into which a diagnosis falls, and not the actual individual diagnosis, you have a problem.