Diagnosis: Paralysis by Analysis
By Dan Harvey
Radiology Today
Vol. 19 No. 5 P. 16
Are quality initiatives in radiology trying to measure the immeasurable?
A new medical product invites clinical trials, and, through statistical evidence, a verdict about its utility can be reached. Succinctly describing the overall value of radiologic technology and images, however, is another matter. Value in radiology needs to be looked at through economics and quality-related metrics, but the value of a medical discipline is much harder to define than a new piece of equipment. Thus far—and possibly for the foreseeable future—radiology has defied measurement.
Defining radiology's value has become somewhat of a hot-button issue, but it is not an area that is easily addressed; it certainly provides no easy answers. It's new territory where philosophical ideas combine with science. Measuring quality could be possible, depending on correct interpretation and subsequent accurate diagnosis, but would that entail a case-by-case examination? Jeffrey D. Robinson, MD, MBA, FACR, an assistant professor of emergency radiology in the department of radiology at the University of Washington, says what's more important right now is to avoid facile measurement.
"In our current society, the quantifying of anything has become something of a cultural fetish," Robinson says. "We try to associate everything to a number. It has gone a bit overboard."
Certain entities are focusing on answer acquisition, however, even if the questions have not been clearly identified. In large part, the effort is pushed by economics.
"Much of what drives all of this is reimbursement," Robinson observes. "As imaging volumes go up, costs go up, and payers such as the insurance industry and Medicare want to know that they are getting the largest value for their imaging dollars as they seek to control costs. It's a natural question: 'How do I know that I am getting value for what is being spent?' So economics is a large factor."
Quality management would seem to loom large in the equation—but what's the best way to determine optimal quality? That involves one very substantial challenge.
"There's inherent subjectivity related to quality," Robinson says. "Everyone seems to have their own quantitative measures of quality, especially in imaging, but in medicine in general as well."
Negative or Positive?
Robinson offers an example that helps underscore the complexities. It involves CT pulmonary angiography, a test that is performed when a patient is suspected of having a pulmonary embolism (PE).
"But we routinely see that most of those patients [who are suspected for PE] don't have pulmonary emboli," Robinson says.
One line of thinking asserts that, because the test didn't result in a diagnosis that led to a beneficial action, it doesn't have value. Robinson disagrees with that point of view.
"A negative test has a value in and of itself," he says. "What it means is that it has been established that a patient doesn't have a PE. That's value."
Not everyone thinks that way, he adds. Those who don't see value are metrically inclined; that is, they are trying to establish firm metrics.
"That line of thinking goes like this: If you do 100 tests and 99 are negative, that doesn't represent very good value," Robinson says. "It can mean that the test is being used excessively."
But there's another perspective that needs to be taken into careful account, specifically because of the conclusion it draws.
"So a patient comes in short of breath and with chest pain. Well, if a test has a negative result, it may be useful information because the physician is able to rule out the possibility of PE," Robinson says. "This reasoning assumes that the physician has done a good job in thinking about what is causing this particular patient's symptoms. It's not a good idea to do CT scans on everyone."
In other words, a suspect has been eliminated from a detective's list of possible perpetrators. Possibilities have been narrowed, a step—albeit only one—that leads to the correct arrest and indictment. To state this value in medical terms, a patient will not be treated for a nonexistent condition. Robinson suggests that this is an oversimplification, but he adds that it reflects what happens many times, in many places, and with many conditions.
"It is just one example of why it is so hard to get our arms around this issue," Robinson says.
Seeking Questions
A viable option would appear to be new research into the issue, but that leads to a significant problem: How can you come up with useful answers when you aren't even sure what the relevant questions are?
"The problem resides within how to formulate the question," Robinson says. "To do a research study, you need a well-defined and characterized question that you seek to answer. And we really haven't focused on 'the question' yet, so it's not yet precise enough. One possibility would seem to be, 'How can we improve quality?' But that's not good enough. From the economic perspective, there seems to be one question. From the patient perspective, there is a quite different question."
And from the radiologist's perspective, there may be still another question. In addition, Robinson brings up one more issue that involves a fact of life about research studies.
"For most research that goes on in this world, you need to formulate a question in a manner that is engaging to whomever pays for the research," Robinson says. That means strong consideration of the funding mechanism. "If a payer is funding the research, they are looking for ways to increase the yield of an exam," he explains. "[In the earlier example], they're thinking, 'How can we find factors that find the patients that have PE, so that we can scan them—and only them—and not the patients who really don't have PE?'"
Robinson believes that's a worthy question, but he adds, "On the other hand, if research is being funded by a patient advocacy group, it might not be nearly as interesting a question. Research from this direction might be more focused on how it can be more clearly established why patients demonstrate shortness of breath and report chest pain. These groups are not primarily interested in increasing the yield."
Defining Quality
Robinson took part in "The Value of Imaging," a session at RSNA 2017. One of the focus areas he discussed was how radiologists are doing, in terms of quality, when interpreting studies.
"That is a totally different phase of quality," Robinson says. "That brings in a whole other layer of subjectivity, in that, if you place 10 radiologists together in one room to look at a scan, very often you will get opinions divided in half. It can become difficult to come to consensus."
That leads to a question of where the mistake is being made. "Radiologists do make mistakes at times, but we also need to know if an error is an outlier," Robinson says. "That would take a tremendous effort, something that needs focus and subsequent activity. It's more than just saying, 'Let's try and not do that again.'
"That then becomes a quality assurance question and, by extension, that could turn toward a malpractice question."
Robinson says one of his areas of interest concerns cases where image interpretation is challenged. "When there is departmental controversy, or even a lawsuit, how do you assess quality in that kind of situation?" he asks.
He indicates that there are a couple of ways to do the assessment, but the fundamental principle is blinding; in experiments or research, information may be strategically withheld to reduce or even eliminate bias, however unconscious the bias may be. Robinson explains how this would work in a lawsuit situation: "The two different unbiased approaches radiologists should use to measure accuracy are blind panel prospective review and blind expert retrospective review. In a lawsuit situation, when there is a radiology expert who is going to define whether the attending radiologist did or didn't meet the standard of care, it is important for that expert to review the study in the situation similar to what the original radiologist did.
"Blinding can be done prospectively, in which an exam of concern gets a new name and today's date and is inserted into the everyday workflow of a radiologist, who is unaware that the exam is anything other than another ordinary exam," Robinson continues. "In retrospective blinding, a reviewer is aware that there is an issue with one exam but is presented with a series of exams, only one of which is the controversial study."
Also taking part in the RSNA 2017 session was Saurabh Jha, MBBS, MRCS, MS, an associate professor of radiology at the University of Pennsylvania. Similarly to Robinson's viewpoint, Jha believes that the underlying problem in determining the value of imaging involves fluid elements such as subjectivity and discernment. Discernment involves the accuracy of a reading and the subsequent diagnosis, as well as the implications of that diagnosis. The correct—ie, accurate—diagnosis is of utmost importance, as it guides the clinical decision-making process.
Like Robinson, Jha also believes that the question of false negatives vs false positives is an important consideration for determining quality. "One perceived problem is that a trade-off appears to be involved," Jha says. "But what a patient doesn't have is just as important as what a patient does have."
Still, ruling out a medical condition doesn't lead to a clearly defined measurement. "Measuring is actually hard to accomplish, as it can involve measuring a radiologist's performance," Jha says. "In turn, this means looking at each and every case and determining whether there was enough 'correctness.'"
Jha observes that much of the discussion involves the development of quality metrics, which leads into concepts like pay-for-performance and, possibly, penalties. And that, he says, could even lead into the question of who is the better physician, based upon their approach.
Two Approaches
To underscore the complexities of such a question, Jha developed a scenario that involves two fictional radiologists, "Dr. Singh" and "Dr. Robert." Dr. Singh turns around reports rather rapidly, 15 minutes on average. The reports are succinct and not made oblique by an offering or consideration of different diagnoses. Rather, Dr. Singh describes findings in concise fashion. Specificity is of high importance and reports are characterized by decisive language. A "flagged pathology" is very likely present. At the same time, the reports appear to lack a certain level of sensitivity; subtle pathologic features are missed. Further, the focus of Dr. Singh's reports is too narrow. Not all normal organs are listed in the report. For example, a mention of the appendix or pancreas may be omitted from a report of the abdominal region. This leads referring physicians to question whether something has been overlooked.
Dr. Robert is much more methodical, which means his turnaround time is much longer. Furthermore, the subsequent reports are expansive, even wordy. His listing of organs indicates that all have been considered. Consideration is given, whether there is suspicion of pathology or evident absence of pathology. This physician considers everything to be of clinical relevance. As a result, Dr. Robert provides a lengthy list of possibilities—when many referrers would prefer that he just cut to the chase. As a result, his diagnoses are not specific, even if they are present in the report.
At the same time, Dr. Robert has a reputation for never missing any pathology. His record includes recognition of the smallest—almost undetectable—lung cancers, the most subtle forms of thyroid cancer, and the tiniest leakages in the brain. There's an advantage to this approach: The list of possibilities is comprehensive. The approach does not lead to a specific diagnosis, however; this leads to a problem involving specificity. What may be perceived as a "false alarm" is sounded loudly—maybe too loudly.
Jha poses a very important question: Which radiologist—or radiologic approach—is of higher quality? And who proves more valuable: the decisive Dr. Singh or the cautious, methodical Dr. Robert?
Such are the questions the industry faces. And, to focus on the area that directs most attention in the current health care climate, which radiological approach would payers prefer? That question also applies to all health care principals, including hospital administrators, quality assurance and safety officials, risk management professionals, and, of course, patients.
"Why cannot we have the sensitivity and the caution of the methodical [Dr. Robert] coupled with the specificity of the efficient and immediately decisive Dr. Singh?" Jha asks.
According to Jha, "It is a known fact of signal theory that no matter how good one is, there is a trade-off between sensitivity and specificity. Meaning, if you want fewer false negatives, eg, fewer missed cancers on chest X-ray, there will be more false positives, ie, negative CT scans for questioned findings on chest X-ray. Trade-off is a fact of life."
Proceed With Caution
Radiologic imaging would seem to be a technology whose value is self-evident. However, there are many nuances that have arisen and need to be addressed.
"I would say that's a pretty accurate assessment," Robinson says. "There are a lot of nuances, and many of the forces that are driving health care are trying to promote standard practices. That's a good thing, but people are not automobiles. There are differences among people. The pendulum swings. For many years, we developed care pathways, standard practices, guidelines, and similar concepts. But we are starting to recognize that there is a lot of nuance in individual patients. That is one of the forces leading to what we call precision medicine, in which therapy is tailored to a particular patient's genetic makeup. Will that go anywhere? I am skeptical, but there may well be something there."
Meanwhile, Jha warns against the push toward developing quality metrics, as he feels it could lead to too much analysis and overintellectualizing. And that, he fears, could lead to paralysis.
— Dan Harvey is a freelance writer based in Wilmington, Delaware.
