The Gorilla in the Room
By Beth W. Orenstein
Radiology Today
Vol. 22 No. 3 P. 22

Studies show why some radiologists may turn a blind eye to unexpected abnormalities.

In Simon and Garfunkel’s 1969 hit “The Boxer,” the duo sings: “Still, a man hears what he wants to hear/And disregards the rest.” That song and those words can be applied to a study from researchers at the University of Utah that found even experienced radiologists, when looking for one abnormality, can completely miss another. In their paper, which was published in Psychonomic Bulletin & Review last fall, the researchers describe this phenomenon as “inattentional blindness.”

Led by associate professor Trafton Drew, PhD, the recent study is a follow-up to work Drew did at Harvard. In 2013, Drew and colleagues published a study on the same topic, “The Invisible Gorilla Strikes Again: Sustained Inattentional Blindness in Expert Observers,” in Psychological Science. In that study, the researchers asked 24 radiologists to look for lung nodules in five chest CTs that had an average of 10 nodules. In the last case they presented, the researchers inserted a gorilla that was 48 times the size of the average nodule. Most (83%) of the radiologists did not notice the gorilla. The researchers blamed the misses largely on “inherent limitations of human attention and perception.”

In the follow-up study published last fall, conducted with colleagues from Utah, UCLA, and Macquarie University in Sydney, Australia, the researchers looked at whether the results would be different if, instead of a gorilla, they placed an abnormality that plausibly could come up on a chest CT. They asked 50 radiologists to look at the CT scans. The final scan the radiologists were asked to read included two clear abnormalities: a significant breast mass and abnormal lymph nodes. The results were similar: Two-thirds of the radiologists did not notice the potentially cancerous mass, and one-third did not notice the lymphadenopathy. In a second experiment, radiologists were asked to report on a broader range of abnormalities. This time, only 3% missed the mass, and only 10% missed the lymphadenopathy.

Lauren H. Williams, PhD, the lead author of the follow-up paper, who was at Utah but is now a postdoctoral scholar at the University of California, San Diego, says the research shows that focusing narrowly on one task may cause radiologists to miss unexpected abnormalities, even if those abnormalities are critical for patient outcomes. “Inattentional blindness occurs when someone misses something that is retrospectively obvious because they’re focused on another task,” Williams says. “For example, if you’re focused on finding your keys before leaving the house, you might completely miss your wallet lying on the table, even if you look directly at it.”

Williams and her colleagues asked radiologists to focus on detecting lung nodules, which led to the high miss rates for a large breast mass and lymphadenopathy. “In a follow-up study, these same abnormalities were easily detected when radiologists were asked to look for them directly,” she says. The results suggest that, “if we want to better understand why retrospectively visible abnormalities are sometimes missed in radiology, we need to better understand the situation that led to those errors. Was the finding consistent with the exam indication and clinical history? What was the radiologist looking for when they missed the abnormality?”

No Surprises
The findings of these studies come as no surprise, says Michael A. Bruno, MD, MS, FACR, a professor of radiology and medicine, vice chair for quality and patient safety, and chief of the division of emergency radiology at Penn State Milton S. Hershey Medical Center. Bruno is also author of Error and Uncertainty in Diagnostic Radiology, published in 2019 by Oxford Press. “There is indeed an increased risk of errors when the ‘target’ is not in the expected location, like the breast cancer at the edge of a lung image from the new Drew/Utah paper,” Bruno says.

Errors also have been reported when radiologists see something but don’t know what it is or think that it’s something it’s not, Bruno says. “Those errors are relatively rare, even among rare errors,” he says. “By far, the main mistake we radiologists make is simply not seeing things on images—often things that are quite conspicuous in retrospect—and sometimes things that are very important and that we may have even been primed to be looking for.”

Eliot Siegel, MD, a professor of radiology at the University of Maryland School of Medicine and chief of imaging services at the VA Maryland Health Care System in Baltimore, agrees that “inattentional blindness” is real. Inattentional bias is but one of several related biases that radiologists bring to the reading room, Siegel says. There is such a thing as confirmational bias or “satisfaction of search,” he says. “That’s where you come up with an idea of what the patient has and evidence to support it. Once you do, you don’t keep looking for any other evidence than to support your original hypothesis.” In that case, Siegel says, “you could find a lung tumor in the patient’s right upper lobe and use it to explain his chest pain and completely miss the perforated ulcer in the abdomen, which is a true emergency.”

To Bruno’s way of thinking, the “satisfaction of search” error is not a cognitive bias but is, instead, a subset of perception errors. While the so-called “satisfaction of search” error is known to occur commonly, it doesn’t mean that radiologists are at all “satisfied” when they find the first abnormality on a radiological exam; eye-tracking studies have shown that they really do keep searching, he says. “So, what keeps them from finding abnormalities that we know they looked directly at? Maybe,” Bruno says, “it’s a form of attentional capture.”

Howard Egeth, PhD, a professor of psychological and brain sciences at Johns Hopkins University, has done some experiments on this problem and has shown that human visual attention can be an “all or nothing” phenomenon, according to Bruno. “That actually explains ‘satisfaction of search’ better, in my opinion.”

The Utah researchers also used eye-tracking tools. “And we didn’t see a relationship between how thoroughly the radiologists searched the images and whether or not they missed the unexpected abnormalities,” Williams says. “We also didn’t see a relationship between detection rate for the lung nodules and detection rate for the secondary abnormalities.”

Nor did the researchers find that experience affected the results. “Radiologists with a wide range of experience participated in our study, which allowed us to examine whether or not expertise provides any protection against inattentional blindness,” Williams says. One would expect a highly experienced radiologist to detect the unexpected abnormalities more readily than a trainee. “But we did not find a relationship between the level of experience and susceptibility to inattentional blindness,” she says. Indeed, she notes that radiologists are often surprised to learn that highly experienced radiologists were just as likely to miss the unexpected abnormalities as trainees.

Advantage: Trainees
Williams believes that experience not affecting outcomes is an important finding because it demonstrates that inattentional blindness “is an inherent limitation of our cognitive system,” she says. “If even highly trained experts miss retrospectively visible abnormalities when attention is narrowly focused on another type of abnormality, it suggests the information given to radiologists plays a crucial role in determining which abnormalities they will ultimately detect.”

Siegel believes that trainees may even have an advantage here because they may not be sure about the importance of a particular finding and report on it, without the filter of experience regarding what is or isn’t clinically relevant. More experienced radiologists, he says, “might know in which types of patients or clinical presentations one would want to look for certain types of things and, as a result, they’re more likely to see patterns of findings that have implications for patient prognosis or diagnosis, paying less attention to other details.”

One can learn from trainees, Siegel says. “For me it’s wonderful to teach residents and fellows. In a significant percentage of cases, they will add value by making observations that I had not noticed or been concerned about. They will ask a question about something based on what they observed,” and the answer requires looking at the study from a different perspective, sometimes resulting in a clinically relevant insight, he says.

Could AI or computer-aided diagnosis (CAD) play a role and help prevent inattentional blindness when reading studies? While it’s true that AI does not have the same cognitive limitations that humans do, AI is only as good as its training and programming. “AI and CAD will only do what they are designed to do,” Williams says. “If an algorithm is trained to detect a specific type of abnormality rather than a broad range of abnormalities, it will ultimately have the same limitations as a human reader.”

Bruno believes that, while AI hasn’t helped solve this problem much to date, it is getting better and “eventually will help.” Having a second reader has proven to greatly help accuracy, Bruno says. But, with the volume of scans on the rise and the number of radiologists available to read declining, second human reads aren’t as likely to be feasible, he adds. “In the future, AI could be that second reader, once it gets good enough. It’s just not there yet.” 

Time Not a Factor
“Luckily,” Bruno says, “these ‘perception’ errors, though the most common ones, are still not all that prevalent in real life.” Thankfully, he says, radiologists get it right about 97% of the time, and have had pretty much the same low error rate (3% to 4%) since L. Henry Garland, MD, published his pioneering article on diagnostic errors in the American Journal of Roentgenology in the 1940s. “And this is despite the fact that almost everything about radiology has changed since then,” Bruno says.

Bruno says the stubborn, unchanging error rate suggests that “we are dealing with a very fundamental, neurobiological mechanism. Our human visual system, including eyes and brain, evolved over millions of years before radiology was ever invented.” Perhaps, he says, it should not come as a surprise that man’s visual system is not optimized for radiology. On the other hand, “radiologists’ training and experience is quite intense, and their expertise is quite impressive, really.”

Williams says it’s important to note that the Utah study “should in no way reflect poorly on radiologists. The critical point here is that the phenomenon of inattentional blindness is experienced by everyone, and we need to better understand the situations that lead to these misses in order to avoid them.”

Two sets of follow-up studies to the research are needed, Williams says. One is to better understand the circumstances that lead to high rates of inattentional blindness. For example, she says, unexpected abnormalities located in a different anatomical region than the focus of the search may be more difficult to detect than incidental findings in the same region. Also, she says, it’s important to determine “how we can provide clinical information to radiologists in a way that helps them focus on what is relevant, without resulting in high rates of inattentional blindness.”

Radiologists face a number of challenges, including sorting through the vast amount of information available to them, Williams says. For example, the growing reliance on cross-sectional imaging easily translates into thousands of images per day that the radiologist is confronted with. When radiologists have access to accurate clinical information and specific exam indications, it can help narrow their focus and make their task more manageable, she says. The original gorilla study demonstrated that focusing on a specific task can lead to significant costs, even when the unexpected findings are highly unusual. However, Williams says, radiologists don’t actually need to keep an eye out for gorillas in their clinical practices. The researchers decided to pursue the follow-up study because “it was important to demonstrate that the inattentional blindness phenomenon applies to more realistic circumstances, as well.”

Solving the problem that the original gorilla study and its follow-up identified isn’t straightforward, Williams says, but there are a number of possibilities. Accurate clinical information and specific exam indications can help radiologists identify abnormalities that are consistent with their expectations. However, “our research demonstrates that having a narrow focus also can cause clinically significant abnormalities to be missed.”

To balance these two concerns, Williams suggests that it could be beneficial to perform a general assessment of an exam, followed by a narrow search. Alternatively, she says, it may be helpful to use checklists or structured reports to ensure that nothing is overlooked. If CAD systems are trained to detect a broader range of abnormalities, she says, they might be able to help as well.

— Beth W. Orenstein of Northampton, Pennsylvania, is a freelance medical writer and regular contributor to Radiology Today.