AI Insights: Model Behavior
By Keith Loria
Radiology Today
Vol. 21 No. 9 P. 7
Researchers believe AI analysis of chest CT is a valuable tool for clinical management of COVID-19.
Nvidia and the National Institutes of Health (NIH) are collaborating on new AI models that will assist researchers in studying the signs of COVID-19 on chest CT scans, with the goal of developing new tools to better understand, measure, and detect infections.
“In February, we started hearing more and more about COVID and knew other companies were using AI to look at CTs,” says Mona Flores, MD, Nvidia’s global head of medical AI. “Then a study from Wuhan came out showing they were able to use CT scan for diagnosis, so we started thinking about how we could help and come up with something useful.”
The study was conducted at Renmin Hospital and revealed that physicians were using CT scans for diagnosis because it was taking too long to get tests back. In these cases, CT scans showed great value in screening and detecting patients with COVID-19 pneumonia, especially in highly suspicious, asymptomatic cases with negative nucleic acid testing.
“We figured if we are able to replicate that, we could get answers that would be useful,” Flores says. “We reached out to NIH, and they were interested in exploring this with us.”
The reason chest CT is useful, Flores notes, is because it can provide an answer in seconds. “You can imagine how helpful that would be in a triage situation, when you’re waiting to confirm the tests by other means and isolate those cases that have a high suspicion of COVID,” she says. “Another way you can look at this is the downstream applications and evidence that can be built on top of this in order to follow cases.”
For example, when a patient is admitted for COVID, clinicians want to follow them over time and see what else is happening. Sometimes, they can see clinically important details earlier on a CT scan than they otherwise could.
“So, you can use CT as a prognostication device in order to anticipate whether a patient is going to get worse, or if you’re going to need to start them on a certain med,” she says. “So, you can use it for patient management, or you can also use it for triage.”
For the research, Nvidia and the NIH utilized data taken from 2,724 scans from 2,619 patients, including 1,029 scans of 922 patients with reverse transcription polymerase chain reaction–confirmed COVID-19 and lung lesions related to COVID-19 pneumonia. The scans were collected from four hospitals across China, Italy, and Japan.
Flores explains that two models were used to create a final classification model. The first offered a segmentation model that was used to define the lung regions that were subsequently used by the classification model. Training converged at highest validation accuracy of 92.4% and 91.7% for hybrid 3D and full 3D classification models, respectively, for the task of determining COVID-19 vs other conditions. The validation accuracy in an unseen independent test set was observed with the 3D classification model (93.9%), with the classification model of COVID-19 disease achieving 0.941 AUC.
Untapped Potential
The models distinguished between COVID-19 and other types of pneumonia, proving the hypothesis that AI could be a strong element of CT-enhanced diagnosis. “The onus is not on us; I think the proof is in the pudding and, as people use this and see the results, they will see how generalizable this model is,” Flores says. “We have given this model to several places to give people access to it, to use it for research, and to just try it out.”
To that extent, people can take the model, run clinical trials on it, and judge for themselves how well it works. Flores believes subsequent models could include resource allocation, point of care detection for isolation of asymptomatic patients, prognostication, or monitoring for response in clinical trials for medical countermeasures, such as antiviral drugs or monoclonal antibodies.
Flores notes these clinical trials can also serve as a point of reference for other models but emphasizes that this is not an FDA-cleared model or something that can clinically be used off the shelf at this point. “This is not the end-all of all models for CT classification of AI,” she says. “It’s a stepping stone to try other populations, perhaps add more data to it, and fine tune it to generate a higher level of accuracy in a specific population. It would be nice for people to try it out, improve on it, and give it back to the public to use.”
There are some who don’t believe chest CT is effective for COVID diagnosis; however, most research is proving otherwise. “I think it’s situational,” Flores says. “If you have a place that you can run tests and get results within five minutes, then you probably don’t need CT for the diagnostic part, though you will still need it to follow the patient over time and see whether they are improving or not. But there are many places that do not have good testing today. As we saw in Wuhan, it was easier to get a CT than it was to get a test result.”
Because controlling the virus depends on isolating patients who may be asymptomatic, CT findings can vastly improve efforts to stop its spread. “I don’t see any negative impact of doing this; it’s just another tool in the arsenal to be able to fight the disease and stop the spread of infection,” Flores says.
Eventually, she sees people adding more use cases, doing clinical trials, and getting FDA clearance. She also expects AI experiments to continue helping with COVID in the months ahead.
— Keith Loria is a freelance writer based in Oakton, Virginia. He is a frequent contributor to Radiology Today.
