Imaging Informatics: Seeking Approval
By Beth W. Orenstein
Radiology Today
Vol. 25 No. 3 P. 28

Radiologists should look beyond just FDA approval when evaluating AI/ML products.

Last October, the FDA published a list of the AI-enabled medical devices it authorized in 2023. More than 170 medical devices were added to the list last year. That’s a 33% increase in one year. With the additions, the FDA has approved 692 devices. Of them, 77% (531 devices) are in radiology. The highest number of submissions are from radiology, as well. In 2022, 75% of the FDA-approved AI/machine learning (ML) devices were in radiology.

By 2025, it’s estimated that more than 1,000 AI products will be FDA cleared, “and most of them are going to be in the radiology market,” says Adam Flanders, MD, a professor of radiology in the division of neuroradiology/ENT at Thomas Jefferson University in Philadelphia. Flanders sees the specialty’s interest in AI products as understandable. AI is seen as a useful tool set to help mitigate against the pressing dual issues of increasing demand for imaging studies and a global shortage of radiologists, Flanders says. “There are a myriad of potential applications of AI in our specialty, and we are just beginning to scratch the surface where they can positively impact medical imaging,” Flanders says.

Flanders and George Shih, MD, a member of the abdominal imaging division in Weill Cornell’s department of radiology, were asked to comment on a recent article that compared the concordance between marketing materials put out by FDA-cleared AI companies and the submission documentation to the FDA. In this article, the authors compared the marketing materials with the FDA submission documentation for 119 AI/ML devices that the FDA cleared between November 2021 and March 2022.

Of those, a total of 75 (82%) were radiology devices and 23 (19%) were cardiovascular devices. The authors reviewed a subset of cleared devices because not all of the FDA materials were publicly available. The authors compared the approval summaries and accompanying marketing materials and categorized each as 1) adherent, 2) contentious, or 3) discrepant. They categorized the product as “adherent” if they found that AI/ML use in marketing materials and FDA summary were similar. “Contentious” meant they found nebulous language in marketing materials implying AI/ML capabilities without the use of specific AI/ ML terms employed in the FDA summary. “Discrepant” meant that they found AI/ML language in marketing materials describing a device that was not FDA cleared or approved as an AI/ML device. Most of the documentation evaluated in this study, 96 (84%), were labeled as adherent. However, a total of 15 (12.61%) were labeled as discrepant and eight (6.72%) as contentious. Radiology and cardiovascular devices showed significantly different distributions for the three categories, Flanders says.

A commonly held fallacy is that FDA approval or clearance of an AI/ML device implies government endorsement of the product’s effectiveness and reliability, Flanders says. However, the FDA does not perform independent testing and relies principally upon data generated and submitted by the vendor. In their editorial, the authors say their intention is not to imply that any class of device is unsafe or ineffective, but rather that there is much room for improvement “by insisting that marketing materials closely coincide with the language used to gain FDA acceptance in bringing the product to market.” The authors recommend that going forward, the FDA use a “highly consistent and structured format for the application process, including controlled vocabulary for specific AI/ML-related devices.”

Flanders says evaluation of marketing materials alone is not an especially useful way to make purchasing decisions, as there are a number of factors to consider when determining whether any AI/ML products would benefit a practice and augment patient care. Besides, he says, most informed consumers would rather trial an AI solution first to determine whether to make a long-term commitment.

Be Scrupulous
Flanders doesn’t believe that vendors intentionally make outlandish claims in their marketing materials. However, he says, the FDA does advise what the marketing materials are allowed to include or exclude.

When thinking about purchasing an AI/ML device, practices must consider whether it will improve patient care and the efficiency and accuracy of the radiologists using it. Flanders believes that when making AI/ML purchasing decisions, radiologists should “spend time interrogating the sales force and talking to other buyers who have experience with it to find out how well it worked or hasn’t worked for them, in addition to trialing the solution in their own environment.” Everyone knows, he says, that a solution that has performed well at one site “may not realize the same level of performance at another.” Although the cost of AI is not nearly what it would be for a major piece of imaging equipment such as a CT or MRI scanner, due diligence is still important, Flanders says.

It’s clear, Flanders and Shih note, that the FDA is working hard to supplement their policies and guidelines around SaMD (software as a medical device). “We expect that these policies around AI/ML will continue to improve with time,” Flanders says.

— Beth W. Orenstein of Northampton, Pennsylvania, is a freelance medical writer and regular contributor to Radiology Today.