Going to Market
By Kathy Hardy
Radiology Today
Vol. 20 No. 11 P. 16
App marketplaces are bridging the gap between AI creators and users.
Words such as “democratize” and “ecosystem” are making their way from government and biology textbooks into the radiology lexicon, as a community of developers, engineers, and clinicians are interacting to establish AI marketplaces. These electronic storefronts are gathering places where vendors and consumers can share ideas and technology to create and use a variety of AI imaging tools.
Within an AI marketplace, radiologists have equal opportunity to pick and choose from any of the available apps. They can also share feedback with developers as to how an app worked for them or even assist in the development of new apps customized for their clinical practices. And, they can be active participants in the development process without any knowledge of writing code.
“With AI, there’s a whole ecosystem of participants—industry leaders, health care startups, and research institutions—getting involved in creating apps and marketplaces where they can be made available to everyone,” says Abdul Hamid Halabi, director of healthcare with NVIDIA. “There are thousands of AI apps being created by more than 600 startups working in AI today, plus thousands of research facilities and hospitals considering creating their own AI apps. We want to enable AI for all, and the marketplace can help with that.”
NVIDIA’s Clara AI toolkit, part of the Clara developer platform, consists of libraries for data and image processing, data annotation, AI model training, and image visualization. Halabi explains that NVIDIA sits at the “computational” level of an AI marketplace. Directly above that is the AI engine, where algorithms are built. On top is the “storefront,” where apps are displayed for use.
“Clara uses GPUs [graphics processing units] to create images as efficiently as possible, to satisfy the needs of the marketplace,” he says. “With AI, we’re working with large-size files, such as those generated by MRI or 3D imaging. You need GPUs to handle these large files, but it doesn’t make sense for algorithm builders to have the GPU. That’s where we come in, to make sure algorithm and app development occurs as smoothly as possible.”
Not Quite the App Store
One of NVIDIA’s AI partners is Nuance, which launched its AI Marketplace for Diagnostic Imaging, a storefront of FDA-cleared AI apps, in 2017. Connecting AI developers directly with subscribers, Nuance’s marketplace offers AI developers a single application programming interface to connect algorithm contents directly into radiologists’ workflow using Nuance’s PowerScribe reporting software and the radiologists’ PACS.
“NVIDIA is an important partnership,” says Karen Holzberger, senior vice president and general manager of diagnostics for Nuance Communications. “To have a robust, two-sided network at scale, you need to connect the developer community with the users that will benefit from the AI applications within the marketplace. It’s a natural fit—NVIDIA is connected to the developers, and Nuance is connected to the providers. NVIDIA works with AI developers so this helps us build up a robust AI development community.”
Woojin Kim, MD, chief medical information officer at the healthcare division of Nuance Communications, says that it can be challenging to put AI technology into everyday use in health care, due to complexities in development, validation, and regulatory issues. An AI marketplace, with vendors of different skill sets, can help streamline adoption by giving physicians and hospitals one-stop access to a range of AI models, creating a channel for feedback between developers and users and providing validation data for FDA postmarket surveillance.
According to Kim, by helping to aggregate the increasing number of AI tools, an AI marketplace helps not only potential users but also developers with limited sales budgets sort through new AI offerings.
“There are a lot of AI start-ups, many with a tremendous investment and interest in radiology,” he says. “One of the main challenges is deploying and bringing various AI apps into practice. The marketplace is a way to facilitate AI adoption in the clinical setting, by bringing together AI companies and consumers. AI models can be submitted to one place, and users can see all the models available in one place.”
Kim notes that comparisons are often made between AI marketplaces and smartphone app stores. He finds similarities in concept only, as there are many challenges with AI apps in medical imaging, such as brittleness of AI models, validation, and importance of integration into the workflow. In addition, current AI applications have a narrow focus.
“An AI app deals with one finding from one modality, like lung nodules on a chest CT, for example,” he says. “There are hundreds, if not thousands, of other possible findings in a chest CT, beyond just nodules, which means that each finding would require its own algorithm. Today, this would lead to thousands of algorithms.”
In addition, app users can provide feedback via Nuance’s PowerScribe reporting platform without interrupting their workflow.
“Through the Nuance AI Marketplace, users’ interactions with various AI models can be shared with the AI companies, providing valuable feedback for ongoing improvement of their models,” Kim says. “In health care, this is especially important, because AI models can be very brittle, meaning that an AI model that works well at one site might not work well at another site. Feedback tells developers how AI models are being used, which is a critical component of validation and further improvement.”
Another Approach
Blackford Analysis is taking a slightly different approach with its AI app marketplace. The health care software company offers its customers the Blackford Curated Marketplace of apps that are tested, vetted, and cleared for clinical use, says CEO Ben Panter. Blackford focuses exclusively on the platform/marketplace approach, as opposed to other companies in the space that are expanding their services by adding marketplace curation. The company’s marketplace of 16 AI products, developed by third-party vendors, does not include a product development or experimental component, although its underlying platform is used by some facilities for that purpose.
“Rather than being a clearinghouse for all AI development, we produce finished products for use by our customers,” Panter says. “There is a curation and vetting process that takes place before an AI product makes it to our marketplace.”
With Blackford, users adopt their platform, which supports multiple apps, all available in their marketplace. Radiology practices and hospitals don’t need to purchase individual apps from different vendors. Panter says Blackford is in the process of onboarding 10 more AI development companies to its marketplace. New companies are welcomed into the marketplace, once their work has been vetted and there is a need for that offering.
“If we see a fit between a product value proposition and a customer need, we add that product to our marketplace,” he says. “There is a cost to us of onboarding but, if it fits the needs of our customers, we will bring it onboard.”
Panter says Blackford’s platform approach started with a clinical application that lined up current and prior images. This was a more efficient way for clinicians to compare areas of concern, diagnose conditions, and track treatment progress. The challenges, however, were in creating the infrastructure to add this to the PACS framework and in creating a process to get data back to the radiologists. There’s also the matter of cost, as the price of the product is often only a small part of the overall cost to integrate a new product into a large radiology IT system.
“We built our platform to ensure that all aspects of product selection, deployment, evaluation, and support were covered by a single vendor and contract, allowing new products to be deployed rapidly and economically,” Panter says. “This allows for cost-efficient deployment of apps for developers and users.”
Standard Fare
Not all participants in the AI marketplace arena deal in apps. With a focus on physician workflow, Palo Alto, California–based Fovia has a technology called XStream aiPlatform, which connects AI developers, PACS, and universal viewer companies to radiologists and physicians. Fovia’s chief technology officer, Kevin Kreeger, PhD, says the company is not developing AI algorithms but creating a streamlined way to visualize and deliver AI results to radiologists and other physicians. It provides the means to display the AI-generated information within the overall workflow in standard DICOM objects or custom objects.
Assistance with visualization is key, he says, as efficient navigation and meaningful interactions can lead to greater acceptance of AI tools.
“When AI is first used, radiologists and physicians need to understand what the tool is showing,” Kreeger says. “How will the doctor interact with all the information AI tools provide? How will they understand it? Will they quickly learn how to understand what they’re seeing? These are all important questions that we help answer.”
Fovia is also positioned to collect feedback from users and offer it to AI app developers. Kreeger sees this process helping immediately and in the future.
“We can help work toward presenting a more standardized image in the future, but also help make objects clearer for users now,” he says. “As marketplaces grow, there will be more data to work with, leading to more apps and a greater need for standardized visualization.”
Kim says Nuance has been working closely with the ACR, RSNA, and other societies, particularly in the area of standardization.
“There are so many AI companies involved, and they’re working on great things for radiology,” he says. “Often, they’re looking at the same things but describing them in different ways.”
Halabi says standardization would enhance the creation, deployment, and utilization of apps in workflows.
“It would help reduce the number of apps that are specific to just one user,” Halabi says.
Focus on Workflow
As developers look to expand AI as a tool in radiology, they rely on available data to create the algorithms to build the apps. For example, many in the field point out that there is a wealth of openly available data on chest lesion diagnosis. Hence, there are a significant number of chest lesion apps and algorithms available.
“The people with the data are dictating the direction of the algorithms,” Halabi says. “There are public data sets available. Hospitals making data available immediately influences which algorithms are built. Whether or not that continues is a matter of the tools that are available now.”
To create a more balanced AI marketplace, Panter suggests that developers focus on speaking with clinicians and finding out their needs for better diagnostic imaging tools.
“You need to find a real clinical problem then develop a product to meet that need,” he says. “There’s no use in having 100 apps if 90 of them are useless. They may all have the potential to be good products, but there needs to be more focus on workflow and utility. We need to build products with clinical focus that will provide value.”
Kim believes the marketplace will help to shine a light on use cases lacking in AI app support.
“Having the marketplace will further promote the need for more AI apps,” he says. “As we further validate available products, the marketplace will start to populate with a wider variety of apps. Also, societies like the ACR are looking at other areas, like noninterpretive use cases, which will result in an even greater number and type of apps in the marketplace.”
Interoperability will be key down the line, Halabi says. Radiologists will want to use AI apps in their scanners, in the emergency department, and throughout their clinical practices.
Panter says that, over time, there will be more AI apps with a strong clinical focus. As that happens, radiologists will begin to find AI tools essential to providing the quality of care they wish to offer their patients.
“I see AI apps over the next five years focused on productivity for radiologists, enhancing value to referrers, and bringing greater volume for practices,” he says. “One of the most promising areas of AI takes high-risk procedures from other areas of the hospital and brings them into the imaging suite, improving patient safety and reducing cost.”
— Kathy Hardy is a freelance writer based in Phoenixville, Pennsylvania. She is a frequent contributor to Radiology Today.
[Sidebar]
‘Lab’ Work Paying Off
Radiology societies are helping vendors find space for collaborative experimentation in the marketplace. For example, Karen Holzberger, senior vice president and general manager of diagnostics for Nuance Communications sees positive results from Nuance’s work with the ACR and its Data Science Institute (DSI). ACR’s AI-LAB, a free software platform to build, share, locally adapt, and validate AI algorithms for radiology, began in June 2019 with a number of pilot programs. Multiple institutions and developers are participating. One of the goals of AI-LAB is to facilitate development of robust AI algorithms while ensuring that patient data used to train the algorithms stay protected at the local institutions.
“ACR and the AI-LAB are doing many things that are helping to get AI to scale,” she says. “That includes addressing the practical elements of moving AI from the lab and into everyday clinical use. Developers and clinicians understand the potential benefits for the technology to improve health care outcomes, so there’s a lot of energy and momentum behind making that happen now. It’s impressive to see not only the range of expertise coming together from across the AI and medical communities but also how grounded and focused the effort is.”
According to Bibb Allen Jr, MD, FACR, CMO of DSI, the AI-LAB is all about collaboration.
“The AI-LAB is designed to be totally vendor neutral. It serves as a data aggregator, provides a place for annotation, and allows users to run algorithms built on data sets from their own cases,” he says. “Once algorithms are built, they sit on the server at your institution. The algorithms can be pushed from the AI-LAB to the local clinical workflow through an AI marketplace, or transferred to other institutions for collaboration with others to add more diversity to the training data, making the algorithm more robust.”
Allen says that when the ACR created DSI in 2017, the initial focus was what radiologists could do to encourage the building of AI tools while ensuring patient safety. From that effort came the idea for a place where radiologists could create, use, and publish AI apps, either working with AI developers or on their own.
Enhanced Tools
Through the vendor-neutral AI-LAB, radiologists have access to user-friendly computational tools that will help them learn about annotating data sets and training AI models as well as sample the AI tools that can be used to train and modify existing AI algorithms. There are modules for building apps without writing code.
“The AI-LAB puts Python [a high-level programming language] behind the scenes and uses a user-friendly GUI [graphical user interface] layer that allows radiologists to build AI apps,” Allen says. “Our goal with the AI-LAB is to create a tool that brings AI to radiologists without them having to learn coding or move their patient data off premises.”
Along with the many collaborative efforts within the AI-LAB, Allen stresses the importance of capturing performance data and ensuring the “patient safety” aspect of any new apps.
“For the FDA to assess AI, we need to monitor the performance of the algorithms in a clinical setting,” he says.” We can do that through the ACR’s AI registry, Assess-AI, and our involvement with AI marketplaces. We can look for common circumstances between users that might cause an algorithm to fail and take information back to the developers to then retrain the algorithms.”
In addition to working with Nuance, ACR has also integrated NVIDIA’s Clara AI toolkit into the AI-LAB. According to Halabi, ACR expressed interest to NVIDIA in incorporating the Clara AI because some of its members were already involved in annotation and were looking for a greater role in working with algorithms and app creation.
“Clara gives radiologists the ability to innovate and create apps that meet their clinical needs,” Halabi says. “Radiologists want to help create AI themselves or use what they learn to validate existing apps.”
Secure Information Sharing
The ACR AI-LAB also provides a platform for training algorithms across multi-institution patient data without sensitive patient data leaving the home institution. Traditionally, Allen explains, it can take months to gather all the institutional approvals required to transfer patient data off site for developers to train the algorithms and build useful apps. The AI-LAB offers an opportunity to train an algorithm at multiple sites while the patient data stay at their respective facilities.
“With AI-LAB, you can build the algorithm then take that algorithm from place to place for more training to improve the algorithm and make it more generalizable to widespread clinical use,” Allen says.
Halabi also recognizes the need for shared data in AI app development. In Nuance’s training mode, he says, users can start app building from scratch, collecting their own data to train the algorithms and build the apps. Users can also train on an existing algorithm. In addition, Clara offers federated learning, which is key when it comes to sharing data among various locations.
“There is a problem with hospitals not being able to share data,” Halabi explains. “The hospital owns the hard disk on which the data are stored. Training can take place on site, but you can’t add to the data. Our model goes to the data at all locations. With the ability to collect more data, this helps to build more robust apps.”
— KH
