Multiple Realities
By Claudia Stahl
Radiology Today
Vol. 24 No. 2 P. 10

Immersive technologies are changing medical practice and education.

Can augmented reality (AR) make the difference in whether a patient receives open surgical treatment for cancer or a minimally invasive procedure such as tumor ablation? Mina Fahim, president and CEO of Medi-View XR, says the technology could go a long way toward removing one obstacle—fear—from the decision-making process. Doctors tell him that they believe in the effectiveness of percutaneous procedures such as ablation, but sometimes “lack the confidence to stick a needle through the abdomen under flat, two-dimensional, black-and-white imaging to get to their targets without potentially damaging other structures along the way.”

MediView’s “X-ray vision” could help. Wearing Microsoft’s Holo- Lens headset, surgeons using the AR-based platform can “see” the anatomy below the skin via a 3D, holographic image created from a patient’s prior CT or MRI scans. An electromagnetic tracking system provides instrument guidance throughout the surgery, eliminating the need for physicians to divert their attention away from their hands—and the patient—to refer to scans on monitors.

Charles Martin III, MD, an interventional radiologist at the Cleveland Clinic, is part of a research team that has used the system for thermal ablations of abdominal and soft-tissue tumors. He says using an AR-guided approach in moving anatomical structures, like the liver, in contrast to “static” neurosurgical or orthopedic environments where it was used previously, is a turning point for medicine.

“It is just the beginning of where we can use [this technology] to help patients,” Martin says. He codeveloped the system at a research lab at the Cleveland Clinic in 2016. “It could have a tremendous impact as [medicine] becomes more minimally invasive.”

Market ‘Realities’
AR and virtual reality (VR) are immersive technologies with unique applications in medical practice and education. Both technologies use a headset, such as the Oculus, or a lens that attaches to eyeglasses or smartphones. AR superimposes computer generated images and experiences onto a person’s real-world environment. VR engages the wearer in a completely computer-simulated environment.

Mixed reality (MR) describes scenarios that integrate both VR and AR. For simplicity, this article will focus on AR and VR, even though many applications incorporate the two.

The health care sector is often slow to adopt new technologies for reasons such as FDA restrictions, patient privacy and safety concerns, and cost. Regardless, the market value of AR and VR in health care topped $2.5 billion in 2022, according to a report from Global Market Insights Inc.

The report attributes $1.5 billion of that total to tech-related hardware such as head-mounted devices and smart glasses, wearable displays, and 3D sensors. Software applications accounted for 35.5% of the AR/VR market in health care.

Academic institutions are another growing market (worth over $687 million) as medical colleges implement AR/VR technologies in their curricula. It’s a global trend. Last June, the Puducherry Institute of Medical Sciences implemented an automated VR laboratory (MediSimVR) as part of its surgical training program, the report says.

Practice ‘Realities’
The financial statistics support what researchers and long-time adopters of VR and AR in medicine identify as key areas for these technologies in health care: educating new trainees, patient education and communication, and presurgical planning and training.

To learn more, Radiology Today asked three experts, Martin; Raul Uppot, MD, an interventional radiologist at Massachusetts General Hospital (MGH) in Boston and cofounder of the Mass General AR/VR RAD lab; and Ali Dhanaliwala, MD, PhD, an abdominal imaging fellow with the University of Pennsylvania Health System in Philadelphia, how these technologies will impact medical practice and education in the months and years ahead.

Radiology Today: How are AR/VR/MR being used for minimally invasive surgery?
Martin: You want to do everything you can to avoid bigger, more invasive procedures in medicine. Being an interventional radiologist, I obviously am not able to open up a patient’s abdomen to ablate a tumor. [Technology] using augmented reality and holograms has allowed us to get that three-dimensionality into percutaneous procedures. It lessens the learning curve. One of my favorite things about using it … in training is watching that light bulb go off in people’s heads. They really understand it much faster.

RT: Are there any applications for patient education?
Martin: One of the more powerful experiences I’ve had was around using the technology to educate a patient about a tumor biopsy in her lungs. She and her husband are my patients, and both were long-term smokers. I showed them [the image] of her lungs using the HoloLens, and then normal, nonsmoking lungs. It brought tears to their eyes, and they haven’t picked up a cigarette since.

Uppot: For patients coming in for procedures, we use [VR] to get them immersed in the environment prior to them coming into the hospital. They know what the operating room or interventional suite looks like. We can use a 3D model to show them their own anatomy and the steps that we’re going to take to do the procedure.

RT: How does this technology change consultation and collaboration?
Martin: Consultation, I think that’s where [this technology] is going to be really powerful as an intervention. One of the more fun experiments we’ve done is to connect with colleagues at [other Cleveland Clinic campuses] during procedures. Right over their phones, they were able to see what I was seeing and communicate with me during the procedure. As the technology continues to mature, I think it will be a great way for us to help others, especially in resource strained areas.

Dhanaliwala: There are a lot of applications for addressing needs in developing regions. Here, Stephen Hunt is using the technology to collaborate with physicians in Botswana and Nigeria. We’ve just started thinking about ways we can provide them with training without having to fly back and forth. With the [HoloLens] headset ... we can talk to them and [oversee] simulations and real-life cases.

Uppot: [Typically] radiologists stand together in a dark reading room with four or five monitors in front of them. But some technology companies are developing virtual reading rooms where, just by putting put on a [VR/XR] headset, you can have access to 20 monitors and talk to people who are there, but not physically next to you, in a virtual space. It gives you the ability to consult with people in remote places.

RT: How does AR/VR/MR affect workflow?
Uppot: It used to be that a lot of presurgical planning was done by surgeons or procedure analysts looking at images on a 2D monitor or a 3D printout of an anatomical model. Now, we’re building digital models instead of an actual physical printout. With augmented reality, you can put a headset on and manipulate the images—turn them around, move them right and left, up and down. There’s less cost and it’s done much quicker.

Dhanaliwala: In … an environment like an IR suite, it can be difficult to pull up the [information] you want when you need it, even when it’s on monitors. You don’t want your sterile gown or gloves to be touching all these knobs as you manipulate images. While you’re wearing the [HoloLens], you see these images in windows in the headset. You can “grab” them [virtually] with your hands to move them and make them bigger or smaller. You can pull information from an electronic medical record or … cross sectional imaging CT. You can even make a call to a colleague, all without having to reach for a phone and scrub out.

RT: Can this technology be used to help train physicians?
Dhanaliwala: Pilots spend hours in high fidelity simulators where they [practice] scenarios they could encounter in real life. In medicine, we don’t have that. They just put you on the wards and hope that you get that experience, but that can be tough to do for rare events. VR is really helping us to develop those skill sets through simulation.

Uppot: There are companies now that have VR modules that can be used to not only teach but also evaluate performance. I foresee a future whereby trainees in procedural fields such as IR and surgery will be asked to go through simulated modules that will assess their technical skills. Based on their score, they will be allowed to proceed to the next step, [performing] a procedure on a patient, or they will be asked to remediate and go through the module … until they get a passing grade.

Dhanaliwala: We’re working to develop a library of simulations using a VR headset to test residents and fellows in scenarios such as “What happens if you give the wrong medication? What are the signs and symptoms of an adverse event?” On the back end, it can track things like “Did you call the code fast enough?”

RT: What are some of the advanced diagnostic capabilities?
Uppot: A company came to visit us in the MGH AR/VR lab [with a product] that uses AR to allow you to “walk through” images taken from a patient’s body. … You can actually walk around inside the patient’s brain, their liver, their heart. Right now, radiologists are used to looking at 2D models on a flat screen. But if you can use AR to go into a person’s brain and look around in more detail at specific nerve tracks and neural pathways … it could help make advancements in diagnosis.

Emerging ‘Realities’
The pandemic heightened the demand for virtual resources in medicine, as COVID restrictions limited training and educational opportunities for physicians of all skill levels. Out of necessity, it accelerated the uptake of telehealth capabilities and applications like Zoom and Teams for professional education and collaboration.

Some pivots have ushered in more innovative, flexible approaches to medical education, such as a remote anatomy course. “When other medical schools were closing down due to COVID, Case Western Reserve University mailed out HoloLens headsets to medical students and professors,” Uppot says. “[The assessments] showed that AR could be used to teach a subject as complex as anatomy from the privacy of people’s homes but, more surprising, was their finding that interaction among the students was better.”

As of January 2023, the FDA lists 39 AR/VR devices on its website that have been authorized for marketing. Dhanaliwala believes the demand for new AR/ VR based products in medicine will only grow as more clinicians discover their potential for improving education and workflow.

“The [HoloLens] headset is just like a computer,” he says. “You wouldn’t use a computer only for something like playing solitaire, right? It’s a matter of figuring out how its applications, like the internet, or spreadsheets, can help you. That’s where we’re going with this technology.”

— Claudia Stahl is a freelance writer based in Ambler, Pennsylvania. She specializes in writing about the health of people and the planet.