Second Sight
By Beth W. Orenstein
Radiology Today
Vol. 21 No. 3 P. 12

Augmented reality opens new windows for investigation, education, and surgical planning.

Every year, about 3 million Americans fracture their ribs—trauma is the most likely cause. Cancers that spread to the bone also can weaken ribs and cause them to fracture. Fractured ribs will often heal on their own in about six weeks with rest and supportive therapy. However, patients who have severe or multiple displaced rib fractures may require a surgery known as rib plating.

During the operation, performed under general anesthesia, a surgeon makes incisions over the broken ribs and inserts titanium plates and screws to stabilize them. One of the keys to a successful operation is making the incision directly over the ribs that are affected, says Babak Sarani, MD, FACS, FCCM, director of the Center for Trauma and Critical Care at the George Washington University Hospital in Washington, D.C. Current protocol uses a CT scan to determine where to make the incision. The CT is taken when the patient is supine, although the procedure is often done with the patient lying on his or her side.

“With the CT alone, we have found that we can be off by 3 or 4 cm,” Sarani says. “While that doesn’t sound like a lot, when you’re trying to operate, this distance feels more like a mile. It makes for longer incisions and longer operating time.”

On the Mark
When Sarani heard about technology that overlays a 3D image of a patient’s body onto the patient to enhance surgical accuracy, he was anxious to try it. The technology is Novarad’s OpenSight Augmented Reality (AR) System powered by Microsoft HoloLens. OpenSight received 510(k) clearance for presurgical planning from the FDA in December 2018. Cocreated by Wendell Gibby, MD, Novarad’s CEO, and Steve Cvetko, PhD, the company’s director of research and development, the technology is designed to visualize 3D holograms of the patient’s insides directly on the patient. The 3D holograms are created by a refractory system in the device using a combination of the Microsoft HoloLens hardware and OpenSight software.

“Until now,” Cvetko says, “the medical industry seemed to separate images from the patients. You look at the patient’s scans on computer screens, and every time you look at the images you’re looking directly away from the patient.” He says the 3D holograms OpenSight creates and displays directly on patients are true to size so “it’s as if you’re looking through the patient.”

Last year, Sarani conducted a pilot clinical trial comparing the first version of OpenSight’s HoloLens holograms projected on patients with CT scans, when performing rib plating. “We used the Novarad OpenSight system to mark where its augmented reality tells us the fracture is and compared it to where we would have opened the patient based on a CT scan alone,” Sarani says. “We wanted to see what the difference was between these two markings.”

Although they were not sure what to expect, they found significant differences. “The margin of error for the Novarad product was 5 mm. The margin of error using the CT scan without the OpenSight system was 4 cm. That’s a huge difference—nearly 10-fold,” Sarani says.

He did not time the cases he included in his trial, so he cannot compare their duration with conventional operations. However, he suspects that the OpenSight system’s accuracy decreased the time it took to complete the procedures. “OpenSight absolutely increases the accuracy of your operation and the ease of your operation. That’s all true,” he says. “I suspect that those advantages translate into less operative time, but I didn’t time myself, so I can’t say.”

New Horizons
To use OpenSight, a provider dons a headset that looks much like virtual reality (VR) goggles but, rather than darkened lenses, the goggles are clear, allowing augmented images to be intertwined with reality. Additionally, AR does not have any image disorientation, a notable aspect of VR, which makes its best suited for enhancing productivity in health care, Cvetko says.

“It sounds like something out of Star Trek,” Sarani says. He did not find the headset to be cumbersome when performing the surgery. “As soon as I put it on, it was a matter of me learning the commands to line up the hologram with the patient. It took a good 10 to 15 minutes because I had to do it manually.”

Since Sarani completed his trial, OpenSight has released an updated version of the HoloLens that automates the alignment of the hologram with the patient. Cvetko says one of the biggest challenges in developing the OpenSight technology was registering the images on the patient. “We needed to figure out how to make sure the 3D images are lined up correctly and that it was as accurate as possible,” he says.

The new version uses a QR (quick response) code system that is coordinated with the CT scan. Stickers printed with QR codes are strategically placed on the patient. OpenSight interprets the QR stickers to center the image. Cvetko says the QR system seems to have largely resolved this issue.

“OpenSight has spent a good year or two automating multiple functions so you don’t have to be a computer genius and you can be a doctor who uses OpenSight easily at bedside,” Sarani says.

Sarani is anxious to try the new version and has determined that the next time he operates on a patient with broken ribs, “We will use it. End of story.” Sarani sees other uses for OpenSight technology as well, including other spine surgeries. “When we’re operating on a patient’s L3, L4, L5, we’re always wondering where we are,” he says. “If one is able to portray a CT scan of the vertebral column onto the patient, you could do something more precisely or invasively and not have to worry about being on the wrong level.”

The top item on Sarani’s to-do list is to share the technology with his neurosurgeon colleagues to “show them the differences with this technology and see what their thoughts are.” Sarani also suspects that interventional radiologists would be interested in OpenSight to supplement or replace percutaneous placement of catheters when draining abscesses.

Complement to 3D Printing
Robert Hannan, MD, a pediatric cardiac surgeon and medical director of quality for Nicklaus Children’s Health System in Miami, has been using 3D printing to help the cardiac surgical team plan operations on children with congenital heart defects. “We’ve printed about 400 hearts of babies with congenital heart disease since we started with 3D printers in 2015,” Hannan says. While the team has found that 3D printing is much more helpful than 2D images for planning corrective surgeries on tiny hearts, Hannan says it’s still lacking something. So, he asked the biomedical engineers on his research teams to look for what’s missing; when one came back to him with OpenSight, he felt it was ideal.

The problem with 3D printing, Hannan explains, “is that you’d have to print about 20 models for each patient” to be able to see all the different views. That’s extremely time consuming and not as helpful as it might seem, he says. Hannan believes that the ability to project a 3D image on a patient in real time could be most helpful in planning surgeries. The concept behind OpenSight, he says, could improve not just open heart surgery but also revolutionize health care.

There’s one issue that must be resolved, however, before a cardiac surgical team can use OpenSight while performing pediatric heart surgeries. That issue is magnification. Pediatric cardiac surgeons “live our lives with 3X magnification loupes on that allow visualization of a space about the size of a baseball,” Hannan says. The early version of the HoloLens isn’t compatible with the magnification loupes. Hannan is confident this compatibility issue is already being addressed in the next generation, and he suspects the Nicklaus surgical team will be able to use OpenSight while performing congenital pediatric surgery in the near future.

Now, he says, “we’re using 3D models and removing various parts of the heart to see if we can make a pathway from A to B or if we can join A and B together. We have to say, ‘The model we printed is good enough, and we can figure it out from here.’ With the HoloLens, we can just keep changing the view and print it out when we see what we need to see, how close together different parts of the heart’s anatomy are, and what the best pathway would be.” The OpenSight technology also offers another advantage: It will take the technology “one step further to 4D, as it allows you to recreate the beating heart,” Hannan says.

Medical Training
Cvetko also sees a role for OpenSight AR in medical training. Multiple headsets can be shared among users to help train less experienced residents, he says. Medical students can use a teaching version of the software to learn to perform virtual dissections on cadavers.

Another potential use for AR technology is in telemedicine. “You could have the images streamed to another doctor in another state or country,” Cvetko says. “The remote doctor wouldn’t need to have a headset on. He could be looking at the computer screen and seeing things from the bedside doctor’s point of view. The remote doctor could point out with a mouse where on the patient’s body the bedside doctor should cut or avoid during treatment. It opens up amazing possibilities.”

The demand for telemedicine required OpenSight to be interoperable with other technologies, Cvetko says. The latest release seems to address ease of use. “We’ve developed a very comprehensive system for using augmented reality that integrates with our PACS and any other hospital’s PACS so anyone could take advantage of it. You have an interface where you see the patient’s name and study and open it straight up,” Cvetko says. “While you have OpenSight running in the background, you can take a picture of what you’re doing and send it back to the PACS.”

In addition, AR could help eliminate medical errors, Cvetko says. “We don’t have hospitals using it in this way, yet, and while medical mistakes are rare, they happen. When they do, it’s horrible for the patient and very expensive for everyone involved,” he says. “If you used OpenSight for two minutes before every surgery and projected images on the patient, you could confirm that you will be doing the right procedure on the right patient. You can see exactly where the surgical area is, and you can be sure you’re not looking at the wrong side of the patient. If the images didn’t fit the patient, you would know you’re doing the wrong procedure.” This technology also can be used in place of X-rays or CT scans and has the additional benefit of exposing patients to less radiation, Cvetko adds.

Despite OpenSight’s many potential benefits, Sarani sees it playing a niche role. It would be unlikely for it to become standard of care, he says, because that would mean every hospital would be obligated to have the device and know how to use it. “It’s more likely,” he says, “that it be used in Centers of Excellence, where hospitals do certain procedures where it would be most helpful more frequently. Hospitals that do certain procedures once in a blue moon wouldn’t be as likely to adopt the technology.”

Sarani and Hannan both say they are open to different ways of doing things, and they had no hesitation in trying OpenSight. Indeed, they were anxious to be on the forefront of its development. However, they say, given the fact that it is something different, they’re not sure all their colleagues would be won over as easily.

— Beth W. Orenstein of Northampton, Pennsylvania, is a freelance medical writer and regular contributor to Radiology Today.