Inside View: The Next Evolution of Medical Imaging
By Daniel A. Ortiz, MD, and José Morey, MD
Radiology Today
Vol. 19 No. 2 P. 26
Among the flurry of groundbreaking technologies making headlines in the health care community, augmented reality (AR) may be one of the least discussed. However, AR has gained a lot of traction in the general media.
It's been a hit with consumers as well. For example, the mobile video game craze Pokémon GO has generated at least $1.2 billion in revenue.1 This success has likely contributed to the rush of tech companies and startups that are developing their own software to ride this wave. One of the biggest signals that AR is ready for primetime is the September 2017 Apple announcement that the company has made AR central to the design and marketing of its latest iterations of its phone and operating system.2
(As a follow-up to the May 2017 Radiology Today article on the applications of AR in medical training and patient education, this article explores some early clinical applications.)
Radiology Applications
To date, the majority of clinical research in AR applications for radiology have focused on interventional procedures. Jan Fritz, MD, from Johns Hopkins University, and his coauthors demonstrated the feasibility of using AR to guide needles to joint spaces in cadavers rather than using standard direct image guidance.3 After an initial scan, radiologists were able to create a virtual trajectory, which was then superimposed on the patient using a hologram. Needle tip location could then be tracked virtually using the virtual projection. When the needle tip appeared to be at the target, the patient was rescanned, and the needle tip placement was confirmed with great accuracy. Similar techniques were used to target various osseous lesions, nerve blocks, and vertebroplasties.4-6
Additionally, researchers at Cincinnati Children's Hospital used AR with a C-arm to create a virtual fluoroscopy system.7 Preprocedure, the radiologist performed a cone beam CT and planned a trajectory, which was synced into the system. The C-arm automatically adjusted so that the skin entry point would allow a "down the barrel" approach to the target. The needle was advanced to the planned depth, and a confirmatory cone beam CT was performed.
Not all radiology-related papers were procedurally based. A paper by Douglas et al described converting a breast CT data set into an "immersive 3D environment."8 The "system provides a separate image to each eye with head tracking, ability to rotate, translate, zoom, and converge the eyes to a particular focal point, change intraocular distance, and display fly-through viewing of the image datasets."
Further research may elucidate instances in which a true 3D virtual environment may provide additional diagnostic information.
Assisting in Surgical Applications
If radiologists proactively develop expertise in utilizing AR technologies, they may be able to assist surgeons in the operating room as they do now with ultrasound assistance. Although the technology is in its infancy, several recent papers have utilized AR for preoperative planning and in the operating suite. Applications have included myomectomy, partial nephrectomy, video-assisted thoracic surgery, and transoral robotic surgery.9-12
A benefit cited by many authors is improved safety profiles due to the ability to visualize traditionally "invisible" anatomy via holographic representations superimposed on surgical anatomy. One way this is achieved is by avoiding dangerous vascular anatomy that can't be seen directly. Also, in a paper by Bourdel et al, authors discuss the importance of minimizing surgical incision numbers and dimensions while performing myomectomies to decrease the risk of complications including adhesions.9 According to the authors, AR supplementation allows surgeons to virtually visualize myomas while selecting their approach and incision site, minimizing the need to "explore" down to the target.
The relative novelty of the technology does have several limitations. The most cited limitation is difficulty in real-time predictive modeling of target tissue given anatomic deformation and mobility during surgery.
Radiology Technologist Applications
Another innovative approach to incorporating AR into the radiology space is by assisting radiology technologists to minimize suboptimal imaging and reduce the need for repeat imaging. MacDougall et al from Boston Children's Hospital used Microsoft Kinect to help develop a proprietary C++ based algorithm "to analyze, in real time, the color and depth video streams from the 3D depth-sensing camera."13
The authors described how "the software calculates patient thickness and tracks in real time [15 frames per second] the patient positioning and patient motion." These new data, according to the authors, have the potential to empower technologists to optimize appropriate framing and dose settings as well as notify them when there is active motion that would degrade the images.
Given radiology's expertise in data and imaging, it seems to be a natural evolution for the specialty to harness this next generation of imaging to maximize imaging quality, supplement image-guided procedures, and assist clinical colleagues.
— Daniel A. Ortiz, MD, is a chief radiology resident at Eastern Virginia Medical School. He serves as the president of the American Alliance of Academic Chief Residents in Radiology, vice chair of the ACR's Resident and Fellow Section, and the Washington, D.C., community manager for 3DHeals, LLC.
— José Morey, MD, is a senior medical scientist for IBM Research, a visiting assistant professor in the department of radiology and medical imaging at the University of Virginia, medical technology and artificial intelligence advisor for NASA iTech, chief engineering counsel for Hyperloop Transportation Technologies, a member of the Health Informatics Leadership Council at the VA, and director of informatics for Medical Center Radiologists in Virginia Beach, Virginia.
References
1. Pan A. Pokémon GO has made $1.2 billion in revenue. Game Rant website. https://gamerant.com/pokemon-go-revenue-1-2-billion/. Published 2017.
2. Statt N. Apple shows off breathtaking new augmented reality demos on iPhone 8. The Verge website. https://www.theverge.com/2017/9/12/16272904/apple-arkit-demo-iphone-augmented-reality-iphone-8. Published September 12, 2017.
3. Fritz J, U-Thainual P, Ungi T, et al. Augmented reality visualization with use of image overlay technology for MR imaging-guided interventions: assessment of performance in cadaveric shoulder and hip arthrography at 1.5 T. Radiology. 2012;265(1):254-259.
4. Fritz J, U-Thainual P, Ungi T, et al. Augmented reality visualization using image overlay technology for MR-guided interventions. Invest Radiol. 2013;48(6):464-470.
5. Marker DR, U Thainual P, Ungi T, et al. 1.5 T Augmented reality navigated interventional MRI: paravertebral sympathetic plexus injections. Diagn Interv Radiol. 2017;23(3):227-232.
6. Fritz J, U-Thainual P, Ungi T, et al. MR-guided vertebroplasty with augmented reality image overlay navigation. Cardiovasc Intervent Radiol. 2014;37(6):1589-1596.
7. Racadio JM, Nachabe R, Homan R, Schierling R, Racadio JM, Babić D. Augmented reality on a C-arm system: a preclinical assessment for percutaneous needle localization. Radiology. 2016;281(1):249-255.
8. Douglas DB, Boone JM, Petricoin E, Liotta L, Wilson E. Augmented reality imaging system: 3D viewing of a breast cancer. J Nat Sci. 2016;2(9).
9. Bourdel N, Collins T, Pizarro D, et al. Augmented reality in gynecologic surgery: evaluation of potential benefits for myomectomy in an experimental uterine model. Surg Endosc. 2016;31(1):456-461.
10. Hekman MCH, Rijpkema M, Langenhuijsen JF, Boerman OC, Oosterwijk E, Mulders PFA. Intraoperative imaging techniques to support complete tumor resection in partial nephrectomy [published online May 12, 2017]. Eur Urol Focus. doi: 10.1016/j.euf.2017.04.008.
11. Rouzé S, de Latour B, Flécher E, et al. Small pulmonary nodule localization with cone beam computed tomography during video-assisted thoracic surgery: a feasibility study. Interact Cardiovasc Thorac Surg. 2016;22(6):705-711.
12. Liu WP, Richmon JD, Sorger JM, Azizian M, Taylor RH. Augmented reality and cone beam CT guidance for transoral robotic surgery. J Robot Surg. 2015;9(3):223-233.
13. MacDougall RD, Scherrer B, Don S. Development of a tool to aid the radiologic technologist using augmented reality and computer vision. Pediatr Radiol. 2018;48(1):141-145.
