Inside View: A New Way Through the Looking Glass?
By José Morey, MD, and Justine Kemp
Radiology Today
Vol. 18 No. 5 P. 8

AR and VR may be poised to reshape the way people view medical images.

Augmented reality (AR) and virtual reality (VR) are shaping the future landscape for continuing medical education (CME), patient care, and nonradiologist understanding of image reports. With an overall shift toward improving radiology imaging report quality, AR/VR can help increase understanding and improve baseline knowledge regarding these reports.

CME is one area that has endless opportunities to improve with AR/VR applications. In addition to surgical training, several medical schools have implemented this technology in anatomy labs by simulating dissections without using actual cadavers. In this sense, AR/VR has improved delivery of anatomy knowledge and accessibility by being available on computers anywhere.

Phone applications may also be created to make reports more user friendly. For example, a doctor who ordered a scan could potentially use his or her phone to scroll over the image report and see a hologram representation of the image. In addition, the report could be read while the hologram changes axial sections, etc, to display what the report is discussing. This application could ensure that the pathology being discussed in the report is actively being displayed on the hologram, removing confusion as to which section or scan should be referenced. This idea should then be taken one step further.

Consider this case: A patient has inferior mesenteric venous thrombosis recognized on a CT scan. For a doctor or medical student not accustomed to analyzing CT scans, this pathology could be missed. A VR application could discuss the finding while displaying the relevant scan section on the hologram. Then, on this scan, the thrombosis could be highlighted in red to show the provider exactly where and what pathology is being discussed.

In terms of radiological training, AR/VR holds possibilities for medical students, radiology residents, and doctors who order certain reports. Radiology reports can be convoluted and difficult to read. Furthermore, some doctors, and especially medical students, have difficulty finding subtler structures on any given scan. AR/VR has the opportunity to decrease this confusion and, thus, the misinterpretation of images.

For medical students on radiology rotations, learning how to interpret and recognize structures on CT scans can prove to be very difficult. VR programs are currently underway to allow medical students to interact with images in a three-dimensional way. Nevertheless, depending on the age of the patient, recognizing structures becomes even more challenging.

Medical students initially become familiar with adult anatomy but have limited exposure to pediatric anatomy and images. In addition, to being able to interact with structures in a 3D sense, the VR program could also allow the user to interact with how a structure changes both anatomically and on radiographs over a lifetime. At the same time, the program could explain the common pathologies and image findings for any organ, depending on whether a pediatric or adult population is selected. These programs help to provide a more comprehensive understanding of imaging and human anatomy.

How Do Patients Benefit From AR/VR?
This technology has implications for patients, as well. The majority of patients have never seen clinical images. This program could again use the same holograms and highlighted regions to describe the image findings. The program could also use simpler language and remove information, such as normal gallbladder findings, that may be irrelevant to the patient. Clearly, AR/VR has the potential to provide integrated visual explanations, improving understanding and interpretation of images. Woojin Kim, MD, chief medical information officer at Nuance Communications, describes how important it is for radiologist reports to be understood by everyone.

"The imaging report really should be for whoever reads it, meaning, it should be for the patients, referring providers, and other radiologists," Kim says. "It should be comprehensive and consistent. Furthermore, it should be readable by both humans and machines."

In the summer of 2016, Cedars-Sinai Medical Center started using this technology for pain and anxiety relief. In terms of psychiatry, AR/VR is being used for exposure treatment to various phobias, etc. The main focus with AR/VR incorporation, at this point, is to improve patient treatment and satisfaction. However, AR/VR can be used to increase patient understanding of their medical condition and accessibility to necessary care.

Currently, a major issue in health care is the consolidation of patient medical records into one place. A patient may get a CT scan at several different hospitals over different hospital stays that tend to be completely unnecessary. In fact, unnecessary care accounts for 34% of health care expenditure, with imaging being an expensive modality that is often avoidable. AR/VR technologies may be able to help with this.

If a patient has a holographic model of their CT scan on a phone application, he or she may be able to simply show this representation to doctors. The holographic representation would show doctors what the prior imaging shows, therefore preventing another CT scan. Both patients and doctors benefit in this scenario. It decreases the medical bill for the patient while increasing overall efficiency for the doctor/health care team, and the health care team no longer needs to track down various images from medical reports.

AR/VR may also be used to increase patients' accessibility to needed care. Consider another situation: A patient is diagnosed with a lesion on an initial CT scan that may indicate pancreatic cancer, and a hologram representation of the CT scan is immediately linked to this report. In the image report, the words "pancreatic" and "cancer" are recognized. This holographic representation is then immediately sent to both surgical and medical oncologists. After receiving the image, these doctors can contact the patient to make an appointment almost instantly. The result is an increase in patient accessibility to care while decreasing time to receiving this care. Hopefully, this technology will decrease the number of patients who are "lost to follow-up."

As Kim states, "NLP [natural language processing] is being used to extract actionable findings and recommendations from radiology reports. This automation can prevent failed follow-up examinations and avoid adverse clinical outcomes. In addition, improving the consistency of follow-up recommendations can reduce inappropriate imaging and decrease costs for patients and payers."

Additionally, AR/VR technologies can be combined with the crowdsourcing phenomenon that has accelerated with the introduction of the internet. Websites such as CrowdMed and Radiopaedia allow for open sharing and communication of different pathologies and disease presentations. If all imaging had hologram representations attached, individuals could post their reports on websites for second opinions.

Furthermore, this framework directly benefits from the idea of collective intelligence. In terms of radiology, collective intelligence has been studied by having several radiologists read simple images, specifically mammograms. Instead of one radiologist giving an opinion, all radiologists reading the images give opinions. Multiple "reads" have led to increased true positive detection rates and decreased false positive rates. By utilizing collective intelligence, crowdsourcing, and virtual representations of images, the inefficiencies in the health care system, specifically in regard to imaging, may be reduced.

 — José Morey, MD, is a senior medical scientist for IBM Research, a visiting assistant professor in the department of radiology and medical imaging at the University of Virginia, medical technology and artificial intelligence advisor for NASA iTech, chief engineering counsel for Hyperloop Transportation Technologies, a member of the Health Informatics Leadership Council at the VA, and director of informatics for Medical Center Radiologists in Virginia Beach, Virginia.

— Justine Kemp is an MD/MBA candidate at the Tulane University School of Medicine and AB Freeman School of Business in New Orleans.

Resources
Carson E. 10 ways virtual reality is revolutionizing medicine and healthcare. TechRepublic website. http://www.techrepublic.com/article/10-ways-virtual-reality-is-revolutionizing-medicine-and-healthcare/. Published April 8, 2015.

Cyber-Anatomy Med VR. Cyber-Anatomy website. http://cyber-anatomy.com/product_CAHA_medVR.php. Published 2009.        

Dwyer J. Uncovering 6 strong use cases for AR & VR in medicine. Optera Group website. http://www.opteragroup.com/medical-use-cases-for-ar-vr-webinar/

Imaging 3.0 FAQ. American College of Radiology website. http://www.acr.org/FAQs/imaging-3-faq

Westervelt A. Virtual reality is coming to medical imaging. The Wall Street Journal. http://www.wsj.com/articles/virtual-reality-is-coming-to-medical-imaging-1455592257. Published February 15, 2016.

Wolf M, Krause J, Carney PA, Bogart A, Kurvers RH. Collective intelligence meets medical decision-making: the collective outperforms the best radiologist. PLoS One. 2015;10(8):e0134269.