Home / Blog / Augmented Reality in the Operating Room

Augmented Reality in the Operating Room

The Augmented Doctor. By using HoloLens 2 Augmented Reality can help a doctor and a surgical team in the operating room. Image credit: Microsoft

Augmented reality is slowly, but definitely, entering the Operating Rooms in many hospitals around the world to help the surgeons and the operating team. We have seen surgeons wearing the Google Glass (you can find a comprehensive review of their use here) and as technology evolves, both in terms of devices and software, we are seeing an extension in applications.

Philips, a major player in the provisioning of advanced equipment in the operating rooms, has teamed up with Microsoft to use their Hololens 2 technology (watch the clip) to assist, actually augment, surgeons in the OR.

Hololens 2 supports multiple interactions, gesture recognition, voice control, eye tracking, thus letting the surgeons’ hands free. Interestingly, Hololens 2 software support cooperative working by letting each component in the team to see the same image. Actually, they can choose to see exactly the same image or different images of the same object as if it were seen from different points of view. When delivering information (data) it makes sense to present the same to all members of the team but when over-layering images, like blood vessels, it makes much more sense to present each person with a view that is linked to that person position. This requires quite a bit of software and sensors to determine the exact position of each one.

Actually, using AR in the OR is tricky because of optical constraints. Devices like Hololens 2 are engineered to place the virtual image to a distance of about 2 m, meaning that your eyes will be focusing at that distance. If at the same time you need to focus on a nearer field, because you are operating on a patient that is 50cm from your eyes, or even closer, the two images do not match. Even by tweaking the focal point you still have to manage possible parallax induced errors (you do not want the surgeons to cut a vessel just because she gets the impression it is just a bit farther away. Parallax errors are difficult to avoid since they require instantaneous tracking of the eyes, of the operating field and the understanding of where is the surgeon focussing on (is it at the edge of the surgical opening or five cm below where the heart is beating?).

I am mentioning this to show that the way to apply AR seamlessly in the OR is still a long one. What we have got so far, nevertheless, is already providing a boost and increasing the surgeons performance making the whole procedure safer. Reaching a truly augmented surgeon through AR will require few more years. For the time being AR, and VR, are becoming more and more common in medical training and in the preparation of surgery, letting the surgeon to practice on a virtual patient that mirrors, accurately, the real patient.

About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the New Initiative Committee and co-chairs the Digital Reality Initiative. He is a member of the IEEE in 2050 Ad Hoc Committee. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.