Augmented reality is slowly, but definitely, entering the Operating Rooms in many hospitals around the world to help the surgeons and the operating team. We have seen surgeons wearing the Google Glass (you can find a comprehensive review of their use here) and as technology evolves, both in terms of devices and software, we are seeing an extension in applications.
Philips, a major player in the provisioning of advanced equipment in the operating rooms, has teamed up with Microsoft to use their Hololens 2 technology (watch the clip) to assist, actually augment, surgeons in the OR.
Hololens 2 supports multiple interactions, gesture recognition, voice control, eye tracking, thus letting the surgeons’ hands free. Interestingly, Hololens 2 software support cooperative working by letting each component in the team to see the same image. Actually, they can choose to see exactly the same image or different images of the same object as if it were seen from different points of view. When delivering information (data) it makes sense to present the same to all members of the team but when over-layering images, like blood vessels, it makes much more sense to present each person with a view that is linked to that person position. This requires quite a bit of software and sensors to determine the exact position of each one.
Actually, using AR in the OR is tricky because of optical constraints. Devices like Hololens 2 are engineered to place the virtual image to a distance of about 2 m, meaning that your eyes will be focusing at that distance. If at the same time you need to focus on a nearer field, because you are operating on a patient that is 50cm from your eyes, or even closer, the two images do not match. Even by tweaking the focal point you still have to manage possible parallax induced errors (you do not want the surgeons to cut a vessel just because she gets the impression it is just a bit farther away. Parallax errors are difficult to avoid since they require instantaneous tracking of the eyes, of the operating field and the understanding of where is the surgeon focussing on (is it at the edge of the surgical opening or five cm below where the heart is beating?).
I am mentioning this to show that the way to apply AR seamlessly in the OR is still a long one. What we have got so far, nevertheless, is already providing a boost and increasing the surgeons performance making the whole procedure safer. Reaching a truly augmented surgeon through AR will require few more years. For the time being AR, and VR, are becoming more and more common in medical training and in the preparation of surgery, letting the surgeon to practice on a virtual patient that mirrors, accurately, the real patient.