Augmented reality advances surgical procedures
The idea of using augmented reality (AR) visualisation modalities to improve a surgeon’s perceptive efficiency during surgery is not new. With AR, it is possible to observe the real world with further information that is virtually projected on a display worn by the user. Head-mounted displays (HMDs) offer the most ergonomic and easily translatable solution, but the quality of the virtual content dictates the efficiency of image-guided surgery through AR. HMDs employ optical see-through (OST) or video see-through (VST) modalities. OST displays operate using optical elements to combine real-world and virtual elements. Virtual information is projected into the user’s line of sight, augmenting their direct view of the real world. In VST systems, a camera captures a digital video image of the real world and combines it with computer-generated images (virtual content).
Combining OST and VST technologies in a single HMD
The key objective of the EU-funded VOSTARS project was to enhance the surgeon’s reality by adding all the information available to improve surgery. The scientific team, including experts in computer-assisted surgery, computer graphics, optics and electronics, developed a disruptive OST/VST hybrid HMD. VOSTARS is the only AR HMD able to offer submillimetric accuracy for the planning in relation to the patient. Researchers embedded two very small cameras into the HMD to provide a highly accurate and aligned VST view. The VST modality further improves the accuracy of surgical procedures by combining numerous video frames. The hybrid HMD offers 3D orientation within the patient anatomy and comes with highly flexible software capable of combining OST and VST applications. “With VOSTARS the preoperative plan takes place in real time, and the AR view is superior thanks to the combination of different modalities,” explains Vincenzo Ferrari, VOSTARS project coordinator and associate professor of Biomedical Engineering at the University of Pisa. Researchers have tested the VOSTARS HMD in the clinic with very promising results. According to Ferrari: “The most significant achievement of the project was undoubtedly the demonstration that AR can accurately guide the surgeon’s hand offering the possibility to improve surgery.”
Is the VOSTARS HMD the future of AR-based surgery?
Image-guided surgery using information on patient anatomy and pathology offers surgeons the opportunity for preoperative planning, deciding, for example, optimal cutting trajectories to minimise invasiveness and improve clinical outcome. The VOSTARS HMD constitutes a disruptive tool in the context of image-guided surgery as it permits planning in real time in front of the surgeon’s eyes. Currently, there is no similar technology on the market for surgical navigation. It offers surgeons the opportunity to visualise the internal anatomy of the patient and parts of the body that are usually hidden from the naked eye. This is extremely useful for surgical interventions including the positioning of an implant. Moreover, the HMD is an alternative to commercially available AR products used in various domains, including the avionics and automotive industries. Ongoing engineering activities to optimise the prototype developed during VOSTARS for surgical interventions have been supported by the ARTS4.0 project funded by the Por Fesr European funding programme.
Keywords
VOSTARS, HMD, AR, VST, OST, image-guided surgery, augmented reality, head-mounted display, video see-through, optical see-through