Periodic Reporting for period 2 - VOSTARS (Video Optical See-Through Augmented Reality surgical System) Période du rapport: 2018-06-01 au 2020-11-30 Résumé du contexte et des objectifs généraux du projet In the medical domain, the idea of integrating the surgeon’s vision with the aid of new AR visualization modalities has become a dominant topic of academic and industrial research since the 90’s. In the ideal AR system, especially if designed to be used in complex manual tasks that demand for great dexterity, there should not be any perceivable difference between the user’s natural view of the world and his/her own augmented view through the device display. In line with this, wearable AR displays based on head-mounted displays (HMDs) intrinsically provide the user with an egocentric viewpoint and represent the most ergonomic and efficient solution for aiding those manual tasks performed under direct vision. Most of the AR HMDs fall into two categories according to the see-through paradigm they implement: video see-through (VST) and optical see-through (OST) HMDs.With respect to technological and human-factor issues, both the approaches have their own strengths and shortcomings. In VOSTARS, we will develop a hybrid OST/VST HMD and will assess its efficacy in improving surgical outcomes and aiding manual tasks. Travail effectué depuis le début du projet jusqu’à la fin de la période considérée dans le rapport et principaux résultats atteints jusqu’à présent Here follows a short outline of the main results achieved in the early phase of the project.1) Hybrid VST/OST HMD. VOSTARS will offer both VST and OST views. As regards the mechanism that manages the transition between the two AR modalities, VOSTARS will comprise an optoelectronic shutter instead of a mechanical cover. The main advantage of this solution is the significant reduction of movable parts, which leads to a more compact housing design and a longer lifetime for the HMD. We performed usability tests inside a surgical room and the results published in[1] (Fig1).2) VST setting with off-axis cameras. In line with the goal of avoiding any movable part in the HMD, we evaluated the possibility to anchor the RGB cameras (for the VST view) in an off-axis setting, therefore outside the user's line of sight. We recently published a study[2], where we proved that we can properly recover natural view of the scene in a region around a predefined distance from the observer, by applying a proper geometrical transformation to the images grabbed by the RGB cameras (Fig2).3) No translation nor rotation of the displays. We investigated on how to offer a good ergonomics and, at the same time, reduce the need for complex and lengthy calibrations for generating an authentic stereoscopic vision. In a paper that we recently submitted to IEEE TVCG, we showed, that the above mentioned geometric transformation can also prevent from phyisically adjusting cameras and/or displays convergence as a function of the working distance.4) Eye-tracking for highly accurate OST view. VOSTARS will comprise a real-time eye pose estimation routine whose goal is to achieve a properly aligned OST view. This functionality is based on the use of 2 internal cameras embedded to the HMD. We selected very small cameras that we are integrating in proximity of the the waveguide of the near-eye-display (NED). Further, we implemented a calibration routine suited for NEDs that achieves high real-to-virtual alignment and whose results were accepted for publication as proceedings of the IEEE ISMAR 2018 conference[3] (Fig3).5) Display field-of-view (FOV) and resolution (res). While the restricted FOV represents one of the most criticized features of commercially available HMDs in terms of usability, also the angular resolution is to be considered as a key characteristic: increasing the FOV of the optical engine of the HMD should be accompanied by an identical increase in the display res to achieve the same angular res. For this reason in VOSTARS we preferred to increase the angular res istead of the FOV alone. 6) New generation of OLED micro displays. Into VOSTARS HMD we will integrate a new generation of compact, high res, low consumption, and very bright LED microdisplays.7) No external trackers are needed for an accurate tracking inside the surgical room. To avoid the use of obtrusive external trackers, and to guarantee high tracking and registration accuracy, we are integrating both RGB and IR cameras.8) High wearability and embedded electronics. We selected a compact computational unit. The selected board can be mounted directly on the headset and this will be beneficial in terms of device usability.9) "Glass-like" OST-VST displays. We designed a vertical waveguide so to preserve full lateral view (as in standard prescription glasses) (Fig4).10) Highly flexible software. In VOSTARS we are designing and developing a software framework capable of managing several video or optical see through-based AR applications. The software will be user-friendly and highly configurable so to make it suitable for many potential use cases.11) Task-oriented and dedicated AR visualization for surgical applications. The definition of the virtual content starts from decomposing the addressed intervention into sub-tasks. In fact, AR is effective in aiding the surgeon only if it is strongly related to the task. During the early phase of the project we published 2 papers where we presented the results of the in vitro assessment of new AR visualization modalities as aid in specific surgical procedures. The studies were performed on a custom-made VST HMD[4] and on a commercial OST HMD[5].12) A prototype, not a demonstrator. Our final goal in VOSTARS is to reach a technology-readiness-level (TRL) 7 that is “system prototype demonstration in operational environment”. For this reason, engineers and industrial designers are working very close to obtain a semi-definitive design (Fig6).[1] F.Cutolo U.Fontana M.Carbone R.D.Amato V.Ferrari "[POSTER] Hybrid Video/Optical See-Through HMD," in 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), 2017, pp. 52-57.[2] F.Cutolo U.Fontana V.Ferrari "Perspective Preserving Solution for Quasi-Orthoscopic Video See-Through HMDs," Technologies, vol. 6, Mar 2018.[3] U.Fontana F.Cutolo N.Cattari V.Ferrari "Closed – Loop Calibration for Optical See-Through Near Eye Display with Infinity Focus", accepted for publication in 2018 International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). [4] F.Cutolo A.Meola M.Carbone S.Sinceri F.Cagnazzo E.Denaro N.Esposito M.Ferrari V.Ferrari "A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom," Comput Assist Surg (Abingdon), vol. 22, pp. 39-53, Dec 2017.[5] M.Carbone S.Condino F.Cutolo R.M.Viglialoro O.Kaschke U.W.Thomale V.Ferrari "Proof of Concept: Wearable Augmented Reality Video See-Through Display for Neuro-Endoscopy," Cham, 2018, pp. 95-104. Progrès au-delà de l’état des connaissances et impact potentiel prévu (y compris l’impact socio-économique et les conséquences sociétales plus larges du projet jusqu’à présent) Most of the objectives listed above are beyond the state of the art. VOSTARS will be a disruptive tool in the context of image-guided surgery: no similar technology is at present available on the market for surgical navigation and therefore no similar system is routinely used in the clinical practice. Furthermore, the stand-alone VOSTARS HMD (i.e. without the surgical navigation platform) could represent per se a valid and European alternative to commercially available AR HMDs. VOSTARS HMD will find a market in a variety of industrial domains, such as monitoring assembly process for complex parts in avionics, automotive, etc. This additional market impact will be boosted by the challenges and requirements set by the forth-industrial revolution (Industry 4.0). Highly accurate AR alignment with OST HMD as presented in [3] The making of the final VOSTARS HMD Custom-made VST HMD with AR application for aiding complex neurosurgical tasks as presented in [4] VST setting with off-axis cameras with method for recovering natural view of the world [2] Embodiment of hybrid VST/OST HMD with commercial components presented at ISMAR 2017 [1] "Glass-like" OST-VST displays with vertical waveguides
