Periodic Reporting for period 2 - HiPerNav (High Performance Soft-tissue Navigation)
Periodo di rendicontazione: 2018-11-01 al 2021-04-30
Liver resection is the treatment of choice in selected patients with primary liver cancer and hepatic colorectal metastases, even in recurrent cases, with 5-year survival rates of up to 58%. A successful surgical resection of HCC requires complete removal of the tumour while sparing as much healthy tissue as possible. Due to technical and clinical difficulties there is an urgent need to increase the patient eligibility and improve the survival prognosis after liver interventions (resection or ablation).
Objectives: The overall goal of HiPerNav is to successfully train and educate young researchers (ESRs) in the multidisciplinary field of image-guided interventions. The scientific and clinical goal is to further develop solutions for computer assisted and image guided surgical and interventional procedures for the treatment of primary and secondary liver cancer. It is aimed at improving eligibility and survival prognosis of cancer patients.
The project is organized through a consortium comprising 5 European universities, 2 university hospitals, 2 research organizations and 5 industrial companies, whereas one SME. Oslo University Hospital is the coordinator. In total 16 young researchers (ESRs) will be financed through the project.
The HiPerNav project aims to improve important bottlenecks in soft-tissue navigation:
–effective pre-operative model and planning
–accurate and fast intra-operative model update
–accurate and fast model-to-patient registration
–intuitive user-interaction and effective workflow
–high performance computing by use of GPU
The 14 organizations in the consortium are selected being one of the leading institutions within their field of expertise, and they are all selected to fill specific competences needed to meet the HiPerNav overall goal. The consortium partner organizations are: Oslo University Hospital, University Hospital Bern, NTNU, SINTEF, INRIA, University of Bern, University Paris13, University of Delft, University of Cordoba, CAScination, SIEMENS, NVIDIA, Yes!Delft and Innovation Norway.
Conclusions of the action: ESRs received academic and industrial training in addition to strong carrier development. In collaboration, consortium further developed existing and found new technological solutions for soft-tissue navigation. Novel navigation workflows have been evaluated in pre-clinical and clinical trials. These achievements may in the future improve treatment of patient with lesion in the liver.
Objectives: The overall goal of HiPerNav is to successfully train and educate young researchers (ESRs) in the multidisciplinary field of image-guided interventions. The scientific and clinical goal is to further develop solutions for computer assisted and image guided surgical and interventional procedures for the treatment of primary and secondary liver cancer. It is aimed at improving eligibility and survival prognosis of cancer patients.
The project is organized through a consortium comprising 5 European universities, 2 university hospitals, 2 research organizations and 5 industrial companies, whereas one SME. Oslo University Hospital is the coordinator. In total 16 young researchers (ESRs) will be financed through the project.
The HiPerNav project aims to improve important bottlenecks in soft-tissue navigation:
–effective pre-operative model and planning
–accurate and fast intra-operative model update
–accurate and fast model-to-patient registration
–intuitive user-interaction and effective workflow
–high performance computing by use of GPU
The 14 organizations in the consortium are selected being one of the leading institutions within their field of expertise, and they are all selected to fill specific competences needed to meet the HiPerNav overall goal. The consortium partner organizations are: Oslo University Hospital, University Hospital Bern, NTNU, SINTEF, INRIA, University of Bern, University Paris13, University of Delft, University of Cordoba, CAScination, SIEMENS, NVIDIA, Yes!Delft and Innovation Norway.
Conclusions of the action: ESRs received academic and industrial training in addition to strong carrier development. In collaboration, consortium further developed existing and found new technological solutions for soft-tissue navigation. Novel navigation workflows have been evaluated in pre-clinical and clinical trials. These achievements may in the future improve treatment of patient with lesion in the liver.
The work carried out during the whole project (reporting period 1 and 2) is in line with the HiPerNav project description and the corresponding Grant Agreement (Annex 1), and is mainly divided into the following areas:
●Preparation and finalizing of the Consortium Agreement (CA) as well as separate agreements with all the Partner Organizations.
●Recruitment of all the 16 Early Stage Researchers (ESRs) in the project.
●Planning and preparation for the training and research of the recruited ESRs.
●Seven Training Events, one week in Oslo, one week in Trondheim, two weeks in Bern, one week in Cordoba, one week in Paris and Strasbourg and one week in Delft, in addition to three HiPerNav Schools in Cordoba, Bern and Oslo (final school held as Zoom seminar).
●Work and research training performed by the ESRs in each Work package defined in the project.
●Technologically, the achievements have been such as:
-Goal-oriented image enhancement with parallel processing for CT liver segmentation. (publication) (ESR3;4;6;8)
-Deep learning based automatic segmentation of liver for faster pre- and intra-operative use. (publication) (ESR2;3;4;7;8;9)
-Deep learning method for frame-level distortion classification and video quality assessment for quality monitoring application. (publication) (ESR3;8)
-Fast blood vessel based image-to-image registration. (ESR14)
-Laparoscopic video to CT/MR image registration. (publication) (ESR1;3;10;12)
-Parallel computing techniques implemented for faster processing of segmentation and registration tasks. (publication) (ESR4;5)
-Use of intraoperative imaging in navigation workflow. (industrial prototype and publication) (ESR1;3;10;12)
-Found that sampling accuracy in point-based registration and intra-operative imaging improves surgical navigation accuracy. (publication) (ESR1;3;10;12)
-Improved Augmented reality visualization for image guided laparoscopic liver surgery. (industrial prototype and publication) (ESR1;3;10)
-Investigated the use of extracted surface from laparoscopic stereo video for 3D liver model-to-patient registration. (publication) (ESR1;3;10;12)
-Demonstrated the possibility of using biomechanical model based on pre-operative segmented model, and how this model can be updated intra-operatively using extracted liver surface from laparoscopic stereo video to drive the deformation of the biomechanical model accordingly. (publication) (ESR1;12)
-Development of method for deep learning-based registration technique of preoperative CT to intraoperative ultrasound images. (publication) (ESR14)
-Use of mixed reality to visualize patient-specific 3D models for surgery planning. (establishment of a company and publication) (ESR3)
-Development of a workflow and a simulation framework for minimally invasive liver treatments. (publication) (ESR15;16)
-Development of an analysis tool for liver ablation (publication) (ESR9)
Explotation: Together with the industrial partners in the HiPerNav project, Cascination (Navigation company) and SIEMENS (here involved with equipment for intraoperative imaging from CBCT and intraoperative CT) the exploitation of the HiPerNav workflow using intraoperative images, automatic and semi-automatic segmented liver models based on these image data, and novel model-to-patient registration methods has been performed.
Dissemination: All results have been scientifically disseminated through peer-reviewed conference- and journal publications.
●Preparation and finalizing of the Consortium Agreement (CA) as well as separate agreements with all the Partner Organizations.
●Recruitment of all the 16 Early Stage Researchers (ESRs) in the project.
●Planning and preparation for the training and research of the recruited ESRs.
●Seven Training Events, one week in Oslo, one week in Trondheim, two weeks in Bern, one week in Cordoba, one week in Paris and Strasbourg and one week in Delft, in addition to three HiPerNav Schools in Cordoba, Bern and Oslo (final school held as Zoom seminar).
●Work and research training performed by the ESRs in each Work package defined in the project.
●Technologically, the achievements have been such as:
-Goal-oriented image enhancement with parallel processing for CT liver segmentation. (publication) (ESR3;4;6;8)
-Deep learning based automatic segmentation of liver for faster pre- and intra-operative use. (publication) (ESR2;3;4;7;8;9)
-Deep learning method for frame-level distortion classification and video quality assessment for quality monitoring application. (publication) (ESR3;8)
-Fast blood vessel based image-to-image registration. (ESR14)
-Laparoscopic video to CT/MR image registration. (publication) (ESR1;3;10;12)
-Parallel computing techniques implemented for faster processing of segmentation and registration tasks. (publication) (ESR4;5)
-Use of intraoperative imaging in navigation workflow. (industrial prototype and publication) (ESR1;3;10;12)
-Found that sampling accuracy in point-based registration and intra-operative imaging improves surgical navigation accuracy. (publication) (ESR1;3;10;12)
-Improved Augmented reality visualization for image guided laparoscopic liver surgery. (industrial prototype and publication) (ESR1;3;10)
-Investigated the use of extracted surface from laparoscopic stereo video for 3D liver model-to-patient registration. (publication) (ESR1;3;10;12)
-Demonstrated the possibility of using biomechanical model based on pre-operative segmented model, and how this model can be updated intra-operatively using extracted liver surface from laparoscopic stereo video to drive the deformation of the biomechanical model accordingly. (publication) (ESR1;12)
-Development of method for deep learning-based registration technique of preoperative CT to intraoperative ultrasound images. (publication) (ESR14)
-Use of mixed reality to visualize patient-specific 3D models for surgery planning. (establishment of a company and publication) (ESR3)
-Development of a workflow and a simulation framework for minimally invasive liver treatments. (publication) (ESR15;16)
-Development of an analysis tool for liver ablation (publication) (ESR9)
Explotation: Together with the industrial partners in the HiPerNav project, Cascination (Navigation company) and SIEMENS (here involved with equipment for intraoperative imaging from CBCT and intraoperative CT) the exploitation of the HiPerNav workflow using intraoperative images, automatic and semi-automatic segmented liver models based on these image data, and novel model-to-patient registration methods has been performed.
Dissemination: All results have been scientifically disseminated through peer-reviewed conference- and journal publications.
Current solutions for surgical navigation have not been adapted to soft-tissue navigation due to challenges related to organ motion and deformation during the procedure. The HiPerNav project aimed to improve several of the bottlenecks existing in current solutions available on the market. In our opinion progress beyond state-of-the art has been achieved in the following areas through integrated cross-disciplinary research and development:
• More effective pre-operative liver model generation and surgical planning tool
• Faster and more accurate intra-operative model updates in combination with prediction of deformation using biomechanical models
• Faster and more accurate model-to-patient registration from intra-operative 3D stereoscopic video and ultrasound
• Novel workflow analysis framework through detailed analysis of the surgical procedure
• In general, the use of parallel computing (HPC/GPU) for smoother, more seamless and better user experience in all steps of the navigation workflow
In general, the total work done by the consortium may in the future and in some aspect has already changed healthcare providers and medical technology industry. Based on explorative research, gathered new knowledge and evidence, consortium disseminated results of the HiPerNav project. These potentially have impact to change clinical workflows and industrial strategies. These achievements may in the future lead to safer medical procedures, lower healthcare costs and increase in quality of life for patients with cancer in the liver.
• More effective pre-operative liver model generation and surgical planning tool
• Faster and more accurate intra-operative model updates in combination with prediction of deformation using biomechanical models
• Faster and more accurate model-to-patient registration from intra-operative 3D stereoscopic video and ultrasound
• Novel workflow analysis framework through detailed analysis of the surgical procedure
• In general, the use of parallel computing (HPC/GPU) for smoother, more seamless and better user experience in all steps of the navigation workflow
In general, the total work done by the consortium may in the future and in some aspect has already changed healthcare providers and medical technology industry. Based on explorative research, gathered new knowledge and evidence, consortium disseminated results of the HiPerNav project. These potentially have impact to change clinical workflows and industrial strategies. These achievements may in the future lead to safer medical procedures, lower healthcare costs and increase in quality of life for patients with cancer in the liver.