Periodic Reporting for period 1 - HoloSurge (Multimodal 3D Holographic tool and real-time Guidance System with point-of-care diagnostics for surgical planning and interventions on liver and pancreatic cancers)
Reporting period: 2024-01-01 to 2025-06-30
The HoloSurge project aims to advance perioperative care by providing surgical teams with a multimodal 3D holographic tool. This tool will fuse various imaging modalities and integrate a real-time guidance system with point-of-care diagnostics. By offering a unified view, it will enable better, faster, and more objective decision-making, significantly advancing current perioperative care workflows.
There is a lack of real-time data to support clinical decision-making during surgical workflows. Each year, a staggering amount of 310 million major surgeries are performed across the globe. Despite many advancements in recent decades, surgical complications continue to occur at too high a rate, ranging from infections or bleeding to organ damage or death. As a result, 1-4% of surgical patients die and up to 15% suffer from serious postoperative morbidity. Every year, ~4.2 million people pass away within 30 days of surgery, thus attributing to a significant 7.7% of all worldwide fatalities, ranking just behind ischaemic heart disease and stroke. Without fused images to guide the procedure, the surgeon’s perception can be inaccurate and imprecise, which can lead to, e.g. vessels not being seen before they are hit, or performing an incomplete tumour resection because some lesions were not properly captured on all non-fused imaging modalities. Moreover, without real-time cell-level data, it is currently impossible to accurately define resection margins and guarantee that no diseased cells remain inside the patient, while in the operating room.
These unmet needs are boldly in display in complex procedures such as liver and pancreatic cancer resection workflows, wherein surgical complications can occur in up to 48% and 60% of patients, respectively (4,5), and if a tumor is located in the pancreatic tail, it is missed in 50% of the cases.
The multifaceted challenges of the clinical problem at hand is best addressed with a collaborative approach. The HoloSurge consortium was brought together to thoroughly identify the gaps in current surgical workflows and
ensure the proposed technological solution meets the needs of the surgeons and patients it aims to serve. A primary objective of HoloSurge is the practical application of its outcomes, leading towards a market-ready solution. A key partner in this endeavor, HoloCare, is at the forefront of applying this research in a real-world setting. They have successfully developed a CE-approved product that transforms CT images to interactive holograms for liver surgery. By utilizing patient-specific holograms generated from CT scans, HoloCare’s technology offers a pioneering approach to surgical preparation. A commercial partner like this underscores the project’s commitment to translating research into tangible healthcare solutions.
There is a lack of real-time data to support clinical decision-making during surgical workflows. Each year, a staggering amount of 310 million major surgeries are performed across the globe. Despite many advancements in recent decades, surgical complications continue to occur at too high a rate, ranging from infections or bleeding to organ damage or death. As a result, 1-4% of surgical patients die and up to 15% suffer from serious postoperative morbidity. Every year, ~4.2 million people pass away within 30 days of surgery, thus attributing to a significant 7.7% of all worldwide fatalities, ranking just behind ischaemic heart disease and stroke. Without fused images to guide the procedure, the surgeon’s perception can be inaccurate and imprecise, which can lead to, e.g. vessels not being seen before they are hit, or performing an incomplete tumour resection because some lesions were not properly captured on all non-fused imaging modalities. Moreover, without real-time cell-level data, it is currently impossible to accurately define resection margins and guarantee that no diseased cells remain inside the patient, while in the operating room.
These unmet needs are boldly in display in complex procedures such as liver and pancreatic cancer resection workflows, wherein surgical complications can occur in up to 48% and 60% of patients, respectively (4,5), and if a tumor is located in the pancreatic tail, it is missed in 50% of the cases.
The multifaceted challenges of the clinical problem at hand is best addressed with a collaborative approach. The HoloSurge consortium was brought together to thoroughly identify the gaps in current surgical workflows and
ensure the proposed technological solution meets the needs of the surgeons and patients it aims to serve. A primary objective of HoloSurge is the practical application of its outcomes, leading towards a market-ready solution. A key partner in this endeavor, HoloCare, is at the forefront of applying this research in a real-world setting. They have successfully developed a CE-approved product that transforms CT images to interactive holograms for liver surgery. By utilizing patient-specific holograms generated from CT scans, HoloCare’s technology offers a pioneering approach to surgical preparation. A commercial partner like this underscores the project’s commitment to translating research into tangible healthcare solutions.
WP1 – Integration of the holographic solution with hospitals’ IT infrastructure
HoloCare and Orbit21 developed and deployed the PACS Integration Engine (PIE), enabling clinicians to transfer CT and MRI data from hospital systems to HoloCare Studio. Data Protection Impact Assessments and Information Governance documentation were completed for multiple hospitals. Integration work was initiated at five hospitals, with virtual server space pending for full deployment.
WP2 – Organ-level imaging processing and generation of holograms for guidance
An AI segmentation engine was trained on liver anatomy and optimized for near real-time performance (<2 minutes). CT-to-CT and CT-to-MRI fusion features were developed using non-rigid transformation algorithms. A new user interface was created with smart editing tools and automated lesion detection. SINTEF initiated MRI-to-ultrasound registration using datasets from NKI.
WP3 – Real-time ultrasound imaging
Retrospective intraoperative ultrasound datasets were collected from NKI and OUS. AI segmentation models for 2D and 3D ultrasound were developed and tested. A 3D reconstruction algorithm was implemented using image correlation to correct tracking errors. Ultrasound simulation from CT/MRI was developed using Nvidia Isaac4Health. Data acquisition protocols were prepared for prospective studies at LMU and NIO.
WP4 – Real-time cell-level cancer diagnosis
Camera setups for laparoscopic and open surgery were built and tested. Data collection began using a high-quality hyperspectral imaging device, with IRB approval secured. Algorithms are being developed using transfer learning from research-grade data to optimize the prototype for clinical use.
WP5 – Real-time hologram-to-patient registration and intraoperative navigation
Real-time streaming of ultrasound and tracking data was implemented. Laparoscopic video data was collected, and AI tools were used to segment anatomical structures. A workflow was developed to register real-time anatomical models using intraoperative data, achieving <1 second latency. Holograms were synchronized with patient anatomy using HoloLens SLAM and tracking technologies.
WP6 – Social aspects and user-centred design
Ethical approvals were secured for user-centred research. A comprehensive protocol was developed for stakeholder engagement. Design Thinking Workshops were scheduled to gather insights from clinicians, patients, and developers. Preliminary evaluations using holographic models were conducted at University of Leeds, informing early system refinements.
WP7 – Preclinical and clinical demonstrations
Early evaluation of HoloCare Studio 2 was completed in liver MDT meetings. Clinical study protocols were drafted and reviewed. Clinical sites were recruited, and international site agreements were initiated ahead of schedule.
WP8 – Regulatory aspects
Semmelweis University identified VR/AR product lines for surgical planning and navigation. Health technology assessment frameworks were initiated. HoloCare liaised with BSI and FDA to ensure compliance with ISO 13485:2016 and MDR regulations. Design History Files were maintained for CE/UKCA marking.
HoloCare and Orbit21 developed and deployed the PACS Integration Engine (PIE), enabling clinicians to transfer CT and MRI data from hospital systems to HoloCare Studio. Data Protection Impact Assessments and Information Governance documentation were completed for multiple hospitals. Integration work was initiated at five hospitals, with virtual server space pending for full deployment.
WP2 – Organ-level imaging processing and generation of holograms for guidance
An AI segmentation engine was trained on liver anatomy and optimized for near real-time performance (<2 minutes). CT-to-CT and CT-to-MRI fusion features were developed using non-rigid transformation algorithms. A new user interface was created with smart editing tools and automated lesion detection. SINTEF initiated MRI-to-ultrasound registration using datasets from NKI.
WP3 – Real-time ultrasound imaging
Retrospective intraoperative ultrasound datasets were collected from NKI and OUS. AI segmentation models for 2D and 3D ultrasound were developed and tested. A 3D reconstruction algorithm was implemented using image correlation to correct tracking errors. Ultrasound simulation from CT/MRI was developed using Nvidia Isaac4Health. Data acquisition protocols were prepared for prospective studies at LMU and NIO.
WP4 – Real-time cell-level cancer diagnosis
Camera setups for laparoscopic and open surgery were built and tested. Data collection began using a high-quality hyperspectral imaging device, with IRB approval secured. Algorithms are being developed using transfer learning from research-grade data to optimize the prototype for clinical use.
WP5 – Real-time hologram-to-patient registration and intraoperative navigation
Real-time streaming of ultrasound and tracking data was implemented. Laparoscopic video data was collected, and AI tools were used to segment anatomical structures. A workflow was developed to register real-time anatomical models using intraoperative data, achieving <1 second latency. Holograms were synchronized with patient anatomy using HoloLens SLAM and tracking technologies.
WP6 – Social aspects and user-centred design
Ethical approvals were secured for user-centred research. A comprehensive protocol was developed for stakeholder engagement. Design Thinking Workshops were scheduled to gather insights from clinicians, patients, and developers. Preliminary evaluations using holographic models were conducted at University of Leeds, informing early system refinements.
WP7 – Preclinical and clinical demonstrations
Early evaluation of HoloCare Studio 2 was completed in liver MDT meetings. Clinical study protocols were drafted and reviewed. Clinical sites were recruited, and international site agreements were initiated ahead of schedule.
WP8 – Regulatory aspects
Semmelweis University identified VR/AR product lines for surgical planning and navigation. Health technology assessment frameworks were initiated. HoloCare liaised with BSI and FDA to ensure compliance with ISO 13485:2016 and MDR regulations. Design History Files were maintained for CE/UKCA marking.
HoloSurge has delivered several innovations that exceed current clinical and technological standards. The AI segmentation engine enables near real-time liver anatomy processing with minimal manual input. CT-MRI fusion and automated lesion detection improve preoperative planning. Real-time ultrasound imaging and hologram registration workflows allow dynamic intraoperative updates, enhancing surgical navigation. Hyperspectral imaging introduces new capabilities for cell-level cancer diagnosis.
These results have the potential to improve surgical precision, reduce planning time, and enhance patient outcomes. Early clinical evaluations confirm usability and relevance. Regulatory alignment and IPR protection are in place to support future deployment.
To ensure further uptake, the project will benefit from continued research, broader clinical validation, access to funding and markets, strong commercialisation pathways, and supportive regulatory and standardisation frameworks.
These results have the potential to improve surgical precision, reduce planning time, and enhance patient outcomes. Early clinical evaluations confirm usability and relevance. Regulatory alignment and IPR protection are in place to support future deployment.
To ensure further uptake, the project will benefit from continued research, broader clinical validation, access to funding and markets, strong commercialisation pathways, and supportive regulatory and standardisation frameworks.