Final Report Summary - FI2P4TUM (Fluorescence intraoperative imaging platform for tumor margin delineation)
What is already known about this subject?
Many imaging modalities ranging from ultrasonography to magnetic resonance imaging (MRI) have become standard clinical practice for cancer tumor detection, staging and treatment evaluation. However, translating these technologies to the operation room remains a major challenge, and when surgery is required, clinical decision making still relies on the visual appearance and palpation of the tumor. Surgery is then confronted with difficulties in cancer spread identification and in the accurate delineation of the tumor margins, leading to incomplete resections of tumors that critically compromise the prognosis of patients. Similarly, the major challenge that physicians face during surveillance and endoscopic resection is the lack of functional and molecular information to facilitate decision-making regarding which lesions to remove. Diagnosis is primarily based on macroscopic endoscopic features, or on the reduced tissue fraction that is sampled even in the cases of rigorous adherence to biopsy protocols, which results in substantial lesion miss-rates that can be as high as 55 % in patients with an inherited predisposition to colon cancer.
Among the various optical techniques considered for surgical and endoscopic imaging, wide field targeted imaging with near-infrared fluorescence appears to be the most promising approach to shape the future of these clinical procedures. Increased light tissue penetration is obtained by imaging in the NIR range, while an enhanced tumor-to-background ratio is attained by targeted fluorescent agents that provide molecularly specific detection of cancer cells and allow the recognition of otherwise invisible disease biomarkers.
What are the new findings?
- Development of a modular fluorescence imaging platform for interventional imaging. We have designed and built an imaging setup to offer video-rate simultaneous color and near-infrared (NIR) fluorescence acquisition. The system makes use of different cameras operating in parallel to simultaneously acquire color and fluorescence videos using a common objective. The acquired image is separated by a beam splitter into a visible and a NIR component and relayed to two CCD cameras by relay lenses. The allocation of a filter wheel in the fluorescence arm of the system allows for the simultaneous visualization of different fluorescent contrast agents. White light illumination for color imaging is provided by a 250 W halogen lamp (KL-2500 LCD, Schott AG, Mainz, Germany) while fluorescence excitation is alternatively provided by CW laser diodes whose central wavelengths match the maximum absorption peaks of the correspondingly employed fluorescent agents. Color images are recorded by a 12-bit color camera (PCO AG, Kelheim, Germany) while the fluorescence videos are guided to an iXon 3 CCD (DU-888, Andor Technology, Belfast, Northern Ireland) that offers a quantum efficiency of over 75 % in the NIR region. Shortpass filters are placed in front of the white light source to minimize cross-talk of white light into the NIR channel. The designed platform is completely modular and user-friendly. To implement real-time operation both cameras are connected to PCI acquisition cards that transfer the data via direct memory access (DMA) into the computer’s RAM. The raw data is saved on a Solid-State-Drive (SSD) (256 GB, SATA-III 6 Gb/s, Liteon, Taiwan) that can handle the high throughput of video-rate images with up to 1.3 MPixel and up to 14-bits. The images are then transferred to the graphics processing unit (GPU) memory for post-processing. By using NVIDIA’s Compute Unified Device Architecture (CUDA) the computationally intensive processing steps can be offloaded to the GPU’s parallel architecture, which results in a significant speedup compared to traditional sequential implementations on the CPU. A custom designed system software displays the co-registered color and fluorescence channel images to the user (the surgeon or the endoscopist) as well as the composite overlay. Furthermore the interface offers easy control of all relevant system parameters and the possibility to capture both individual images and H.264/MPEG-4 recordings of all channels. The modularity of the platform also includes an ease of adaptation between open and minimally invasive surgery and surveillance endoscopy environments, since any required modification of the implementation for open surgery to make it compatible with conventional clinical laparoscopes and videoendoscopes is accomplished in front of the beamsplitter that separates the two image components. For minimally invasive surgery (laparoscopy) and for surveillance endoscopic procedures, the light sources for color imaging and fluorescence excitation are coupled through a multimode fiber (Leoni FiberOptics, Neuhaus-Schierschnitz, Germany) into the illumination port of the laparoscope / endoscope, while the acquired images are relayed by the lenses inside the laparoscope or the coherent fiber-optic bundles inside the flexible clinical grade endoscope to the highly sensitive fluorescence imaging module of the camera system, to satisfy the real-time constraints of interventional procedures. Pre-clinical validation of the different modules of the imaging platform for interventional imaging was performed in murine mouse models of human cancer diseases with significant prevalence. In these preclinical validations NIR fluorescence probes mainly based on clinically approved monoclonal antibodies, specifically cetuximab, targeting epidermal growth factor receptor (EGFR), bevacizumab, targeting vascular endothelial grow factor (VEGF), and trastuzumab against epidermal growth factor receptor 2 (HER2) were utilized. This strategy gives priority to the reduction of the risk in clinical translation by using fluorescent agents based on approved drugs instead of new chemical entities that have never been administered to humans.
- Research into optical parameter models. The designed fluorescence imaging platform acquires color and fluorescence images in epi-illumination and detection geometry (simple “photographic” or “video” methods). Although this configuration is appropriate for the geometrical constraints of the surgical and endoscopy environments, it implies that the measured fluorescence intensity is modulated by the tissue optical properties and the depth of the recorded activity, and does not directly report the underlying agent concentration. Consequently, investigation into optical parameter models that allow the extraction of the optical properties (absorption and scattering) of the underlying tissue has been performed. The determination of the tissue optical properties from multispectral measurements of the total diffuse reflectance requires the description of the latter as a function of wavelength invariant parameters, namely the chromophore concentrations (blood concentration assuming negligible the influence of other absorbers), the scattering amplitude and the scattering power. It has been demonstrated based on theory-based techniques, numerical simulations and empirical measurements that only the ratio between the blood concentration and the scattering amplitude, the oxygen saturation and the scattering power can be obtained independently of the number of measured wavebands. This is, in principle, insufficient to achieve an overall compensation of the effects of absorption and scattering in tissue to quantify the absolute fluorophore concentration. However, based on these outcomes, studies on the optimal wavebands for the accurate determination of the oxygen saturation from multispectral measurements of the diffuse reflectance have been determined. Further modification of the fluorescence imaging platform to utilize these wavebands instead of traditional RGB is envisaged for the near future, since the complete reflectance spectrum, assuming blood as the sole absorber, can be reconstructed from the measurements at these wavelengths. In addition, the computation of the oxygen saturation maps of the field-of-view has a diagnostic meaning by itself.
- Image interpretation and segmentation. The challenges arising from the necessity to establish a propagation model that accounts for scattering and light diffusion in tissue have been confronted by the employment of segmentation and interpretation techniques based on blind signal separation (BSS) to generate the pseudo-diagnosis images to be presented to the practitioner. Traditional BSS such us Principal Component Analysis (PCA) and Independent Component Analysis (ICA) were investigated to decouple from the fluorescence images the influence of the tissue optical properties varying across the field of view. Additionally, and regarding tumor and lesion delineation and the computation of the target (tumor or lesion) – to – background ration, different segmentation alternatives were considered and compared regarding the accurate estimation of the malignancy boundaries and the induced false alarms and false positive. In the current implementation of the imaging platform and for the generation of the pseu-diagnosis images, the fluorescence images are segmented according to the Otsu’s method and the segmented images are overlaid on the color information to provide the practitioner with feedback about the malignancy location and spread.
How might it impact on clinical practice in the foreseeable future?
Fluorescence molecular imaging is expected to have outstanding clinical relevance, mainly as regards early detection and accurate estimation of the extent of diseases and patient-specific therapy selection and monitoring. During the FI2P4TUM project, a comprehensive work program regarding the development of acquisition systems, and modeling and interpretation algorithms of fluorescence images and videos has been carried out. The latter has been combined with the anticipation in preclinical studies of the technique based on NIR fluorescence-guidance for minimally invasive surgery as well as surveillance endoscopy. Altogether the accomplished research bridges NIR fluorescence molecular imaging and its expected outstanding relevance to clinical translation. More importantly the relevance of the impact on clinical practice of these developments and, as a consequence their socio-economic impact, will be explored in two clinical studies that will commence shortly (NL36315.042.13 and NL-43407.042.113).
Many imaging modalities ranging from ultrasonography to magnetic resonance imaging (MRI) have become standard clinical practice for cancer tumor detection, staging and treatment evaluation. However, translating these technologies to the operation room remains a major challenge, and when surgery is required, clinical decision making still relies on the visual appearance and palpation of the tumor. Surgery is then confronted with difficulties in cancer spread identification and in the accurate delineation of the tumor margins, leading to incomplete resections of tumors that critically compromise the prognosis of patients. Similarly, the major challenge that physicians face during surveillance and endoscopic resection is the lack of functional and molecular information to facilitate decision-making regarding which lesions to remove. Diagnosis is primarily based on macroscopic endoscopic features, or on the reduced tissue fraction that is sampled even in the cases of rigorous adherence to biopsy protocols, which results in substantial lesion miss-rates that can be as high as 55 % in patients with an inherited predisposition to colon cancer.
Among the various optical techniques considered for surgical and endoscopic imaging, wide field targeted imaging with near-infrared fluorescence appears to be the most promising approach to shape the future of these clinical procedures. Increased light tissue penetration is obtained by imaging in the NIR range, while an enhanced tumor-to-background ratio is attained by targeted fluorescent agents that provide molecularly specific detection of cancer cells and allow the recognition of otherwise invisible disease biomarkers.
What are the new findings?
- Development of a modular fluorescence imaging platform for interventional imaging. We have designed and built an imaging setup to offer video-rate simultaneous color and near-infrared (NIR) fluorescence acquisition. The system makes use of different cameras operating in parallel to simultaneously acquire color and fluorescence videos using a common objective. The acquired image is separated by a beam splitter into a visible and a NIR component and relayed to two CCD cameras by relay lenses. The allocation of a filter wheel in the fluorescence arm of the system allows for the simultaneous visualization of different fluorescent contrast agents. White light illumination for color imaging is provided by a 250 W halogen lamp (KL-2500 LCD, Schott AG, Mainz, Germany) while fluorescence excitation is alternatively provided by CW laser diodes whose central wavelengths match the maximum absorption peaks of the correspondingly employed fluorescent agents. Color images are recorded by a 12-bit color camera (PCO AG, Kelheim, Germany) while the fluorescence videos are guided to an iXon 3 CCD (DU-888, Andor Technology, Belfast, Northern Ireland) that offers a quantum efficiency of over 75 % in the NIR region. Shortpass filters are placed in front of the white light source to minimize cross-talk of white light into the NIR channel. The designed platform is completely modular and user-friendly. To implement real-time operation both cameras are connected to PCI acquisition cards that transfer the data via direct memory access (DMA) into the computer’s RAM. The raw data is saved on a Solid-State-Drive (SSD) (256 GB, SATA-III 6 Gb/s, Liteon, Taiwan) that can handle the high throughput of video-rate images with up to 1.3 MPixel and up to 14-bits. The images are then transferred to the graphics processing unit (GPU) memory for post-processing. By using NVIDIA’s Compute Unified Device Architecture (CUDA) the computationally intensive processing steps can be offloaded to the GPU’s parallel architecture, which results in a significant speedup compared to traditional sequential implementations on the CPU. A custom designed system software displays the co-registered color and fluorescence channel images to the user (the surgeon or the endoscopist) as well as the composite overlay. Furthermore the interface offers easy control of all relevant system parameters and the possibility to capture both individual images and H.264/MPEG-4 recordings of all channels. The modularity of the platform also includes an ease of adaptation between open and minimally invasive surgery and surveillance endoscopy environments, since any required modification of the implementation for open surgery to make it compatible with conventional clinical laparoscopes and videoendoscopes is accomplished in front of the beamsplitter that separates the two image components. For minimally invasive surgery (laparoscopy) and for surveillance endoscopic procedures, the light sources for color imaging and fluorescence excitation are coupled through a multimode fiber (Leoni FiberOptics, Neuhaus-Schierschnitz, Germany) into the illumination port of the laparoscope / endoscope, while the acquired images are relayed by the lenses inside the laparoscope or the coherent fiber-optic bundles inside the flexible clinical grade endoscope to the highly sensitive fluorescence imaging module of the camera system, to satisfy the real-time constraints of interventional procedures. Pre-clinical validation of the different modules of the imaging platform for interventional imaging was performed in murine mouse models of human cancer diseases with significant prevalence. In these preclinical validations NIR fluorescence probes mainly based on clinically approved monoclonal antibodies, specifically cetuximab, targeting epidermal growth factor receptor (EGFR), bevacizumab, targeting vascular endothelial grow factor (VEGF), and trastuzumab against epidermal growth factor receptor 2 (HER2) were utilized. This strategy gives priority to the reduction of the risk in clinical translation by using fluorescent agents based on approved drugs instead of new chemical entities that have never been administered to humans.
- Research into optical parameter models. The designed fluorescence imaging platform acquires color and fluorescence images in epi-illumination and detection geometry (simple “photographic” or “video” methods). Although this configuration is appropriate for the geometrical constraints of the surgical and endoscopy environments, it implies that the measured fluorescence intensity is modulated by the tissue optical properties and the depth of the recorded activity, and does not directly report the underlying agent concentration. Consequently, investigation into optical parameter models that allow the extraction of the optical properties (absorption and scattering) of the underlying tissue has been performed. The determination of the tissue optical properties from multispectral measurements of the total diffuse reflectance requires the description of the latter as a function of wavelength invariant parameters, namely the chromophore concentrations (blood concentration assuming negligible the influence of other absorbers), the scattering amplitude and the scattering power. It has been demonstrated based on theory-based techniques, numerical simulations and empirical measurements that only the ratio between the blood concentration and the scattering amplitude, the oxygen saturation and the scattering power can be obtained independently of the number of measured wavebands. This is, in principle, insufficient to achieve an overall compensation of the effects of absorption and scattering in tissue to quantify the absolute fluorophore concentration. However, based on these outcomes, studies on the optimal wavebands for the accurate determination of the oxygen saturation from multispectral measurements of the diffuse reflectance have been determined. Further modification of the fluorescence imaging platform to utilize these wavebands instead of traditional RGB is envisaged for the near future, since the complete reflectance spectrum, assuming blood as the sole absorber, can be reconstructed from the measurements at these wavelengths. In addition, the computation of the oxygen saturation maps of the field-of-view has a diagnostic meaning by itself.
- Image interpretation and segmentation. The challenges arising from the necessity to establish a propagation model that accounts for scattering and light diffusion in tissue have been confronted by the employment of segmentation and interpretation techniques based on blind signal separation (BSS) to generate the pseudo-diagnosis images to be presented to the practitioner. Traditional BSS such us Principal Component Analysis (PCA) and Independent Component Analysis (ICA) were investigated to decouple from the fluorescence images the influence of the tissue optical properties varying across the field of view. Additionally, and regarding tumor and lesion delineation and the computation of the target (tumor or lesion) – to – background ration, different segmentation alternatives were considered and compared regarding the accurate estimation of the malignancy boundaries and the induced false alarms and false positive. In the current implementation of the imaging platform and for the generation of the pseu-diagnosis images, the fluorescence images are segmented according to the Otsu’s method and the segmented images are overlaid on the color information to provide the practitioner with feedback about the malignancy location and spread.
How might it impact on clinical practice in the foreseeable future?
Fluorescence molecular imaging is expected to have outstanding clinical relevance, mainly as regards early detection and accurate estimation of the extent of diseases and patient-specific therapy selection and monitoring. During the FI2P4TUM project, a comprehensive work program regarding the development of acquisition systems, and modeling and interpretation algorithms of fluorescence images and videos has been carried out. The latter has been combined with the anticipation in preclinical studies of the technique based on NIR fluorescence-guidance for minimally invasive surgery as well as surveillance endoscopy. Altogether the accomplished research bridges NIR fluorescence molecular imaging and its expected outstanding relevance to clinical translation. More importantly the relevance of the impact on clinical practice of these developments and, as a consequence their socio-economic impact, will be explored in two clinical studies that will commence shortly (NL36315.042.13 and NL-43407.042.113).