Diagnosis and Monitoring of Inflammatory and Arthritic diseases using a COmbined approach Based on Ultrasound, optoacoustic and hyperSpectral imaging

Final Report Summary - IACOBUS (Diagnosis and Monitoring of Inflammatory and Arthritic diseases using a COmbined approach Based on Ultrasound, optoacoustic and hyperSpectral imaging)

Executive Summary:
Rheumatoid arthritis (RA) is a destructive inflammatory polyarthritis with a prevalence of 1-2%. The prevalence for rheumatoid arthritis increases with age and approaches 5% in women older than 55. Both incidence and prevalence of rheumatoid arthritis are two to three times greater in women than in men. Psoriatic arthritis (PSA) is another destructive arthritis which is associated with the chronic immune-mediated skin condition psoriasis vulgaris. PSA typically appears about 10 years after the onset of psoriasis, but also can develop spontaneously without earlier skin symptoms in up to 30% of the cases. It has a prevalence of 0.25 – 0.75% equally distributed between women and men. Both types of arthritis lead to joint destruction and deformation which results in loss of joint function. Since hand and finger joints are commonly affected, the progress of the disease severely affects the patient’s quality of life. Untreated or insufficiently treated arthritis often leads to disabilities including an inability to work. Furthermore, an early diagnosis and therapy is critical for avoiding long-term consequences for the patients.
To achieve early diagnosis and treatment of arthritis within the so-called therapeutic “window of opportunity”, a combined multimodal imaging approach was chosen in IACOBUS.
The IACOBUS approach is based on a hybrid system with iterative use of hyperspectral imaging (HSI) for overview screening and close-look high resolution ultrasound/optoacoustic (US/OA) imaging for detailed investigation of small finger joints. While ultrasound is well established in clinical routine, optoacoustic imaging is an emerging technique combining the benefits of optics and ultrasound. In the overview screening mode, hyperspectral imaging provides a wide-field scan of the affected hand and allows the identification of potential sites of inflammation based on the detection of local hyperperfusion.
The objective of the project was to make use of this effect for detection of hypervascularisation, which is a characteristic pathophysiological feature of arthritis.
During the IACOBUS project, significant innovative technology was developed in different disciplines such as capacitive micromachined ultrasound transducer (cMUT) probes, diode-pumped lasers, combined multichannel ultrasound and optoacoustic electronics, hyperspectral imaging devices and algorithms.
At the beginning of the project, specifications for the imaging platform consisting of the hyperspectral imaging (HSI) and the acoustic/optoacoustic imaging (US/OAI) subsystems were elaborated jointly by the consortium partners. In order to facilitate full-view tomographic imaging of the smaller finger joints, the proximal interphalangeal (PIP) and distal interphalangeal (DIP) joints, as well as a top and bottom view of the metacarpophalangeal (MCP) joints, a special probe geometry and scanning concept were developed. The cMUT-based 10 MHz probe developed by VERMON was used together with a diode pumped OPO laser system (EKSPLA), a multichannel electronics system and a mechanical scanning system (both by FRAUNHOFER) for combined 3D ultrasound and optoacoustic tomographic imaging.
With respect to hyperspectral imaging, a first-of-its-kind hand scanner was developed by the project partners NEO and NTNU. This device allows both transmission and reflection mode imaging of human hands. Advanced data processing algorithms have furthermore been developed to analyze the spectral signature of light for identification of inflamed joints. With respect to the combined ultrasound/optoacoustic system, evaluation has been performed both on finger mimicking phantoms and on probands. In a first approach, the reproducibility of the measurements (low intra- and inter-user variability) has been demonstrated. Furthermore, the performance of the device has been characterized and a resolution in the range of 150 µm could be achieved in US/OA. First measurements on probands have shown the ability of the technology to detect and visualize vasculature in fingers. The sensitivity and specificity of the technology for detection of joint inflammation in comparison to existing standard diagnostic techniques are still under evaluation.
At the end of the project, two fully integrated systems for hyperspectral imaging and combined ultrasound/optoacoustic tomography have been developed. The systems have been fully tested with respect to all relevant standards according to the MDD 93/42/EEC for being usable in a clinical environment.
Furthermore, first tests on probands have been conducted showing the systems’ usability in an in vivo setting. Images of the hands/fingers and the subcutaneous vasculature could be obtained with both of the developed technologies. Although the actual performance and sensitivity of the systems for detection of early signs of arthritis, especially when compared with currently existing gold standard methods, is still under final evaluation, a first assessment of the capabilities of the systems has highlighted their diagnostic potential.

Project Context and Objectives:
Rheumatoid arthritis (RA) is a destructive inflammatory polyarthritis with a prevalence of 1-2%. The prevalence for rheumatoid arthritis increases with age and approaches 5% in women older than 55. Both incidence and prevalence of rheumatoid arthritis are two to three times greater in women than in men. Psoriatic arthritis (PSA) is another destructive arthritis which is associated with the chronic skin condition psoriasis vulgaris. PSA typically appears about 10 years after the onset of psoriasis, but can develop spontaneously without earlier skin symptoms in up to 30% of the cases. It has a prevalence of 0,25-0,75% equally distributed over men and women. Both types of arthritis lead to joint destruction and deformation which result in loss of function. Since hand and finger joints are often affected, the progress of the disease severely affects the patient’s quality of life. Untreated or insufficiently treated arthritis often leads to disabilities including an inability to work.
Inflammatory joint disease, in particular RA, therefore causes significant costs of 3.000-5.000€ of direct and 11.000-16.000€ of indirect costs per patient and year. The functional outcome and subsequently the indirect costs are strongly dependent on the time lag between the onset of the disease and the diagnosis and thus initiation of treatment. Different studies have shown that a window of opportunity exists in the first year and especially in the first 3 months, in which a treatment has a higher likelihood for improving the course of these devastating diseases.
Clinical studies of RA have demonstrated that even a brief delay (as little as 8–9 months) in starting therapy has a significant impact on disease parameters years later. Additionally, mortality among RA patients who present early is lower than that among those who are treated late in the course of disease. Recognition that erosive changes occur early in the disease, often in the first year, has also highlighted the importance of early intervention.
The cause of RA as well as PSA is unknown, but autoimmune inflammatory processes leading to subsequent tissue destruction are already set off at the very beginning of the diseases. Early diagnosis and therapeutic intervention should therefore focus on the detection and stopping of the disease in order to prevent irreversible damage.
To initiate an adequate therapeutic response, not only an early detection of symptoms but also a precise classification of the disease is necessary. A frequent challenge is the differentiation between polyarticular osteoarthritis (OST) of finger joints and RA or PSA in their early stages. OST is a degenerative joint disease characterized by joint pain and discrete loss of function at a low level of inflammation. Conversely, RA is a disease with rapid onset of joint destruction and severe loss of joint function, exhibiting strong, exsudative inflammation. PSA frequently is also destructive to joints, but shows a less exsudative type of inflammation. This difference of clinical presentation is reflected in different therapeutic regimens. Since no definitive diagnostic method is available and symptoms of OST, RA and PSA may closely resemble each other, rheumatologists have to use patient history, physical examination, blood tests, and, most importantly, imaging techniques such as ultrasound and X-ray to establish an accurate diagnosis. A differentiation of OST, RA, and PSA in finger joints is frequently difficult due to the small size of the joints involved and due to the lack of sensitivity of established imaging tools to early changes. The therapeutic success depends on the correct distinction between the different arthritis entities. Only a precise diagnosis allows to stratify patient groups and thereby to develop a personalized treatment concept tailored to the specific needs resulting from the disease pattern of the patient. Thus, early and accurate diagnosis, combined with a fast therapeutic response is the crucial step in achieving optimal control of disease progression, therapy monitoring and prognosis as well as disease associated costs of RA and PSA. Tools for cost-effective and reliable diagnosis of early inflammatory signs in finger joints are therefore of highest importance to ensure a successful therapy, to reduce the loss of quality of life for the patient and to decrase the disease-related indirect costs.
At an early stage of the disease, arthritic inflammations become noticeable through enhanced development of microvascularization of the synovial tissue inside the joint capsule. Accordingly, there is a strong need for an imaging modality which facilitates visualization of this inflammation-induced hyperperfusion. Such an image-based tool would not only be suitable for early diagnosis but would further provide a sensitive tool for monitoring of arthritis treatment.
Since the detection of early symptoms such as finger joint hyperperfusion is critical to catch the therapeutic window of opportunity, the objective of IACOBUS was to provide a novel image-based diagnostic and monitoring approach for arthritic inflammation of finger joints, which exceeds the existing standards in terms of image resolution and sensitivity and thereby allows an earlier and more reliable diagnosis. For being widely usable as a screening tool, only methods allowing non-invasive and cost-efficient imaging were considered.
Currently used imaging modalities for the diagnosis of inflammatory joint diseases such as X-ray, (Doppler-)ultrasound (US), scintigraphy and magnetic resonance imaging (MRI) have drawbacks, as they require ionizing radiation, have a high variability between users (especially Doppler/US), or high costs preventing their use as standard tool. Also, the analysis of finger joints with US and MRI is less sensitive than the analysis of larger joints due to their small size and resulting difficulties in achieving adequate tissue contrast and proof of inflammatory hyperperfusion.
To achieve early diagnosis and treatment of arthritis within the window of opportunity, a combined multimodal imaging platform was developed within IACOBUS.
The IACOBUS concept is based on a hybrid imaging system with iterative use of hyperspectral imaging for overview screening and close-look high-resolution optoacoustic/ultrasound imaging for detailed investigation of suspicious joints. While ultrasound is well established in clinical routine, optoacoustic imaging is an emerging technique combining the benefits of optics and ultrasound. When irradiated with short laser pulses, biological tissue generates broadband ultrasound waves with amplitudes proportional to the local optical absorption coefficient. Optoacoustic imaging therefore allows non-invasive in-vivo imaging with acoustical resolution and the high contrast of optical modalities.
In order to take advantage of the intrinsic benefits of the involved imaging modalities, the proposed approach is based on a dual examination procedure. In the overview screening mode, hyperspectral imaging provides a wide-field scan of the affected hand and allows the identification of potential inflammation sites based on the detection of hyperperfusion. In the close-look mode for detailed investigation of small joints, high sensitivity for visualization of inflammation related vascularization is provided by optoacoustic imaging, which already has shown its suitability for high contrast imaging of vasculature. Optoacoustic imaging benefits from proven sonographic image formation procedures but permits to overcome the low acoustic contrast of soft tissue by using the target structures (e.g. microvasculature) as sound sources themselves. Therefore, the goal was to use optoacoustic imaging for three-dimensional high resolution imaging of arthritis-related (micro-)vascularization in affected joints with enhanced resolution. For imaging with intrinsic tissue contrast, the sensitivity of the device was tuned to the optical spectra of oxy- and desoxyhemoglobin through appropriate choice of the wavelength used. The strong laser light absorption in hemoglobin leads to the generation of high-amplitude optoacoustic signals. Based on the strong absorption contrast between hemoglobin and surrounding tissue, the close-look optoacoustic mode will allow a precise image of the hypervascularisation that is a characteristic pathophysiological feature of arthritis. In addition, acoustic imaging (ultrasound) provides information on the anatomy (osteochondral information) for characterization of bone erosion.
In addition to a “native contrast mode”, where imaging is performed based on the intrinsic tissue contrast, the subsequent application of the approved fluorescence contrast agent ICG (Indocyanine green) could facilitate retrieving further diagnostically relevant information. Although the usage of contrast agents such as ICG has not been considered in IACOBUS, the systems were designed so that molecular imaging can be included in a diagnostic process at a later stage. In particular, optoacoustic signals can be generated at different wavelengths, such that signals that are characteristic for defined contrast agents (based on their spectral signature) can be detected and localized. Thereby, optoacoustic and hyperspectral imaging allow obtaining both morphological and functional information from the inflammation-induced hypervascularization.
With regard to the clinical usability of the developed platform, the imaging systems were designed so that data acquisition can be performed in a standardized way in both the hyperspectral and the US/OA mode. Automated image acquisition can be performed after defined positioning of the patients hand by fast scanning of the respective probe (hyperspectral sensor and US/OA transducer respectively). A high acquisition speed in US/OA mode is guaranteed by the usage of multichannel electronics platforms allowing simultaneous read-out of various sensor elements in all imaging modes. This assures a high reproducibility and minimizes motion artifacts. Since there is no dependency of the optoacoustic or hyperspectral signal amplitude on the respective orientation between the investigated vessels and the probe (as in Doppler US), more objective data with lower inter-observer variations can be obtained.
Furthermore, the validation of the proposed technology is an essential aspect of the IACOBUS concept. The new platform will undergo a strong validation procedure prior to first clinical use. In order to allow a later uptake of the new diagnostic approach in clinical routine, the project concept further includes a certification of the device according to all European legal standards (MDD 93/42/EWG). During this testing, which is performed by independent certified laboratories, different aspects such as electrical safety, electromagnetic compatibility, acoustic and optical safety are addressed. These tests preceding the actual assessment of the IACOBUS platform in a realistic environment guarantee, that the procedure is risk-free both for the probands and the users.

Project Results:
Hyperspectral imaging system:

A self-contained, non-invasive hyperspectral imaging system (HSI) for identifying inflamed finger joints that visualizes arthritic inflammation by means of capturing, processing and analyzing spectroscopic information in the VIS/NIR region from a patient’s hand was developed within IACOBUS. The main advantage of this approach is an increased capability for identifying the inflammatory process at its earliest stages in a simpler, faster and/or more reliable way than the currently used methods. HSI takes advantage of the unique way in which different materials interact with light. In the case of viable human tissue, particularly the complex tissue of the hands, there are different properties that limit the spectral range of such interactions. For our study the most relevant properties are are those which affect transmittance, reflection and diffusion of incoming light. Those properties are such that some ranges of wavelengths of light are favored and others not. Because one of the major guidelines of this project is to produce a non-invasive and non-damaging system, the spectral range is limited to the visible (VIS) and near infrared (NIR) regions (figure 2). This is composed by non-ionizing radiation of low energy that at moderate levels or short exposures causes no damages and none or tolerable discomfort in patients.
The type of interaction of light with tissue as well as the spectral range (defined in terms of the shortest and longest wavelengths of the light used) heavily influences the overall approach to collecting the spectroscopic information. Although there are different technologies capable of collecting light at different wavelengths and with varying degrees of spatial and spectral resolution, in most cases the generated data is stored as a hyperspectral cube. This structure contains the spatial information of the scene (2 spatial dimensions of the cube) as well as the spectral information associated with each pixel in the scene (1 spectral dimension). In this way it is possible to establish a classification and search for algorithms to differentiate and identify different parts of a scene based on the spectral differences. The level of detail and precision that can be achieved by any of the possible technologies used is limited by several factors such as the collective light throughput of the optical system, the electronic sensitivity of the sensor to different wavelengths, the size of the structures in the scene, the amount of light of different wavelengths available from the scene, the calibration of the instrument, etc. Also, the algorithms used for processing the data play a crucial role in extracting useable information from the measurements. It should also be mentioned that none of these data are relevant if there is no correlation between the observed spectroscopic interaction and the medical interpretation of the results. Thus, a close collaboration is needed between the hardware and software development and implementation stages and the medical end users.
The development of a hyperspectral sensor adequate for use in a clinical environment and suitable for measuring the spectroscopic characteristics expected to be associated with the inflammatory process of arthritis consisted of several stages. The first step was to determine which spectral range has to be used for the measurements. From previous experiments and projects carried out by NEO and NTNU, it was known that the spectral region between 400 nm and 1600 nm is well suited for analyzing biological samples. To refine the range further, a series of brief exploratory experiments were carried out at NTNU in collaboration with St. Olav’s University Hospital (Trondheim, Norway). In these experiments, the hands of some of the members of NEO’s and NTNU’s IACOBUS groups were exposed to illumination by different light sources. The images were taken using different models of NEO’s hyperspectral camera system HySpex to cover the spectral range between 400 nm and 2500 nm. The images were analyzed to identify the spectral regions that contained more information for both light reflectance from and light transmission through the tissue of the hands, particularly the tissue around the joints.
The most promising spectral region was experimentally found to be 600 nm to 1200 nm. In this region it was shown that high spectral contrast between highly absorbing and weakly absorbing tissue constituents could be achieved for different wavelengths. Once the spectral range was defined it was possible to start designing the optical system. First it was decided which type of sensor array to use. Conventional silicon based sensors are not suitable because they are not sensitive for light of wavelengths beyond 1000 nm. More exotic materials such as Indium-Gallium-Arsenide alloy (InGaAs) are used in the 900-1700 nm region with good results but cannot be used directly for the shorter wavelengths. However, InGaAs technology can be extended to 400 nm - 1700 nm by different manufacturing means and there is a small range of commercially available sensors with these properties.
Several optical designs were developed in detail using computational modelling and simulations. During the design of the optical system, the actual spatial resolution that could be achieved with the imaging system was investigated. Originally, it was planned to have full hands imaged in a single scan and having a spatial resolution of 100 µm. For this to be suitable for the clinical environment (in which patients might be suffering from conditions that limit their mobility or capability to remain still), the scan must be done along the direction of the fingers. This configuration places the width of the hand perpendicular to the movement of the camera. Literature was consulted to identify the average size of hands and it was found that the deviation in the width was too large (from 10 to 17 cm in adults) to make it practical for a single scan. Also, this imposes a minimum size of the sensor of [17 cm / 100 µm per pixel = 1700 pixels] in its longer direction, which is 2.5 times larger than the selected sensor. It was also pointed out that having two scans would take longer and increase the chances of having mismatching or incomplete data due to the movement of the patient between scans. Reducing the field of view (FOV) to 4 fingers (no thumb scan) allowed refining the size of the FOV of the system to 10 cm, which results in a spatial resolution of 160 µm when using a 640 pixel sensor. After the definition of the optical and sensor requirements, a robust mechanical design was implemented to ensure thermal stability during use and transportation.
The concept of the whole imaging module was designed following ergonomic principles and emphasizing on ease of access and functionality at the clinic. The pushbroom imaging system requires that there is a relative movement of the camera and the scene. It was decided to have the camera moving and maintain the hand fixed in order to reduce the chances of spatial mismatches in the scene. To allow flexibility in the measurements during the clinical trials (WP6 and WP7) it was decided that both reflectance and transmission measurements were needed. It was also determined after the early tests by NTNU that an autofocus system was needed, which requires vertical adjustment of the distance between the hand and the camera. Once again it was chosen to move the camera instead of the hand. At a later stage it was considered that having the capability of acquiring a 3D image of the hand while collecting the hyperspectral data could be beneficial for mapping the inflammation process. A 3D imaging module is under development. Finally, the instrument is controlled by software developed by NEO which also collects the spectral data from the measurements. This is done by a high-end computer specified by NEO and NTNU and assembled at NEO.
For the illumination of the two chosen imaging modalities two independent illumination configurations are required. Once the FOV was defined as 10 cm, a universal hand support was designed and fabricated that integrated the light sources that would be used for the transmission measurements.
Each lamp has 2 fiber optic guides that direct the light directly on top of or under the hand in a line pattern. In the case of the reflectance configuration, the light is focused perpendicular to the hand (and parallel to the IFOV of the camera) in two overlapping 1cm illumination fields. This prevents overexposure and overheating of the patient during the duration of the scan. The illumination guides are mounted onto the camera such that the light moves along when the camera is scanned.
Once hyperspectral data cubes are acquired, post-processing and analysis algorithms need to be run. Different techniques have been explored to prepare the data for analysis in real time. Such techniques aim to reduce the noise in the images, eliminate the regions showing unphysical data and the conversion of HSI to reflectance or transmittance.
After the analysis of the data from the pre-trial at St. Olav’s Hospital, three parameters have been identified as potential indicators of arthritic diseases in finger joints: Vasculature density, blood content and transmission. Each of these parameters is suspected to contribute with a different weight to the probability of the patient developing arthritis and further data analysis could be used to determine each of the coefficients. The techniques used for identifying such parameters are inverse diffusion model, wavelet analysis algorithm, vessels extraction algorithm and statistical algorithms based on minimum noise fraction (MNF).
For setting up the hyperspectral imaging system in a clinical environment, a high level of automation regarding the interaction between the individual components was necessary. For this purpose, significant effort was put in the development of a common software interface for data acquisition, pre-processing and display. Various hardware is controlled in the IACOBUS setup to enable a successful scan:
• The hyperspectral camera
• Vertical translation stage for focusing, horizontal translation stage for scanning
• Reflectance light source, transmittance light source
• Autofocus camera for acquiring focused information
In addition to software integration, the final integration of the different mechanical and optical components of the hyperspectral imaging system was a major task. The instrument consists of 2 main components, the processing module and the imaging module. The processing module is the smaller cabinet containing a data acquisition unit and the necessary hardware/software to control the imaging module. It also contains the main power inlet and the general power supply for the entire system. The main power supply is a medical grade isolating transformer guaranteed to protect the electric network and the users form accidental electric shock and external influence. This component also features the data ports for the peripherals needed for user interaction.
The imaging module is the larger cabinet and contains the instruments necessary to generate the hyperspectral images of the fingers (thumbs excluded) on each hand in the transmission and reflectance modalities. It also contains a laser-based 3D-profiling subsystem used to generate a 3D model of the hand. Power and control capabilities are supplied from the processing module to the imaging module by means of dedicated data ports and power inlets. All the power supplies used for operating all components are medical-grade and satisfy the relevant directives. The materials used for the structure and enclosure as well as the assembly of both modules are chosen such that it ensures a negligible electromagnetic interference with/by external electric devices.
The hyperspectral system has passed all the internal tests carried out by NEO and NTNU and has been used with volunteers to evaluate its performance. The hardware and the operational software are fully integrated and completed. The system has been designed in compliance with ISO 14971 standard and the relevant parts of the EN 60601 directive and is ready for certification tests.
In the final version of the system, the scanning process provides high-resolution hyperspectral information on the properties of the hand as registered by the transmission and reflectance modes. The device has been validated both on phantoms and in proband tests.
Data from probands has been acquired and processed with different algorithms (in particular vessel-enhancement filters) for improved visibility of vascular structures. The screening of vessel-enhanced images has shown that it is possible to obtain vessels directly from the transmittance images. However, the physiological state of the imaged hand might influence the clearness of the vessel images. Individual variation might also influence the results.
Comparable measurements were also repeated in the reflectance mode in order to assess the suitability of the imaging mode for clinical investigations and the influence of imaging parameters such as the integration time on the final image quality.
After measurements have been performed both in transmittance and reflectance mode, it can be stated that the systems shows sufficient image quality in both modes with high resolution in both spatial and spectral directions. Convincing images of the hand vascularization could be obtained thanks to the signal processing and image filtering images developed by NTNU. Furthermore, the integration times are short enough, such that no discomfort for users and patients is given during the measurement.

Ultrasound/Optoacoustic imaging system:

The combined US/OA imaging system consists of several components. While an OPO laser system is used for signal generation, the signals are detected with a special cMUT based probe, digitized and processed with a multichannel electronics system. A mechanical translation scanner is used for translation of the probe for acquiring 3D tomographic data. These different components are described below.
Optoacoustic signals are generated when light is absorbed by biological tissue. The spectral and temporal properties of the light used for signal generation have a strong influence on the characteristics of the latter. For instance, the amplitude of optoacoustic signals is directly proportional to the fluency of the laser pulse. Furthermore, the duration of the laser pulse determines the frequency spectrum of the generated acoustic signal. The spectral properties of biological tissue are another essential boundary condition when specifying a laser system dedicated to optoacoustic imaging. In particular, the absorption of tissue chromophores limits the penetration of light in tissue. For this reason, a spectral range between 600 and 1100 nm, the so-called optical window, is mostly preferred. With respect to the time profile of the laser light, a pulsed emission is required to generate optoacoustic signals. In order to obtain high resolution images, as it is required for diagnostic applications, a laser pulse duration in the nanosecond range is needed. In addition, a laser should have a high repetition rate, such that real-time imaging is possible, and should be synchronizable with external devices. Of course, safety requirements resulting from the medical device directive MDD 93/42/EEC and associated standards in terms of electrical safety, electromagnetic compatibility and optical safety need to be considered if the laser is used in a clinical study.
These boundary conditions have been considered when defining the laser system developed in IACOBUS. Accordingly, an OPO laser system based on a diode pumped frequency-doubled Nd:YAG was developed. This design allows generating wavelengths in the mentioned spectral range and to adjust them with a high resolution. In such a system, a laser pulse with a wavelength of 1064 nm is generated first. A frequency doubling crystal is then used to convert the pulse into light of 532 nm, which then is used as input to the OPO (optical parametrical oscillator). The latter is responsible for generation of light of the desired wavelength and can be tuned by changing the incidence angle of the 532 nm laser beam onto the OPO-crystal. The tasks in developing such a laser system include not only the efficient arrangement of the optical components. Driving electronics, a cooling concept and a software for control of the overall system also need to be developed.
In addition to the aforementioned requirements, one challenge in developing the laser system was achieving a high pulse energy. Given that the amplitude of optoacoustic signals is proportional to the optical fluency (and thereby to the pulse energy when the irradiation surface is constant), a high sensitivity can only be achieved when high pulse energies are available.
When a first laser prototype was set up, the characterization of the system was started. In a first step, the beam profile was investigated. This is of high relevance since the light has to be transferred to the region of interest (ROI) by means of optical fibres. An inhomogeneous beam profile with hot spots has the potential to damage the fibres. Accordingly, a good mode distribution inside the beam was essential. Long-term pulse stability tests were performed as well and could confirm that the system works according to the specifications.
Once the laser development was completed, it needed to be integrated with the other components. As described previously, light transport is performed by means of optical fibres. According to the tomographic detection and imaging geometry, probes are placed on 4 sides (above, below, left, right) around a finger that is imaged. Accordingly, light delivery needs to be performed from these directions.
For this purpose, the laser was redesigned according to the needs of beam splitting to four channels and the need to transport four laser pulses to the illumination zones. The laser was disassembled and its frame was adapted for acceptance of additionally designed and manufactured beam splitting devices and optical cables.
Different options for light transport were investigated. Either a transport through a single fibre or through fibre bundles can be considered. In both cases, given the low damping of optical fibres, the actual transport of the light is less of an issue than the coupling into the fibre and the targeted delivery at the distal end of the fibre/bundle.
While transport through a single fibre provides higher efficiency (fewer losses during coupling of light into a fibre), the usage of fibre bundle leaves more degrees of freedom regarding the illumination pattern at the distal end.
Therefore, the following approach was chosen. Light was coupled into single fibres. This does not only allow high coupling efficiencies but also ensures homogenization of the beam. In an intermediate step, light is then coupled into a fibre bundle. The proximal end of the fibre bundle consists of fused fibres, which increases the coupling efficiency, reduces losses and improves the thermal stability. At the distal end of the bundle, a special output geometry providing the required illumination pattern and guaranteeing that the illumination zone overlaps with the focal zone of the acoustic probe was realized.
When it comes to the probe itself, a new concept based on cMUTs was used in IACOBUS. cMUTs are capacitive micromachined ultrasound transducers, where each element is composed of several hundreds of small suspended silicon-type membranes, which can transmit and receive ultrasonic waves around their mechanical resonance frequency. In contrast to classical piezoelectric transducers, a cMUT has a larger bandwidth, a larger directivity and, if a good electrical impedance matching has been achieved, a good sensitivity. All of these characteristics make the cMUT device a good candidate for high-sensitivity detection of optoacoustic signals and thus, the best candidate for the IACOBUS setup.
The probe consist of 768 elements in total divided into 4 curved sub-probes with an angle of 90 ° each and different curvature radii. Two “large arcs” consisting of 8 cMUT apertures of 32 elements are designed to be placed above and below the finger and two “small arcs” of smaller curvature radius with 4 cMUTs apertures of 32 elements are placed at each side of the finger. The cMUTs themselves are components developed with a microelectronic process. Accordingly, it was necessary to define the process flow as first step of cMUT development in partnership with our Si-foundry company.
The process flow was composed of 12 different steps and needed the design of 5 different photolithographic masks. Moreover, the thicknesses of the different materials was defined by taking into account a process risk evaluation (for instance, impact of the stress depending on the thickness) and their compatibilities with the specifications of IACOBUS cMUT probe.
The design was determined with the help of a multiphysics modelling software based on a finite difference scheme. This model can take into account 3 different physics domains: mechanics (for the vibration of the membrane), electrostatics (for the applied excitation voltage) and acoustics (for the radiation of the membrane in fluid and coupling between the different membranes). The ideal configurations for the cMUTs were then defined based on such multiphysics simulations.
After the fabrication of the cMUTs, a characterization was performed in two steps. In the first step, the geometry of the cMUTs was controlled by means of optical microscopy. In particular, the size of the cavities, of the electrodes and the occurrence of defects was verified. In the next step, an electromechanical characterization was made by impedancemetry.
Once the characterization guaranteed that the cMUTs fulfil the specifications, it was proceeded with the integration and the interconnection. Specific boards allowing the preamplification of the signals and the impedance matching between the cMUT elements and the digitizing electronics were developed and integrated into the probe. Finally, the probe housing was developed with the help of a CAD software. Materials for the housing were chosen according to both stability and biocompatibility considerations (in view of the usage of the probe in a realistic clinical context).
While the task of the probe is the detection and acquisition of laser-induced signals in the optoacoustic mode, and the emission and acquisition of acoustic waves in the ultrasound mode, the multichannel electronics system is needed for amplification, digitization and processing of the signals. The system furthermore ensures the communication and synchronization with the laser system and the mechanics. The platform integrates transmit circuitry for imaging in the conventional ultrasound mode as well as receiving electronics usable both in ultrasound and optoacoustic mode.
The design of the system is based on a universal modular approach. That means that individual components of the system can be replaced and the system can be adapted to the special needs of the application.
The multichannel electronics system, which is based on FRAUNHOFER IBMT´s multichannel beamformer DiPhAS (Digital Phased Array System), consists of power supplies, a housing, front-end boards, the mainboard, a multiplexing board and an internal PC. The components of the system are described briefly below.

Computer:
The PC is in the same housing as the DiPhAS system. All data processing is performed in software. Time critical tasks (image reconstruction) are performed in a graphics card in order to ensure the real time capabilities of the system. For this purpose, an NVidia GTX consumer graphics adapter with many CUDA cores was chosen for implementation in the system. Furthermore, the software with the GUI (graphical user interface) allowing to control the system, visualize and store the data runs on the PC.

Mainboard:
The main function of the mainboard is the communication with the FE-Boards (Front-End Board) and with the PC. For instance, it provides all commands to single FE-Boards and transmits the relevant operational parameters for the different tasks (e.g. sampling rate, transmit parameters, amplification parameters). It further handles the real-time system (sequence) control and the transfer of ultrasound data from the FE-Boards to the PC. Furthermore, sorting the data in order to obtain a format that is suitable for transfer via Gigabit Ethernet is one of the functions executed on the main board.
The mainboard has slots for 16 FE-Boards handling 8 channels each. For data processing tasks, a powerful available FPGA chip is used. Data from all channels are transferred to the FPGA through 64 serial lanes and are converted there to a parallel format. A microcontroller is furthermore integrated into the FPGA for control tasks. A control bus is available for control of all FE-Boards on one hand. Communication with the PC is performed via a Gigabit Ethernet Inferface.

Power Supply:
The primary power supply for the US hardware is a +24 V, 4.2 A medically certified power supply. A secondary custom-made power supply integrated in the housing of the electronics supplies the US hardware with the needed voltage and power levels. It generates the different voltages required by the system components (PC, microcontroller, pulser module).

Front end:
The frontend boards implement the transmit pulse sequencing, data transfer management and data pre-processing. They have several functions with respect to generation and acquisition of ultrasound signals. A communication protocol is established between front board and mainboard through which commands and digitized data (ultrasound signals) are transferred. The front board generates the transmit pulses according to the parameters obtained from the mainboard. The range of parameters includes voltage, delays or signal shape (modulation, burst count). In addition, the front end handles the received signals. They are first amplified and digitized according to the settings provided by the mainboard (respectively gain/TGC and time/bit resolution). After digitization, data are buffered before the transfer to the mainboard is performed. In addition, data processing (e.g. averaging through accumulation of consecutive samples) can be performed on the front ends.
One Front End Board (FE-Board) contains 8 transmit and 8 receive paths. Transmit data are received from the mainboard and are prepared for transfer to a pulse module.
In the receive path, signals from 8 transducer elements are digitized in a 12 bit AD converter.

Transducer connector board/multiplexer board:
The multiplexer board was needed since the final specifications of the probe consist of several sub-probes having a total of 768 numbers. The multichannel electronics system has 128 “true” transmit and receive channels. For this reason, a multiplexer board allowing distributing the 768 elements onto the 128 channels (by switching after an image sequence) was needed.
It allows to distribute the signals from the 768 elements of the probe to the 16 available FE-Boards (each handling 8 signals) adequately. It further has some mechanical function since it ensures the mechanical connection of the ITT cannon connector to the DiPhAS hardware.

The assembled boards (multiplexer, front end board, mainboard) have been integrated together with the custom made power supply and the computer in one single housing. The lower part of the housing is reserved for the PC and the power supply, while the upper part is reserved for the acoustic system components.

Once the development of the multichannel electronics system had been completed, the IACOBUS team could proceed with the integration of all components into a single setup. In particular, given the boundary condition of performing combined US/OA 3D tomography of all three finger joints (MCP, DIP, PIP), the mechanical integration was a challenging task.
The mechanical setup consists of a water tank, in which both the probe and the hand need to be immersed in order to ensure acoustic coupling, a hand and fingertip support to reduce motion artefacts, and the actual mechanical scanner which ensures the displacement of the probe along the finger. The mechanics were manufactured under strong consideration of safety aspects. For instance, biocompatible materials were used and the scan path was chosen such that no squeezing of the finger or the hand is possible. The translation stage was furthermore chosen such that a scanning path of 150 mm is possible, which guarantees that data from all finger joints can be acquired in a single scan step without the need for repositioning of the hand.
After the successful integration of the system, first tests were performed to assess the performance. In particular, the image quality and the inter- and intra-user variability were investigated. When it comes to image quality, the resolution of the system strongly relies on the used reconstruction algorithm. In particular in the acoustic mode of the device, different beamforming techniques (plane wave compounding) were investigated with the goal of optimizing the resolution. After different tests and optimization steps, an in-plane resolution in the range of 160 µm could be achieved both in the acoustic and the optoacoustic mode. In addition to internal tests by the IACOBUS team members, a testing was performed by certified laboratories to confirm the compliance of the device with the relevant national and European standards. In the case of the US/OA components, the compliance with IEC 60601 (electrical safety and electromagnetic compatibility), IEC 60285 (optical safety) and IEC 60601-2-37 were certified, such that all requirements for an assessment of the device’s performance in a realistic clinical setting are fulfilled.
The last phase of the project was dedicated to the validation of the technique. Just as for the hyperspectral imaging system, measurements were first performed on phantoms and on probands in a second step. This was done in order to prove the usability of the device in a realistic (clinical) setting and to demonstrate the ability of the system to image the diagnostically relevant structures, such as the finger bones and joints (in ultrasound mode) and the (micro)vascularization (in optoacoustic mode).

Potential Impact:
Dissemination activities

The IACOBUS technology has provided significant opportunities for scientific and commercial exploitation. The developed technology can be used to answer different scientific questions, which go beyond the topics addressed in IACOBUS. Therefore the IACOBUS consortium communicated the project results as foreseen in scientific papers, on congress contributions and presented the prototype at international fairs.

The IACOBUS consortium has undertaken significant efforts to disseminate the project results among all stakeholders. Besides numerous contributions for peer-reviewed journals and conference presentations, the results were presented at fairs and events open to a wider public.
In particular, the hyperspectral imaging system was exhibited at the Technoport Exhibition in Trondheim. The exhibition was co-located with the Technoport conference 2016. The Technoport website describes the conference as follows: “The Technoport conference is an annual international conference on innovation, entrepreneurship and commercialisation that aims to smash together the knowledge quality and mastery of complexity from the academic world with the rapid and agile workflow of startups in an attempt to solve global challenges.” The conference is hosted by the Norwegian University of Science and Technology with partners such as SpareBank 1, Sør-Trøndelag county, the municipality of Trondheim, Statoil, Innovation Norway and SINTEF.
On the second day of Technoport 2016, the exhibition area opened for a wider public to discover promising prototypes and products from Trondheim and beyond. The exhibit was intended as a look and feel exhibition where people could get hands on experience with the presented products and prototypes. The technology trends represented at the exhibit were GreenTech, HealthTech and Robotics. IACOBUS was invited by the organizing committee to have a booth. The booth was well visited, and the visitors were allowed to have their hands scanned by the prototype. The booth was also featured from the main stage by one of the invited speakers. The audience consisted of people from academia (students, researchers etc), companies, investment funds, banks, and the general public as the event was open to anyone who registered.
The combined ultrasound/optoacoustic IACOBUS system was presented in autumn 2015 at the MEDICA (Düsseldorf, Germany), which is the world´s largest trade fair in the field of medical devices. The fair addresses all stakeholders. IBMT´s booth raised strong interest and was visited in particular by industry, medical professionals and students. In addition, the project coordinator Dr. Marc Fournelle (Fraunhofer IBMT) was interviewed on the IACOBUS booth during MEDICA. The feature was aired on Forschung Aktuell (Deutschlandfunk) and the scope and results of IACOBUS could thereby be presented to an even wider national audience.
Further dissemination activities include three press releases, a public website maintained during the project lifetime and a project leaflet distributed at fairs, local and international demonstration and scientific events . All partners were involved in at least one dissemination activity. The dissemination rules as agreed in the Consortium Agreement were considered. TP21 set up the public website www.iacobus-fp7.eu at project start to inform the general public and the specialist on the project. The website was updated regularly with news, publications and press releases. The IACOBUS consortium implemented 21 dissemination activities and five peer-review publications in 42 months. One additional peer-reviewed publication is in preparation.

Exploitation:
The coordinator as well as the industrial and clinical partners have a strong interest in sustainable exploitation of the entire system but also of the device modules. In the frame of an exploitation workshop the exploitation opportunities of the partners have been discussed. Partners decided to list their achievements and exploitation options and to discuss future collaboration opportunities.

At the end of the project, two fully integrated systems for hyperspectral imaging and combined ultrasound/optoacoustic tomography have been developed. The systems have been fully tested with respect to all relevant standards according to the MDD 93/42/EEC for being usable in a clinical environment.
Furthermore, first tests on probands have been conducted showing the systems’ usability in an in vivo setting. Images of the hands/fingers and the subcutaneous vasculature could be obtained with both of the developed technologies. Although the actual performance and sensitivity of the systems for detection of early signs of arthritis, especially when compared with currently existing gold standard methods, is still under final evaluation, a first assessment of the capabilities of the systems has highlighted their diagnostic potential. In addition to the scientific impact, the project has led to significant exploitation options for the involved partners. The expertise generated with respect to cMUT technology and especially regarding integrated electronics for impedance matching and pre-amplification developed by VERMON will allow to improve the performance of future cMUT-based products. The laser developed by EKSPLA within IACOBUS has already been released as a new product. This diode-pumped OPO represents a new benchmark in terms of high PRF performance. In addition, IACOBUS has allowed NEO to expand their expertise to the biomedical field by developing a hyperspectral imaging system being close to market maturity. FRAUNHOFER could strengthen its expertise in acoustic imaging by developing a first-of-its-kind tomography system combining optoacoustics and ultrasound. The technology developed, especially with respect to dual-imaging mode multichannel electronics, is of highest relevance for future applications involving multimodal imaging. Finally, NTNU has developed algorithms for automated analysis of hyperspectral data of biological tissue. This could not only be published in renowned scientific journals, but will also serve as a basis for future developments with respect to automated disease-recognition in optical (hyperspectral) data. Through the tests conducted with both the hyperspectral and the combined US/OAI technology, the partners JLU and NTNU (through the collaboration with St. Olav’s Hospital Trondheim), have contributed to making new sensitive, non-ionizing and non-invasive diagnostic imaging modalities available to patients. The performed tests have demonstrated the usability of the technologies in a realistic clinical environment and have underlined the potential of the IACOBUS approach for an improved diagnosis of arthritis. The IACOBUS consortium will jointly seek for opportunities to pursue this research further with the ultimate goal to transfer the developed technologies from a research environment to clinical routine.

List of Websites:
www.iacobus-fp7.eu

Coordinator:
Fraunhofer Institute for Biomedical Engineering IBMT
Joseph-von-Fraunhofer-Weg 1
D-66280 Sulzbach

Dr. Marc Fournelle
E-mail: marc.fournelle@ibmt.fraunhofer.de

Final Report Summary - IACOBUS (Diagnosis and Monitoring of Inflammatory and Arthritic diseases using a COmbined approach Based on Ultrasound, optoacoustic and hyperSpectral imaging)

Share this page

Download