Periodic Reporting for period 1 - MULTIPLE (Multimodal spectral sensors and orchestrated deep models for integrated process optimisation)
Berichtszeitraum: 2019-12-01 bis 2021-05-31
MULTIPLE will develop cost-effective multimodal monitoring solutions with a breakthrough impact on production quality and efficiency. It will bring cutting-edge OLED-based sensors, snapshot hyperspectral filters and dual-aperture imaging to deliver cost-effective spectrometers and camera cores in a broad VIS/SWIR range, complemented with cost effective laser-based chemometric sensors in the MWIR. On top of these sensors, MULTIPLE will develop novel multimodal monitoring systems that will be IoT native, by combining them with cloud, big data, and deep learning for agile development and orchestration of complex AI-based models to optimize production improving EU manufacturing competitiveness. MULTIPLE overall objectives of the project are:
o To develop cost-effective HSI SWIR (0.9-1.7 μm) camera cores ready for volume production. To develop a compact and cost-effective dual-aperture HSI imager combining snapshot CMOS and InGaAs hyperspectral sensors based on mosaic filters covering the whole VIS/SWIR range (0.4-1.7 μm).
o To develop novel cost-effective spectrometer solutions for highly specific chemometric analysis in an extended VIS/MWIR range (0.4-3.5 μm wavelength), based on innovative OLED sensors and laser-based spectroscopy.
o To develop embedded deep models for regression, classification and RT control. It will guarantee onsite and online measurements and classification of different types of categories with high flexibility, accuracy, and reliability.
o To develop an integral and scalable approach to process monitoring, quality assurance, and process optimization through the orchestration of photonic monitoring systems under an IoT native approach. The aim is to leverage SoA cloud and big data technologies to create new services and microservices enhancing decision support tools.
o To demonstrate MUTLIPLE in 3 manufacturing scenarios: steelworking, woodworking and food.
o To develop cost-effective HSI SWIR (0.9-1.7 μm) camera cores ready for volume production. To develop a compact and cost-effective dual-aperture HSI imager combining snapshot CMOS and InGaAs hyperspectral sensors based on mosaic filters covering the whole VIS/SWIR range (0.4-1.7 μm).
o To develop novel cost-effective spectrometer solutions for highly specific chemometric analysis in an extended VIS/MWIR range (0.4-3.5 μm wavelength), based on innovative OLED sensors and laser-based spectroscopy.
o To develop embedded deep models for regression, classification and RT control. It will guarantee onsite and online measurements and classification of different types of categories with high flexibility, accuracy, and reliability.
o To develop an integral and scalable approach to process monitoring, quality assurance, and process optimization through the orchestration of photonic monitoring systems under an IoT native approach. The aim is to leverage SoA cloud and big data technologies to create new services and microservices enhancing decision support tools.
o To demonstrate MUTLIPLE in 3 manufacturing scenarios: steelworking, woodworking and food.
i. Achieving a consortium wide understanding of the 3 use-cases AS-IS scenario (end-users status at the beginning of the project) and the TO-BE scenario (desired end-users status at the demonstration phase), as well as the requirements needed to accomplish that level. MULTIPLE system architecture was defined.
ii. Initial data acquisition at ‘lab-scale’ with early available sensors (M3-M9), new data collection with project imagers and spectrometers at ‘pilot-scale’ (M10-M16) and data acquired with final MULTIPLE prototypes in line (M16-M18, to be continued until M24) for models further development and optimization.
iii. Development of laser-based spectrometers for combustion gases (CO +O2) measurement in LDL use-case and acetone emissions detection in ROYO lacquering line.
iv. Development of snapshot spectral filters for the InGaAs VGA detector and optoelectronic integration of the InGaAs-based cameras cores in the SWIR spectral range.
v. Development of the dual band VIS-SWIR HSI imager consisting in a dual aperture snapshot camera covering the whole VIS-SWIR range.
vi. Development of OLED-based spectrometers to be integrated in the monitoring system of JOTIS for particle size and viscosity measurement.
vii. Preliminary thermography and colorimetry models. First thermography model for measuring high temperatures (500ºC-1000ºC) from a SWIR HSI camera (900-1700nm). Implementation of 2 colorimetry models for estimating chromatic coordinates (CIE XYZ) from a HSI pushbroom camera (370-900nm) and MULTIPLE HSI snapshot SSM4x4 mosaic camera (470-620nm): direct linear fitting and reflectance-based model.
viii. Development of edge computing devices for data capture from the HSI sensors developed in MULTIPLE and other factory field buses, HSI images processing for calibration and dimensionality reduction purposes and cloud connection, adopting RAMI4.0 protocols as standard way to interact with the devices. Cloud data messages and ingestion system has been already designed to transfer and store HSI images and ease the development of AI algorithms and manufacturing optimization tools.
ix. Significant activity delivered in terms of dissemination and exploitation tasks. During this period, the project website was created (https://multipleproject.eu/) and all the communication material was prepared. In terms of exploitation, the first Exploitation Plan of the project was submitted, including the list of Key Exploitable results identified, with its preliminary IP information and business intentions from each partner.
ii. Initial data acquisition at ‘lab-scale’ with early available sensors (M3-M9), new data collection with project imagers and spectrometers at ‘pilot-scale’ (M10-M16) and data acquired with final MULTIPLE prototypes in line (M16-M18, to be continued until M24) for models further development and optimization.
iii. Development of laser-based spectrometers for combustion gases (CO +O2) measurement in LDL use-case and acetone emissions detection in ROYO lacquering line.
iv. Development of snapshot spectral filters for the InGaAs VGA detector and optoelectronic integration of the InGaAs-based cameras cores in the SWIR spectral range.
v. Development of the dual band VIS-SWIR HSI imager consisting in a dual aperture snapshot camera covering the whole VIS-SWIR range.
vi. Development of OLED-based spectrometers to be integrated in the monitoring system of JOTIS for particle size and viscosity measurement.
vii. Preliminary thermography and colorimetry models. First thermography model for measuring high temperatures (500ºC-1000ºC) from a SWIR HSI camera (900-1700nm). Implementation of 2 colorimetry models for estimating chromatic coordinates (CIE XYZ) from a HSI pushbroom camera (370-900nm) and MULTIPLE HSI snapshot SSM4x4 mosaic camera (470-620nm): direct linear fitting and reflectance-based model.
viii. Development of edge computing devices for data capture from the HSI sensors developed in MULTIPLE and other factory field buses, HSI images processing for calibration and dimensionality reduction purposes and cloud connection, adopting RAMI4.0 protocols as standard way to interact with the devices. Cloud data messages and ingestion system has been already designed to transfer and store HSI images and ease the development of AI algorithms and manufacturing optimization tools.
ix. Significant activity delivered in terms of dissemination and exploitation tasks. During this period, the project website was created (https://multipleproject.eu/) and all the communication material was prepared. In terms of exploitation, the first Exploitation Plan of the project was submitted, including the list of Key Exploitable results identified, with its preliminary IP information and business intentions from each partner.
- MULTIPLE scalable monitoring and optimization solution for complex manufacturing chains
The Key Exploitable Result (KER) is a scalable platform for process monitoring and optimisation integrating the different OERs, ranging from the novel spectral sensors to gather relevant product and process parameters, edge devices with embedded AI processing capabilities enabling an easy integration with factory automation through industrial open standards e.g. OPC-UA), to the cloud services for production management and AI model development.
- Snapshot VIS/SWIR hyperspectral imaging cameras
Hyperspectral Imaging (HSI) cameras with snapshot acquisition of spectral images for measurement of process/product parameters: VIS/NIR HSI cameras, SWIR HSI cameras, Dual aperture VIS-SWIR HSI cameras.
- Laser-based IR analysers
Infrared multispectral in-situ and real-time Gas Analysers with high sensitivity and selectivity.
- Organic electronics based spectrometers
Organic electronics based micro-spectrometer combined with an optics module to allow measurements from a distance.
- Multimodal monitoring systems
These Multimodal Monitoring Systems are combinations of technologies and are intended to be modular and adaptable. The various Multimodal monitoring systems integrate the sensors and the VIS/NIR HSI cameras, SWIR HSI cameras and dual aperture VIS-SWIR HSI cameras; the models and the other consumables and materials needed for the components integration (cases, protection, optics …).
- Cloud and edge computing platform and service
The cloud and edge computing platform (with related services) targets the management, processing and analysis of HSI data for industrial applications. It’s built on top of off-the-shelf SoC boards, open source AI and analytics library and ABRAIA cloud services and API for the management and processing of image-based data. It provides a complete platform for the development of ML and deep learning models for industrial inspection and monitoring using HSI data, and the deployment and orchestration of edge devices with real-time monitoring capabilities.
- Potential impact
• Increased competitiveness of the European production industry and significant contribution to the digitization of European industry.
• MULTIPLE will strengthen EU photonics manufacturing base by reinforcing Europe’s industrial competitiveness and leadership in photonics and beyond.
• Boosting resources (energy and materials) efficiency, emissions and waste reduction and circular economy.
• MULTIPLE will contribute to high quality job creation in the photonics and manufacturing industry and beyond in high-tech sectors oriented to the development of ICT tools.
• Promotion of women participation in photonic and manufacturing industry.
• Enhancing life quality of European citizens. Regions and countries benefiting from highly productive industrial capacities developed at MULTIPLE will generate skilled jobs with decent salaries, thus contributing to European social cohesion.
The Key Exploitable Result (KER) is a scalable platform for process monitoring and optimisation integrating the different OERs, ranging from the novel spectral sensors to gather relevant product and process parameters, edge devices with embedded AI processing capabilities enabling an easy integration with factory automation through industrial open standards e.g. OPC-UA), to the cloud services for production management and AI model development.
- Snapshot VIS/SWIR hyperspectral imaging cameras
Hyperspectral Imaging (HSI) cameras with snapshot acquisition of spectral images for measurement of process/product parameters: VIS/NIR HSI cameras, SWIR HSI cameras, Dual aperture VIS-SWIR HSI cameras.
- Laser-based IR analysers
Infrared multispectral in-situ and real-time Gas Analysers with high sensitivity and selectivity.
- Organic electronics based spectrometers
Organic electronics based micro-spectrometer combined with an optics module to allow measurements from a distance.
- Multimodal monitoring systems
These Multimodal Monitoring Systems are combinations of technologies and are intended to be modular and adaptable. The various Multimodal monitoring systems integrate the sensors and the VIS/NIR HSI cameras, SWIR HSI cameras and dual aperture VIS-SWIR HSI cameras; the models and the other consumables and materials needed for the components integration (cases, protection, optics …).
- Cloud and edge computing platform and service
The cloud and edge computing platform (with related services) targets the management, processing and analysis of HSI data for industrial applications. It’s built on top of off-the-shelf SoC boards, open source AI and analytics library and ABRAIA cloud services and API for the management and processing of image-based data. It provides a complete platform for the development of ML and deep learning models for industrial inspection and monitoring using HSI data, and the deployment and orchestration of edge devices with real-time monitoring capabilities.
- Potential impact
• Increased competitiveness of the European production industry and significant contribution to the digitization of European industry.
• MULTIPLE will strengthen EU photonics manufacturing base by reinforcing Europe’s industrial competitiveness and leadership in photonics and beyond.
• Boosting resources (energy and materials) efficiency, emissions and waste reduction and circular economy.
• MULTIPLE will contribute to high quality job creation in the photonics and manufacturing industry and beyond in high-tech sectors oriented to the development of ICT tools.
• Promotion of women participation in photonic and manufacturing industry.
• Enhancing life quality of European citizens. Regions and countries benefiting from highly productive industrial capacities developed at MULTIPLE will generate skilled jobs with decent salaries, thus contributing to European social cohesion.