Multimodal spectral sensors and orchestrated deep models for integrated process optimisation

Project Information

MULTIPLE

Grant agreement ID: 871783

DOI

10.3030/871783

Project closed

EC signature date 5 November 2019

Start date 1 December 2019

End date 30 April 2023

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 7 368 755,91

EU contribution

€ 5 719 779,39

Coordinated by

ASOCIACION DE INVESTIGACION METALURGICA DEL NOROESTE
Spain

Periodic Reporting for period 2 - MULTIPLE (Multimodal spectral sensors and orchestrated deep models for integrated process optimisation)

Reporting period: 2021-06-01 to 2023-04-30

Summary of the context and overall objectives of the project

MULTIPLE develops cost-effective multimodal monitoring solutions with a breakthrough impact on production quality and efficiency. It brings cutting-edge organic electronics based sensors, snapshot hyperspectral filters and dual-aperture imaging to deliver cost-effective spectrometers and camera cores in a broad VIS/SWIR range, complemented with cost effective laser-based chemometric sensors in the MWIR. On top of these sensors, MULTIPLE develops novel multimodal monitoring systems that are IoT native, by combining them with cloud, big data, and deep learning for agile development and orchestration of complex AI-based models to optimize production improving EU manufacturing competitiveness. MULTIPLE overall objectives of the project are:
o To develop cost-effective HSI SWIR (0.9-1.7 μm) camera cores ready for volume production. To develop a compact and cost-effective dual-aperture HSI imager combining snapshot CMOS and InGaAs hyperspectral sensors based on mosaic filters covering the whole VIS/SWIR range (0.4-1.7 μm).
o To develop novel cost-effective spectrometer solutions for highly specific chemometric analysis in an extended VIS/MWIR range (0.4-3.5 μm wavelength), based on innovative organic electronics sensors and laser-based spectroscopy.
o To develop embedded deep models for regression, classification and RT control. It guarantees onsite and online measurements and classification of different types of categories with high flexibility, accuracy, and reliability.
o To develop an integral and scalable approach to process monitoring, quality assurance, and process optimization through the orchestration of photonic monitoring systems under an IoT native approach. The aim is to leverage SoA cloud and big data technologies to create new services and microservices enhancing decision support tools.
o To demonstrate MUTLIPLE in 3 manufacturing scenarios: steelworking, woodworking and food.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

i. Development of novel and cost-effective Snapshot VIS/SWIR Hyperspectral Imaging Cameras integrating spectral imagers with filters on chip (IMEC) into high-speed industrial camera platform (PHOTONFOCUS):
- Improved products: Snapshot VIS/NIR HSI cameras
- New products: Snapshot SWIR HSI cameras
- New products: Snapshot Dual aperture VIS/NIR-SWIR HSI cameras
PHOTONFOCUS will extend its product lines with new / improved HSI cameras and directly sale to end-users and system manufacturers for integration in monitoring or sorting solutions & machines.

ii. Development of Laser-based real-time IR Gas Analysers:
- New gas analyser optimized to measure combustion gases CO/O2/CH4/temperature.
- New extractive analyser for acetone detection.
AIROPTIC will extend its product line with improved and new products and directly sale to end-users and to manufacturers of systems for integration in multimodal monitoring solutions.

iii. Development of Organic electronics based Micro-spectrometers. New hardware prototype has been developed combining an optics module with the OE spectrometer to enable contact-less measurements. SENORICS will extend its product line with improved products OE-based micro-spectrometer to be integrated in SWIR spectroscopy systems and solutions by system integrators like IRIS.

iv. Development of Multimodal in-line process monitoring solutions. The multimodal monitoring systems are combinations of technologies (novel spectral sensors and edge devices with AI capabilities for online data processing) for different use cases. Monitoring system to measure temperature and steel composition (steelworking), quality inspection of lacquered panels for defect detection and colour check, estimation of impregnation layer thickness (woodworking), monitoring of tempering, monitoring of gloss in chocolate (food production). Different exploitation strategies are assessed by system integrators RUSSULA, SCM, IRIS

v. Development of a Cloud and edge computing platform for management, processing, and analysis of HSI data for industrial applications. Complete platform for the development of ML and deep learning models for industrial inspection and monitoring, and the deployment and orchestration of edge devices with real-time monitoring capabilities. ABRAIA will extend its product line with new services and new product.

vi. Development of an Industrial IoT scalable monitoring and optimisation solution. MULTIPLE scalable monitoring and optimization solution for complex manufacturing chains. Industrial IoT scalable platform for process monitoring and optimisation integrating the different exploitable results, ranging from the novel spectral sensors to gather relevant product and process parameters, edge devices with embedded AI processing capabilities enabling an easy integration with factory automation, to the cloud services for production management and AI model development. MRA will commercialise the whole system (MULTIoT) comprising all subsystems & technologies developed by project partners.

vii. The technology developed in the project has been integrated and validated in three application scenarios: steelworking (optimisation of a rolling mill), woodworking (quality control in a lacquering line) and food production (quality control in chocolate production).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

MULTIPLE has developed outstanding multimodal spectral sensors offering unique features compared with existing solutions: industrial cameras in compact form factors combining hyperspectral and high speed (350fps), OE-based micro-spectrometers for material sensing with lowest price in SWIR and contact-less measurement capabilities, multi-gas sensing spectrometers in a single device.

The cloud-edge HSI platform is designed to work seamlessly with factory automation systems, easily connecting HSI cameras and spectral sensors of different types. It enables users to focus on what is important, getting rid of time-consuming software development and integration tasks, and hindering challenging aspects of working with HSI such as spectral reconstruction or high dimensionality.

The innovation resulting from MULTIPLE open clear opportunities to deliver novel on-line and in-situ analysis for any automated production line. The affordability and easy-to-use technology will enable the widespread of HSI and spectral sensing for monitoring production processes, beyond sorting, and providing feedback for optimising processes and product quality.

Potential impact
• Increased competitiveness of the European production industry and significant contribution to the digitization of European industry.
• MULTIPLE will strengthen EU photonics manufacturing base by reinforcing Europe’s industrial competitiveness and leadership in photonics and beyond.
• Boosting resources (energy and materials) efficiency, emissions and waste reduction and circular economy.
• MULTIPLE will contribute to high quality job creation in the photonics and manufacturing industry and beyond in high-tech sectors oriented to the development of ICT tools.
• Promotion of women participation in photonic and manufacturing industry.
• Enhancing life quality of European citizens. Regions and countries benefiting from highly productive industrial capacities developed at MULTIPLE will generate skilled jobs with decent salaries, thus contributing to European social cohesion. 

Figure 5. HSI dual aperture camera installed in the rolling mill.

Figure 2. Snapshot HSI SWIR cameras.

Figure 4. Laser-based gas analysers installed in the reheating furnace wall.

Figure 1. Multiple concept

Figure 3. Organic-electronics spectrometer prototype

Figure 6. Quality inspection of lacquered wood panels.

Figure 7. Integration of SWIR spectometer in the conching machine (chocolate production)