Multimodal spectral sensors and orchestrated deep models for integrated process optimisation

Información del proyecto

MULTIPLE

Identificador del acuerdo de subvención: 871783

DOI

10.3030/871783

Proyecto cerrado

Fecha de la firma de la CE 5 Noviembre 2019

Fecha de inicio 1 Diciembre 2019

Fecha de finalización 30 Abril 2023

Financiado con arreglo a

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Coste total

€ 7 368 755,91

Aportación de la UE

€ 5 719 779,39

Coordinado por

ASOCIACION DE INVESTIGACION METALURGICA DEL NOROESTE
Spain

Periodic Reporting for period 2 - MULTIPLE (Multimodal spectral sensors and orchestrated deep models for integrated process optimisation)

Período documentado: 2021-06-01 hasta 2023-04-30

Resumen del contexto y de los objetivos generales del proyecto

MULTIPLE develops cost-effective multimodal monitoring solutions with a breakthrough impact on production quality and efficiency. It brings cutting-edge organic electronics based sensors, snapshot hyperspectral filters and dual-aperture imaging to deliver cost-effective spectrometers and camera cores in a broad VIS/SWIR range, complemented with cost effective laser-based chemometric sensors in the MWIR. On top of these sensors, MULTIPLE develops novel multimodal monitoring systems that are IoT native, by combining them with cloud, big data, and deep learning for agile development and orchestration of complex AI-based models to optimize production improving EU manufacturing competitiveness. MULTIPLE overall objectives of the project are:
o To develop cost-effective HSI SWIR (0.9-1.7 μm) camera cores ready for volume production. To develop a compact and cost-effective dual-aperture HSI imager combining snapshot CMOS and InGaAs hyperspectral sensors based on mosaic filters covering the whole VIS/SWIR range (0.4-1.7 μm).
o To develop novel cost-effective spectrometer solutions for highly specific chemometric analysis in an extended VIS/MWIR range (0.4-3.5 μm wavelength), based on innovative organic electronics sensors and laser-based spectroscopy.
o To develop embedded deep models for regression, classification and RT control. It guarantees onsite and online measurements and classification of different types of categories with high flexibility, accuracy, and reliability.
o To develop an integral and scalable approach to process monitoring, quality assurance, and process optimization through the orchestration of photonic monitoring systems under an IoT native approach. The aim is to leverage SoA cloud and big data technologies to create new services and microservices enhancing decision support tools.
o To demonstrate MUTLIPLE in 3 manufacturing scenarios: steelworking, woodworking and food.

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

i. Development of novel and cost-effective Snapshot VIS/SWIR Hyperspectral Imaging Cameras integrating spectral imagers with filters on chip (IMEC) into high-speed industrial camera platform (PHOTONFOCUS):
- Improved products: Snapshot VIS/NIR HSI cameras
- New products: Snapshot SWIR HSI cameras
- New products: Snapshot Dual aperture VIS/NIR-SWIR HSI cameras
PHOTONFOCUS will extend its product lines with new / improved HSI cameras and directly sale to end-users and system manufacturers for integration in monitoring or sorting solutions & machines.

ii. Development of Laser-based real-time IR Gas Analysers:
- New gas analyser optimized to measure combustion gases CO/O2/CH4/temperature.
- New extractive analyser for acetone detection.
AIROPTIC will extend its product line with improved and new products and directly sale to end-users and to manufacturers of systems for integration in multimodal monitoring solutions.

iii. Development of Organic electronics based Micro-spectrometers. New hardware prototype has been developed combining an optics module with the OE spectrometer to enable contact-less measurements. SENORICS will extend its product line with improved products OE-based micro-spectrometer to be integrated in SWIR spectroscopy systems and solutions by system integrators like IRIS.

iv. Development of Multimodal in-line process monitoring solutions. The multimodal monitoring systems are combinations of technologies (novel spectral sensors and edge devices with AI capabilities for online data processing) for different use cases. Monitoring system to measure temperature and steel composition (steelworking), quality inspection of lacquered panels for defect detection and colour check, estimation of impregnation layer thickness (woodworking), monitoring of tempering, monitoring of gloss in chocolate (food production). Different exploitation strategies are assessed by system integrators RUSSULA, SCM, IRIS

v. Development of a Cloud and edge computing platform for management, processing, and analysis of HSI data for industrial applications. Complete platform for the development of ML and deep learning models for industrial inspection and monitoring, and the deployment and orchestration of edge devices with real-time monitoring capabilities. ABRAIA will extend its product line with new services and new product.

vi. Development of an Industrial IoT scalable monitoring and optimisation solution. MULTIPLE scalable monitoring and optimization solution for complex manufacturing chains. Industrial IoT scalable platform for process monitoring and optimisation integrating the different exploitable results, ranging from the novel spectral sensors to gather relevant product and process parameters, edge devices with embedded AI processing capabilities enabling an easy integration with factory automation, to the cloud services for production management and AI model development. MRA will commercialise the whole system (MULTIoT) comprising all subsystems & technologies developed by project partners.

vii. The technology developed in the project has been integrated and validated in three application scenarios: steelworking (optimisation of a rolling mill), woodworking (quality control in a lacquering line) and food production (quality control in chocolate production).

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

MULTIPLE has developed outstanding multimodal spectral sensors offering unique features compared with existing solutions: industrial cameras in compact form factors combining hyperspectral and high speed (350fps), OE-based micro-spectrometers for material sensing with lowest price in SWIR and contact-less measurement capabilities, multi-gas sensing spectrometers in a single device.

The cloud-edge HSI platform is designed to work seamlessly with factory automation systems, easily connecting HSI cameras and spectral sensors of different types. It enables users to focus on what is important, getting rid of time-consuming software development and integration tasks, and hindering challenging aspects of working with HSI such as spectral reconstruction or high dimensionality.

The innovation resulting from MULTIPLE open clear opportunities to deliver novel on-line and in-situ analysis for any automated production line. The affordability and easy-to-use technology will enable the widespread of HSI and spectral sensing for monitoring production processes, beyond sorting, and providing feedback for optimising processes and product quality.

Potential impact
• Increased competitiveness of the European production industry and significant contribution to the digitization of European industry.
• MULTIPLE will strengthen EU photonics manufacturing base by reinforcing Europe’s industrial competitiveness and leadership in photonics and beyond.
• Boosting resources (energy and materials) efficiency, emissions and waste reduction and circular economy.
• MULTIPLE will contribute to high quality job creation in the photonics and manufacturing industry and beyond in high-tech sectors oriented to the development of ICT tools.
• Promotion of women participation in photonic and manufacturing industry.
• Enhancing life quality of European citizens. Regions and countries benefiting from highly productive industrial capacities developed at MULTIPLE will generate skilled jobs with decent salaries, thus contributing to European social cohesion. 

Figure 5. HSI dual aperture camera installed in the rolling mill.

Figure 2. Snapshot HSI SWIR cameras.

Figure 4. Laser-based gas analysers installed in the reheating furnace wall.

Figure 1. Multiple concept

Figure 3. Organic-electronics spectrometer prototype

Figure 6. Quality inspection of lacquered wood panels.

Figure 7. Integration of SWIR spectometer in the conching machine (chocolate production)