Periodic Reporting for period 2 - MISEL (MULTISPECTRAL INTELLIGENT VISION SYSTEM WITH EMBEDDED LOW-POWER NEURAL COMPUTING)
Periodo di rendicontazione: 2022-07-01 al 2023-12-31
Animals, even small insects have superior capabilities of perceiving/understanding the environment and effortlessly perform tasks in unknown and changing terrain. This has led to a widespread interest in trying to mimic the way biology computes – neuromorphic computing, which could greatly advance autonomous car, robots, drones, and surveillance systems.
The MISEL project aims at low-power bio-inspired standalone vision system with multi-band sensing and in-sensor spatio-temporal neuromorphic processing based on complex events for advanced situation awareness, featuring smaller size, weight and power consumption than what is possible with existing components. The heterogeneous integration of a neuromorphic computing scheme with high-density memory arrays and adaptive photodetector technology enable fast and energy-efficient operation. The ambition is a compact system exploiting a wide range of biological principles: human eye like adaptivity with cellular processor, as well as data fusion, learning and reasoning performed by cerebellar processor (reflexes) and cortical processor (conscious decisions). By demonstrating and evaluating the system operation in end-to-end use scenarios, the project targets outperforming conventional state-of-the-art (SoA) systems.
The main objective is to demonstrate the advantages of the MISEL holistic sensing and computing approach over conventional approaches by 1) elaborating, designing, and implementing the MISEL sensing and computing system with multi-band visual sensory front end, 2) identifying application scenarios that benefit from event-based approach, where the proposed system could become a natural tool, and 3) benchmarking.
To achieve this main objective, the project will develop all crucial components of the system to demonstrate:
1. adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels are based on quantum dot/metal-insulator-graphene (QD/MIG) diodes monolithically fabricated on top of a silicon computing layer to work on multiple wavelengths.
2. in-sensor computing for data reduction and adaptation. Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
3. dense FeRAM monolithically integrated on top of CMOS computing layer, used for synaptic communication and plasticity.
4. effectiveness, which stems from organizing computing hierarchically similar to biology:
- near sensor processing (cellular sensor-processor) – sensors and processors form cells; neighboring cells interact like in the retina;
- fast spatio-temporal processing (cerebellar processor); spatio-temporal networks and hyperdimensional computing (HDC) for on-line learning and inference;
- high-level processing with prediction capabilities (cortical processor) + feedback to cellular/cerebellar processor.
5. competitiveness of neuromorphic computing. Map real-life use scenarios (e.g. related to sensory-motor systems) to the MISEL system and benchmark performance. Hardware and algorithms are co-designed to find the best possible trade-off between complexity and performance metrics.
The MISEL project aims at low-power bio-inspired standalone vision system with multi-band sensing and in-sensor spatio-temporal neuromorphic processing based on complex events for advanced situation awareness, featuring smaller size, weight and power consumption than what is possible with existing components. The heterogeneous integration of a neuromorphic computing scheme with high-density memory arrays and adaptive photodetector technology enable fast and energy-efficient operation. The ambition is a compact system exploiting a wide range of biological principles: human eye like adaptivity with cellular processor, as well as data fusion, learning and reasoning performed by cerebellar processor (reflexes) and cortical processor (conscious decisions). By demonstrating and evaluating the system operation in end-to-end use scenarios, the project targets outperforming conventional state-of-the-art (SoA) systems.
The main objective is to demonstrate the advantages of the MISEL holistic sensing and computing approach over conventional approaches by 1) elaborating, designing, and implementing the MISEL sensing and computing system with multi-band visual sensory front end, 2) identifying application scenarios that benefit from event-based approach, where the proposed system could become a natural tool, and 3) benchmarking.
To achieve this main objective, the project will develop all crucial components of the system to demonstrate:
1. adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels are based on quantum dot/metal-insulator-graphene (QD/MIG) diodes monolithically fabricated on top of a silicon computing layer to work on multiple wavelengths.
2. in-sensor computing for data reduction and adaptation. Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
3. dense FeRAM monolithically integrated on top of CMOS computing layer, used for synaptic communication and plasticity.
4. effectiveness, which stems from organizing computing hierarchically similar to biology:
- near sensor processing (cellular sensor-processor) – sensors and processors form cells; neighboring cells interact like in the retina;
- fast spatio-temporal processing (cerebellar processor); spatio-temporal networks and hyperdimensional computing (HDC) for on-line learning and inference;
- high-level processing with prediction capabilities (cortical processor) + feedback to cellular/cerebellar processor.
5. competitiveness of neuromorphic computing. Map real-life use scenarios (e.g. related to sensory-motor systems) to the MISEL system and benchmark performance. Hardware and algorithms are co-designed to find the best possible trade-off between complexity and performance metrics.
The MISEL research is conducted in 4 technical work packages (WP).
WP1 deals with fabrication of multi-band vision sensor and ferroelectric memory arrays as well as their integration on top of CMOS circuits. To achieve a reliable QD/MIG photodetector, sets of PbS nanoparticles were synthesized and their application on the MIG diodes was improved by structuring the nanoparticle layer. InAs QDs are researched as lead-free alternative. First experiments on the integration of QD/MIG photodetectors have been carried out. A variety of ferroelectric devices based on polymer or metal oxide (FeCap, FTJ, FeFET) were researched, fabricated, and measured. Then the work focused on metal oxide (HZO) based FeRAM due to better scaling properties. Monolithic integration of FeFETs with the CMOS chips was successfully demonstrated achieving very promising performance. This was enabled by development of wafer-scale processes for vias and first metal optimized for lattice orientation. The programming achieved up to 4 orders of magnitude current modulation. The memory also has high switching endurance (>10M). A Verilog-A model fitted the measured data, so that in can be used for CMOS-FeRAM co-design. A measurement setup to characterize the performance of a full array of post-CMOS FeFETs is being prepared.
In WP2 the aim is to implement neuromorphic circuits that gain performance from massive parallelism and computing topology. Several integrated circuits were taped-out (jointly with WP3) including QD/MIG sensor readout circuit, associative memory, convolution accelerator, and HDC arrays. Cellular and cerebellar processors are put on the same chip to take advantage of high speed on-chip data buses. The sensor front-end provides temporal difference events and fixed-pattern compensated intensity images. This data is used to generate complex events that compactly describe spatial and temporal features. The complex events are used for interest point detection, optical flow computation, and motion pattern extraction using an associative processor. On-chip cellular neural net provides means for segmentation, clustering, and region of interest (ROI). The processing hardware accommodates data path for both CMOS and QD pixels. A chip-to-chip link between cellular/cerebellar and cortical processor collects and transfers ROI data. On-chip controllers (custom and RISC-V) support large-scale real-time system operation and communication.
WP3 tackles higher level information processing and system integration. The designed cortical architecture, comprising a DNN for feature extraction combined with a HDC model for data and sequence classification in ROI, shows a very low power consumption. It was simulation-tested for cars, cyclists and pedestrians on the VIRAT and KAIST datasets. Datasets were turned into events representation by means of a SoA video-to-events generator. Gesture recognition is tested as new use-case. A hardware-oriented design methodology has been adopted, leading to a 5-layer neural network with a total of less than 4k parameters. A digital twin of the system supports hardware-realistic algorithm evaluation and co-design in WP2, WP3 and WP4.
WP4 is dedicated to evaluation of the sensor layer, the computational layer, and the entire system in selected applications. While the sensory and computational components are being developed, the use-case was defined, taking into account predicted hardware capabilities and legal issues. Evaluation criteria were defined, an evaluation corpus for the sensor was developed, and a comparative evaluation was done between algorithms applied on the original corpus data and the data from HW-realistic sensor emulator.
WP1 deals with fabrication of multi-band vision sensor and ferroelectric memory arrays as well as their integration on top of CMOS circuits. To achieve a reliable QD/MIG photodetector, sets of PbS nanoparticles were synthesized and their application on the MIG diodes was improved by structuring the nanoparticle layer. InAs QDs are researched as lead-free alternative. First experiments on the integration of QD/MIG photodetectors have been carried out. A variety of ferroelectric devices based on polymer or metal oxide (FeCap, FTJ, FeFET) were researched, fabricated, and measured. Then the work focused on metal oxide (HZO) based FeRAM due to better scaling properties. Monolithic integration of FeFETs with the CMOS chips was successfully demonstrated achieving very promising performance. This was enabled by development of wafer-scale processes for vias and first metal optimized for lattice orientation. The programming achieved up to 4 orders of magnitude current modulation. The memory also has high switching endurance (>10M). A Verilog-A model fitted the measured data, so that in can be used for CMOS-FeRAM co-design. A measurement setup to characterize the performance of a full array of post-CMOS FeFETs is being prepared.
In WP2 the aim is to implement neuromorphic circuits that gain performance from massive parallelism and computing topology. Several integrated circuits were taped-out (jointly with WP3) including QD/MIG sensor readout circuit, associative memory, convolution accelerator, and HDC arrays. Cellular and cerebellar processors are put on the same chip to take advantage of high speed on-chip data buses. The sensor front-end provides temporal difference events and fixed-pattern compensated intensity images. This data is used to generate complex events that compactly describe spatial and temporal features. The complex events are used for interest point detection, optical flow computation, and motion pattern extraction using an associative processor. On-chip cellular neural net provides means for segmentation, clustering, and region of interest (ROI). The processing hardware accommodates data path for both CMOS and QD pixels. A chip-to-chip link between cellular/cerebellar and cortical processor collects and transfers ROI data. On-chip controllers (custom and RISC-V) support large-scale real-time system operation and communication.
WP3 tackles higher level information processing and system integration. The designed cortical architecture, comprising a DNN for feature extraction combined with a HDC model for data and sequence classification in ROI, shows a very low power consumption. It was simulation-tested for cars, cyclists and pedestrians on the VIRAT and KAIST datasets. Datasets were turned into events representation by means of a SoA video-to-events generator. Gesture recognition is tested as new use-case. A hardware-oriented design methodology has been adopted, leading to a 5-layer neural network with a total of less than 4k parameters. A digital twin of the system supports hardware-realistic algorithm evaluation and co-design in WP2, WP3 and WP4.
WP4 is dedicated to evaluation of the sensor layer, the computational layer, and the entire system in selected applications. While the sensory and computational components are being developed, the use-case was defined, taking into account predicted hardware capabilities and legal issues. Evaluation criteria were defined, an evaluation corpus for the sensor was developed, and a comparative evaluation was done between algorithms applied on the original corpus data and the data from HW-realistic sensor emulator.
The aim of the MISEL is to address complex scene understanding with a system characterized by low Size, Weight and Power (SWaP). Combining CMOS neuromorphic computing methods and circuits with post-processed QD sensors and NVMs extends capabilities of CMOS ("more than Moore"). We pursue to demonstrate and evaluate performance advances of neuromorphic computing in full end-to-end applications. Such a vision system would be useful especially on-board of small autonomous vehicles such as drones or robots working in complex harsh environments. If applied to a safety surveillance or a rescue mission, it could contribute to saving human lives.