Periodic Reporting for period 1 - NimbleAI (ULTRA-ENERGY EFFICIENT AND SECURE NEUROMORPHIC SENSING AND PROCESSING AT THE ENDPOINT)
Periodo di rendicontazione: 2022-10-01 al 2024-03-31
Most processor architectures are very inefficient in comparison to brains, which are honed by natural selection. Think of a honeybee miniature brain that has about a cubic millimetre and runs on a drop of nectar! Yet bees outperform AI running on CPU/GPUs, showing sophisticated learning capabilities and high-precision spatial orientation skills to fly at high speeds while avoiding collisions with obstacles and evading familiar predators. NimbleAI leverages key principles of energy-efficient visual information acquisition and processing in eyes and brains to design the next-generation neuromorphic vision chips that build upon the latest advances in 3D-integrated circuit technology and AI breakthroughs. The project aims to deliver 100x energy-efficiency improvement and 50x latency reduction in visual perception tasks – w.r.t. using mainstream CPU/GPUs – to enable the development of autonomous drones the size of bumblebees and lightweight augmented reality wearables, among other innovations.
NimbleAI considers that processing begins in the sensor, which is positioned in the outer layer of the 3D-stacked architecture. Significant efficiency gains are anticipated by adopting novel dynamic vision sensing concepts, which will be further enhanced in subsequent processing stages. Unlike traditional frame-based sensors that capture full images at fixed intervals, Dynamic Vision Sensors (DVS) detect changes in visual scenes and encode them in the form of discrete visual events in the spatiotemporal domain. NimbleAI pioneers the development of digitally-foveated DVS (DF-DVS), replicating the highly efficient peripheral and central vision in vertebrate visual systems, and insect eye-inspired light-field DVS (LF-DVS), which enables ultra-efficient and (almost) instantaneous event-based 3D perception.
The NimbleAI architecture implements selective attention mechanisms powered by event-driven Spiking Neural Networks (SNNs) to detect Regions of Interest (ROIs) in the visual scene, and drive the foveation settings of DF-DVS accordingly. ROIs are sensed in high-resolution, whereas the rest of the scene is still sensed in low-resolution to enable detection of new ROIs. A DVS front-end composes data structures (including depth maps when using LF-DVS) compatible with mainstream AI models using ROI events, bridging neuromorphic components and AI engines in the NimbleAI architecture.
The AI engines in the NimbleAI architecture have various processing capabilities, enabling them to run different AI models matching the properties of each detected ROI. To achieve efficient end-to-end inference of ROIs, the AI engines implement processing specialization using eFPGA-based custom CPU instructions, in-memory processing, and event-driven dataflow architectures. 3D-stacked high-density non-volatile memory layers are used to store and run multiple AI models simultaneously, each on a different ROI. These memory layers are tightly coupled with the AI engines, enabling dynamic swapping of neural network parts, virtually extending the processing resources implemented in silicon. We have coined this concept as Virtual Neural Networks (VNNs).
NimbleAI considers that processing begins in the sensor, which is positioned in the outer layer of the 3D-stacked architecture. Significant efficiency gains are anticipated by adopting novel dynamic vision sensing concepts, which will be further enhanced in subsequent processing stages. Unlike traditional frame-based sensors that capture full images at fixed intervals, Dynamic Vision Sensors (DVS) detect changes in visual scenes and encode them in the form of discrete visual events in the spatiotemporal domain. NimbleAI pioneers the development of digitally-foveated DVS (DF-DVS), replicating the highly efficient peripheral and central vision in vertebrate visual systems, and insect eye-inspired light-field DVS (LF-DVS), which enables ultra-efficient and (almost) instantaneous event-based 3D perception.
The NimbleAI architecture implements selective attention mechanisms powered by event-driven Spiking Neural Networks (SNNs) to detect Regions of Interest (ROIs) in the visual scene, and drive the foveation settings of DF-DVS accordingly. ROIs are sensed in high-resolution, whereas the rest of the scene is still sensed in low-resolution to enable detection of new ROIs. A DVS front-end composes data structures (including depth maps when using LF-DVS) compatible with mainstream AI models using ROI events, bridging neuromorphic components and AI engines in the NimbleAI architecture.
The AI engines in the NimbleAI architecture have various processing capabilities, enabling them to run different AI models matching the properties of each detected ROI. To achieve efficient end-to-end inference of ROIs, the AI engines implement processing specialization using eFPGA-based custom CPU instructions, in-memory processing, and event-driven dataflow architectures. 3D-stacked high-density non-volatile memory layers are used to store and run multiple AI models simultaneously, each on a different ROI. These memory layers are tightly coupled with the AI engines, enabling dynamic swapping of neural network parts, virtually extending the processing resources implemented in silicon. We have coined this concept as Virtual Neural Networks (VNNs).
The world’s first LF-DVS has been assembled using a Prophesee-Sony IMX636 sensor and a Raytrix custom microlens array. It has also been demonstrated the feasibility of estimating depth using adapted light-field algorithms with time-surfaces created out of events delivered by the LF-DVS. Ultra energy-efficient event-driven algorithms for processing LF-DVS inputs are being designed and validated using real-world automotive ADAS datasets. These algorithms will next be implemented in silicon in time to meet the “Neuromorphic 3D sensing” milestone by 2026 foreseen in the “2024 Yole Neuromorphic Computing, Memory, and Sensing” report, where the NimbleAI technology is featured. The expectation is to achieve near-VGA resolution and tens of mm depth accuracy for up to a dozen metres operational range with sub-ms latency using tens of mW.
A DF-DVS chip design with 384x304 pixels has already been submitted to the foundry. The foveation circuitry imposes a minimal area overhead in the DF-DVS pixel compared with state-of-the-art DVS pixels (Prophesee-Sony IMX636). While the DF-DVS chip is under fabrication, a software toolchain to simulate visual event data downscaling and foveation has been created and used to train attentional SNNs that guide ROI detection based on DVS activity and saliency. Network tuning is under investigation to enable deployment on an SNN processor chip that is being implemented in the project.
The DVS front-end to drive foveation of the DF-DVS has been designed and is being prototyped on an FPGA. This front-end also includes an optical flow-informed lightweight event filter to remove noise and control the rate of events delivered to the attentional SNNs. Likewise, RISC-V-based AI engines that integrate in-memory processing blocks and eFPGA fabric are also being prototyped on an FPGA and provided with TinyML framework support. Likewise, a high-level description of a 3D-stacked memory layer and associated peripheral circuitry has been completed and the accompanying strategies for static and dynamic mapping of neural network parts onto event-driven dataflow AI engines have been validated in simulation. NimbleAI is advancing into creating novel tools and hardware components that expose sparsity in incoming DVS event-flows to improve energy use in event-driven AI engines and to reduce AI model sizes.
NimbleAI is taking a pseudo-3D integration approach that reuses the 2D layouts of the DF-DVS and SNN processor chips as the top and bottom layers of a 3D stack, respectively, and do physical implementation of the DVS front-end for the intermediate layer. A 3D EDA tool is also being developed to explore the 3D design space and guide optimal partition and floorplan its components across 3D layers.
To enable the use of the NimbleAI technology by use cases and to promote broader adoption, the project is building a functional prototype that integrates NimbleAI chips (i.e. DF-DVS and SNN processor), an FPGA to prototype the DVS front-end and the AI engines, as well as a Prophesee-Sony IMX636 DVS sensor.
A DF-DVS chip design with 384x304 pixels has already been submitted to the foundry. The foveation circuitry imposes a minimal area overhead in the DF-DVS pixel compared with state-of-the-art DVS pixels (Prophesee-Sony IMX636). While the DF-DVS chip is under fabrication, a software toolchain to simulate visual event data downscaling and foveation has been created and used to train attentional SNNs that guide ROI detection based on DVS activity and saliency. Network tuning is under investigation to enable deployment on an SNN processor chip that is being implemented in the project.
The DVS front-end to drive foveation of the DF-DVS has been designed and is being prototyped on an FPGA. This front-end also includes an optical flow-informed lightweight event filter to remove noise and control the rate of events delivered to the attentional SNNs. Likewise, RISC-V-based AI engines that integrate in-memory processing blocks and eFPGA fabric are also being prototyped on an FPGA and provided with TinyML framework support. Likewise, a high-level description of a 3D-stacked memory layer and associated peripheral circuitry has been completed and the accompanying strategies for static and dynamic mapping of neural network parts onto event-driven dataflow AI engines have been validated in simulation. NimbleAI is advancing into creating novel tools and hardware components that expose sparsity in incoming DVS event-flows to improve energy use in event-driven AI engines and to reduce AI model sizes.
NimbleAI is taking a pseudo-3D integration approach that reuses the 2D layouts of the DF-DVS and SNN processor chips as the top and bottom layers of a 3D stack, respectively, and do physical implementation of the DVS front-end for the intermediate layer. A 3D EDA tool is also being developed to explore the 3D design space and guide optimal partition and floorplan its components across 3D layers.
To enable the use of the NimbleAI technology by use cases and to promote broader adoption, the project is building a functional prototype that integrates NimbleAI chips (i.e. DF-DVS and SNN processor), an FPGA to prototype the DVS front-end and the AI engines, as well as a Prophesee-Sony IMX636 DVS sensor.
- The world’s first light-field DVS (LF-DVS) sensor for event-based ultra-energy efficient and low-latency 3D perception. LF-DVS is expected to help improve navigation of (lightweight) SNN-powered robots, especially drones that can move in the 3D space, by providing visual events enriched with depth with sub-ms latency. These events also enable to efficiently estimate more accurate 3D scene flows.
- The world’s first digitally-foveated DVS (DF-DVS) sensor that delivers simultaneously low- and high-resolution event-flows. This functionality of DF-DVS enhances energy efficiency when using the low-resolution event-flows to spot ROIs to be sensed in high-resolution. This is the intended use of DF-DVS in NimbleAI. Alternatively, when both low- and high-resolution event-flows are simultaneously activated for the whole sensor, DF-DVS facilitates the creation of multi-resolution maps of the surroundings as an initial step for estimating depth and optical flow.
- The world’s first digitally-foveated DVS (DF-DVS) sensor that delivers simultaneously low- and high-resolution event-flows. This functionality of DF-DVS enhances energy efficiency when using the low-resolution event-flows to spot ROIs to be sensed in high-resolution. This is the intended use of DF-DVS in NimbleAI. Alternatively, when both low- and high-resolution event-flows are simultaneously activated for the whole sensor, DF-DVS facilitates the creation of multi-resolution maps of the surroundings as an initial step for estimating depth and optical flow.