Most processor architectures are very inefficient in comparison to brains, which are honed by natural selection. Think of a honeybee miniature brain that has about a cubic millimetre and runs on a drop of nectar! Yet bees outperform AI running on CPU/GPUs, showing sophisticated learning capabilities and high-precision spatial orientation skills to fly at high speeds while avoiding collisions with obstacles and evading familiar predators. NimbleAI leverages key principles of energy-efficient visual information acquisition and processing in eyes and brains to design the next-generation neuromorphic vision chips that build upon the latest advances in 3D-integrated circuit technology and AI breakthroughs. The project aims to deliver 100x energy-efficiency improvement and 50x latency reduction in visual perception tasks – w.r.t. using mainstream CPU/GPUs – to enable the development of autonomous drones the size of bumblebees and lightweight augmented reality wearables, among other innovations.
NimbleAI considers that processing begins in the sensor, which is positioned in the outer layer of the 3D-stacked architecture. Significant efficiency gains are anticipated by adopting novel dynamic vision sensing concepts, which will be further enhanced in subsequent processing stages. Unlike traditional frame-based sensors that capture full images at fixed intervals, Dynamic Vision Sensors (DVS) detect changes in visual scenes and encode them in the form of discrete visual events in the spatiotemporal domain. NimbleAI pioneers the development of digitally-foveated DVS (DF-DVS), replicating the highly efficient peripheral and central vision in vertebrate visual systems, and insect eye-inspired light-field DVS (LF-DVS), which enables ultra-efficient and (almost) instantaneous event-based 3D perception.
The NimbleAI architecture implements selective attention mechanisms powered by event-driven Spiking Neural Networks (SNNs) to detect Regions of Interest (ROIs) in the visual scene, and drive the foveation settings of DF-DVS accordingly. ROIs are sensed in high-resolution, whereas the rest of the scene is still sensed in low-resolution to enable detection of new ROIs. A DVS front-end composes data structures (including depth maps when using LF-DVS) compatible with mainstream AI models using ROI events, bridging neuromorphic components and AI engines in the NimbleAI architecture.
The AI engines in the NimbleAI architecture have various processing capabilities, enabling them to run different AI models matching the properties of each detected ROI. To achieve efficient end-to-end inference of ROIs, the AI engines implement processing specialization using eFPGA-based custom CPU instructions, in-memory processing, and event-driven dataflow architectures. 3D-stacked high-density non-volatile memory layers are used to store and run multiple AI models simultaneously, each on a different ROI. These memory layers are tightly coupled with the AI engines, enabling dynamic swapping of neural network parts, virtually extending the processing resources implemented in silicon. We have coined this concept as Virtual Neural Networks (VNNs).
