Periodic Reporting for period 3 - ULPEC (Ultra-Low Power Event-Based Camera) Reporting period: 2019-07-01 to 2021-06-30 Summary of the context and overall objectives of the project The long term goal of ULPEC is to develop advanced vision applications with ultra-low power requirements and ultra-low latency.The output of the ULPEC project is a demonstrator consisting of an event-based visual sensor and a neural network on silicon and ferroelectric memristive synapses. Although ULPEC worked at low technology maturity levels (TRL4), it is a highly application-oriented project by studying prospective use cases and delivering an application roadmap. Further, the considering interoperability for an integration in “systems of systems” has been studied, together with the definition of upper power consumption limits depending on future applications.The project consortium included an industrial end-user, who in particularly investigated the use case of autonomous and computer assisted driving. Autonomous and computer assisted driving are indeed a major disruption in the transport and car manufacturing sector. Vision and recognition of traffic event must be computed with very low latency (to improve security) and low power (to accommodate the power limited environment in a car, such as power budget and heat dissipation). The work resulted in two automotive event training databases for algorithm development. Currently, those are the largest databases in this field that are openly available. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far The work has been roughly split into three research groups: memristive devices, CMOS design and system architecture, and industrial use of the ULPEC’s technology.All partners have strongly been involved in the project, including large industry (IBM & Bosch) and innovative SMEs in the field (Prophesee & TSST). The key technology results of ULPEC include:-A System-On-Chip (SoC) with a visual sensor array has been designed, fabricated, validated and tested. The visual sensor has been embedded in a HERMES chip with the final SoC containing the vision sensor, a Digital Control Block, 784 pre-synaptic and 100 post synaptic neurons and a ball pads array for memristor crossbar flip-chip. The flip chip assembly is much smaller than a traditional carrier-based system as the chip sits directly on the circuit board, and is much smaller than the carrier both in area and height. This allows to reach the ULPEC target in small chip size.-A FEMR device technology suited to fabricate a spiking neural network. With this work ULPEC demonstrated a manufacturable integration route towards the target SoC comprising the analogue vision sensor and the FEMR array.Two training databases were developed and made openly accessible for algorithm development in traffic event detection. To the best of our knowledge, the databases are the largest and most comprehensive of their kind.Besides the technology achievements, a detailed analysis of the advantages and limitations of ULPEC technologies was performed with a resulting list of most promising applications and an Exploitation Road Map. Both intend to provide the wider expert community with suggestions and strategies how academia, industry, and policy makers in Europe should proceed to achieve the greatest possible success. A special focus was put on future applications of high societal and socio-economic impact. The exploitation road map was complemented by dissemination, communication and stakeholder engagement activities and formats to maximise ULPEC impacts. Formats included two Policy Briefs, an open online course for training and skills development, and a number of ULPEC fact sheets in easy-to-understand language for the wider public. Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) ULPEC intended to deliver a SoC that is compact and that consumes less than a few mW of power for its demonstration task. It will be by nature a scalable technology that could benefit from more advanced technology, so that a roadmap can established for future SoC generations. ULPEC specifically targeted the challenge of user acceptability by involving a large company already at low TRLs as a first steppingstone for the upscaling and commercial exploitation of technologies over the coming years. The commercial activities of partner Prophesee, partially based on results and breakthroughs achieved in ULPEC, show that ULPEC indeed had and will have proven impact on a new generation of miniaturised smart systems by building the foundation for a system that will meet the required targets in size, power consumption and reliability once the demonstrator could be fully finalised and targets validated.The second expected impact concerns socio-economic aspects.ULPEC industrial partners already at the project start have been global leaders in their fields and within ULPEC committed themselves to protecting IP (3 potential patents were identified), scaling up the technology, mature it further and commercially exploit it (or components thereof).By identifying the most promising application cases for the event-based vision systems targeted by ULPEC, and via a detailed Exploitation Road Map, ULPEC provides market guidance for the wider community. The third addressed impact concerns the provision of innovative solutions for addressing societal needs and expectations in particular for the health and well-being, safety and security and environment.Additional societal impacts can be achieved based on the ultra-low power consumption of ULPEC technologies, which makes them potentially important players in the context of the EU Green Deal, especially as AI applications in the automotive sector or for smart farming. logo-transparent-final.png
