Deep AR Law Enforcement Ecosystem

Project Information

DARLENE

Grant agreement ID: 883297

DOI

10.3030/883297

Project closed

EC signature date 21 April 2020

Start date 1 September 2020

End date 31 December 2023

Funded under

• Secure societies - Protecting freedom and security of Europe and its citizens

Total cost

€ 6 954 860,00

EU contribution

€ 6 954 860,00

Coordinated by

ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXIS
Greece

Periodic Reporting for period 2 - DARLENE (Deep AR Law Enforcement Ecosystem)

Reporting period: 2022-03-01 to 2023-12-31

Summary of the context and overall objectives of the project

Police officers in the field often operate under pressure making difficult decisions in a split second. Under such circumstances, critical information may be overlooked or simply delivered too late, with adversarial effects for public safety and officers’ well-being. DARLENE wants to tackle this problem through a technological ecosystem based on Augmented Reality and Artificial Intelligence towards enhancing officers’ situational awareness and allowing them to carry out their mission more effectively and safely.
The DARLENE ecosystem is based on portable AR gear that can overlay in the officer’s field of view critical information, enabling them to make informed decisions even under high pressure and take the initiative over their adversaries. Powerful computer vision algorithms are implemented and enhanced personalized visualization approaches are developed that leverage agents’ situational awareness. DARLENE can analyse several data streams from multiple IoT devices via a scalable architecture utilizing a tailor-made 5G network to offload computation to cloud nodes and deliver analysis results with minimal latency.
DARLENE also explores mechanisms to adapt to user preferences and contextual factors. Intelligent User Interfaces anticipating different levels of detail according to individual LEA’s characteristics have been developed while Machine Learning analyses officers’ stress status and dynamically adapts the level of rendered details. Aiming to improve cooperation between HQ and police officers during an operation, an Augmented Command & Control application is developed that renders a 3D reconstruction of an operation site, enriched with real-time results from computer vision algorithms, offering HQ officers a holistic and real-time view of an operation.
The project foresees two use cases; the first one focuses on the visual understanding of complex real-world scenes, where the officer will receive information about dangerous objects, injured people and suspects during a public safety incident. The second use case refers to the tactical neutralization of armed perpetrators during a crisis event. At the same time, a critical goal for DARLENE is its compliance with European legislation, societal values and fundamental rights. To this end, effort is put into establishing a robust regulatory framework for DARLENE, whereas research is oriented towards ways that can minimize the need for data storage and mitigate any bias in AI algorithms.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

As DARLENE will be used by police officers, a major objective was the elicitation of end-user requirements and the definition of use cases that realistically capture the conditions under which police operations are carried out. To this end, online surveys and co-creation workshops were carried out with the participation of police officers, following a systematic approach that led to the definition of the DARLENE architecture in a way that satisfies the needs of police, supports the foreseen use cases, and conforms to legal and ethical requirements.
A significant result is the wearable, portable prototype developed for police officers, which comprises an Augmented Reality helmet and a computation unit that implements advanced Artificial Intelligence (AI) to analyse in real-time user’s field of view. This prototype, entitled Wearable Edge Computing Node (WECN) is further evolved and updated based on hands-on interaction and evaluation with police officers in the planned training and piloting sessions. In a related line of research, AI methodologies have been developed for the other two computation layers of DARLENE, namely the cloud and the intermediate Patrol Car Edge Node (PCEN).
To interconnect the different computation layers of DARLENE, research has been focussed on the development of a private 5G infrastructure, while cyber-secure mechanisms have been developed for the authentication of different computation nodes. To allow for a personalized and adaptive AR experience, AR widgets have been designed that support different levels of detail that adapt to user preferences and stress status. Another interesting technological result has been the development of the DARLENE Command & Control application that visualises a 3D model of an operation site enriched with real-time information from AI algorithms. Further to this end, 3D reconstructions have been extracted from the project’s pilot sites.
Major effort has been put into defining a concrete ethical and legal framework for the application of DARLENE and the realisation of its piloting activities. To this end, extensive research has been performed on the ethical implications of the project with specific contributions in the way the proposed technology can be used to mitigate AI bias and limit the need for data storage via real-time analysis. Complementary activities to this end, have been the operation of an external Ethics Advisory Board and clearance from independent Ethics Committees prior to any research with human participation. In the same period, several horizontal activities took place to raise awareness concerning project results with extensive coverage by the press and media.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

DARLENE has progressed beyond state of the art for real-time computer vision technology for wearable and embedded computational units with limited computational resources. In this context, lightweight algorithms have been developed and integrated into portable wearable devices with Augmented Reality interfaces. Real-time performance on wearable nodes has a significant societal impact as it minimizes the need for data storage and transmission, allowing DARLENE to control data flow in space and time thus mitigating privacy and data protection risks. Significant results have been also achieved on personalized and context-aware AR that can adapt to user preferences and stress levels. As the uptake of AR technology gradually scales up, these characteristics will be critical for the proliferation of AR.
Another major result of the reporting period is the prototype for the Wearable Edge Computing Node (WECN) which consists of a portable computation unit with embedded AI functionalities and an Augmented Reality helmet where AI results are presented to the user’s field of view. This prototype will be a focal point for the training activities of the next period where police officers will be trained on the use of DARLENE technology and provide their feedback, guiding further updates and development. Furthermore, preparatory work has been carried out to ensure the smooth progression of the piloting activities in the next period of the project, by surveying the piloting sites and ensuring the compliance of the foreseen activities with ethics recommendations and regulations. To this end, a pipeline has been developed for the digitalization of large facilities, whereas a Command & Control application has been released where a reconstructed 3D model of an operation site can be rendered while real-time information from other software modules can be visualized.
Another line of research, where DARLENE made a meaningful contribution with a wider societal impact, is the use of data augmentation techniques to mitigate bias and improve the performance of AI algorithms. As AI constantly finds new applications, techniques that limit the need for huge data collection while at the same time mitigating any imbalanced datasets, can be quite useful and find use in several AI applications.

darlene-overview.png