Periodic Reporting for period 2 - CONNEXIONs (InterCONnected NEXt-Generation Immersive IoT Platform of Crime and Terrorism DetectiON, PredictiON, InvestigatiON, and PreventiON Services)
Berichtszeitraum: 2020-03-01 bis 2022-02-28
The continuous evolution of serious and organised crime and terrorism poses a challenge for Law Enforcement Agencies (LEAs) that need to provide effective countermeasure solutions for the detection, prediction, investigation, and ultimately prevention of such activities. In particular, the challenges for LEAs are to: (a) effectively correlate data from multiple heterogeneous sources during all stages (intelligence gathering, surveillance, operation, and investigation) of the process; (b) enable efficient access to the multimodal information pertinent to the different stakeholders involved; (c) efficiently simulate in 3D sites of interest in an accurate and visually convincing manner so as to facilitate operational and tactical planning and/or assist investigation; and (d) support the path-to-court of digital evidence.
CONNEXIONs aims to equip LEAs with an interconnected suite of advanced next-generation technologies for significantly improving their capabilities to gather intelligence, analyse evidence, and investigate crime and terrorism effectively and efficiently, with particular focus on improving the capabilities of officers across the board by encompassing the entire lifecycle of law enforcement operations, including: (a) pre-occurrence crime prediction and prevention; (b) during-occurrence LEA operations; and (c) post-occurrence investigation, and crime-scene simulation and 3D reconstruction.
To this end, CONNEXIONs builds upon two main concepts: (1) the multidimensional integration and correlation of heterogeneous multimodal data, including Web and social media multilingual and multimedia content, data acquired by Internet of Things (IoT) devices, police reports, and digital evidence, and (2) the delivery of pertinent information to various stakeholders in an interactive manner tailored to their needs, including through immersive (augmented reality - AR and virtual reality - VR) environments.
The main CONNEXIONs results include: (1) Multimodal information extraction, understanding, integration, correlation, retrieval, and delivery services; (2) Immersive environments for improving situational awareness, investigation, and training capabilities; (3) Operation Command Centre for real-time management of LEA operations; and (4) Integrated platform based on ethics and privacy by-design principles, implementing EU legal requirements, whilst being highly customisable to local legislation.
The developed CONNEXIONs tools and solutions have been validated by the end users in field demonstrations in three operational pilot use cases: (1) counter-terrorism security in large scale public events; (2) human trafficking investigations and mitigation, and (3) crime investigation and training through 3D scene reconstruction.
CONNEXIONs aims to equip LEAs with an interconnected suite of advanced next-generation technologies for significantly improving their capabilities to gather intelligence, analyse evidence, and investigate crime and terrorism effectively and efficiently, with particular focus on improving the capabilities of officers across the board by encompassing the entire lifecycle of law enforcement operations, including: (a) pre-occurrence crime prediction and prevention; (b) during-occurrence LEA operations; and (c) post-occurrence investigation, and crime-scene simulation and 3D reconstruction.
To this end, CONNEXIONs builds upon two main concepts: (1) the multidimensional integration and correlation of heterogeneous multimodal data, including Web and social media multilingual and multimedia content, data acquired by Internet of Things (IoT) devices, police reports, and digital evidence, and (2) the delivery of pertinent information to various stakeholders in an interactive manner tailored to their needs, including through immersive (augmented reality - AR and virtual reality - VR) environments.
The main CONNEXIONs results include: (1) Multimodal information extraction, understanding, integration, correlation, retrieval, and delivery services; (2) Immersive environments for improving situational awareness, investigation, and training capabilities; (3) Operation Command Centre for real-time management of LEA operations; and (4) Integrated platform based on ethics and privacy by-design principles, implementing EU legal requirements, whilst being highly customisable to local legislation.
The developed CONNEXIONs tools and solutions have been validated by the end users in field demonstrations in three operational pilot use cases: (1) counter-terrorism security in large scale public events; (2) human trafficking investigations and mitigation, and (3) crime investigation and training through 3D scene reconstruction.
CONNEXIONs achieved all the objectives set and reached all the milestones.
First, a user-centred approach was followed to perform a study on data and resources for supporting the use cases and for the LEAs in the consortium to collaboratively define the user requirements, based on which the CONNEXIONs technical specifications were set and the platform architecture was designed.
The CONNEXIONs architecture was first realised by an operational prototype, which integrated the skeleton versions of the technical modules, and then by three versions of the CONNEXIONs prototype platform which integrated iterative versions of the following components: (a) Discovery & acquisition of relevant Web and social media content; (b) IoT data normalisation, propagation, fusion, and analytics; (c) Multilingual textual content transcription, translation, and analysis; (d) Visual understanding of multimedia data; (e) Multimodal analytics, textual generation, and threat assessment; (f) Μultimodal retrieval and semantic reasoning; (g) 3D reconstructions and BIM and GIS models; (h) Immersive AR/VR applications with personalisation capabilities; (i) Investigative hypothesis simulation and analysis; and (j) Operation Command Centre for real-time management of operations.
The three versions of the CONNEXIONs platform were evaluated in three iterations of pilot demonstrations, respectively; training sessions preceded all pilot demonstrations and a comprehensive training manual was also produced.
Dissemination and collaboration activities were also performed, including the participation in (and organisation of) events, conferences and workshops, the publication of scientific papers, the generation of dissemination material, the setting up of a website and social media presence, the creation of synergies with related projects and initiatives, the establishment of a stakeholder network, and the organisation of three workshops, each focused on one of the use cases, and also the CONNEXIONs Final Conference. Moreover, a market analysis, exploitation plan, and business models were also delivered. Finally, an overall legal and ethical framework was set up.
First, a user-centred approach was followed to perform a study on data and resources for supporting the use cases and for the LEAs in the consortium to collaboratively define the user requirements, based on which the CONNEXIONs technical specifications were set and the platform architecture was designed.
The CONNEXIONs architecture was first realised by an operational prototype, which integrated the skeleton versions of the technical modules, and then by three versions of the CONNEXIONs prototype platform which integrated iterative versions of the following components: (a) Discovery & acquisition of relevant Web and social media content; (b) IoT data normalisation, propagation, fusion, and analytics; (c) Multilingual textual content transcription, translation, and analysis; (d) Visual understanding of multimedia data; (e) Multimodal analytics, textual generation, and threat assessment; (f) Μultimodal retrieval and semantic reasoning; (g) 3D reconstructions and BIM and GIS models; (h) Immersive AR/VR applications with personalisation capabilities; (i) Investigative hypothesis simulation and analysis; and (j) Operation Command Centre for real-time management of operations.
The three versions of the CONNEXIONs platform were evaluated in three iterations of pilot demonstrations, respectively; training sessions preceded all pilot demonstrations and a comprehensive training manual was also produced.
Dissemination and collaboration activities were also performed, including the participation in (and organisation of) events, conferences and workshops, the publication of scientific papers, the generation of dissemination material, the setting up of a website and social media presence, the creation of synergies with related projects and initiatives, the establishment of a stakeholder network, and the organisation of three workshops, each focused on one of the use cases, and also the CONNEXIONs Final Conference. Moreover, a market analysis, exploitation plan, and business models were also delivered. Finally, an overall legal and ethical framework was set up.
CONNEXIONs aims to equip LEAs with an interconnected suite of advanced next-generation technologies for significantly improving their capabilities to gather intelligence, analyse evidence, and investigate crime and terrorism effectively and efficiently.
CONNEXIONs thus aimed to achieve the following impacts: (a) Provision of better tools tailored to the needs of LEA officers across the board to support them on their daily work by encompassing the entire lifecycle of law enforcement operations; (b) Prevention of more terrorist endeavours both at the investigation stage also and at the operation stage during specific missions; (c) Better identification and understanding of criminal and terrorist activities based on enhanced multimedia and multilingual content analysis for the automatic detection, classification, and tracking of objects and activities of interest; (d) Improved investigation and training capabilities through advanced next-generation technologies; and (e) Rapid crime solving that has the potential to reduce societal distress, investigative costs, and impact on victims / relatives.
The CONNEXIONs final result is an interconnected suite of advanced next-generation technologies improving the state-of-the-art in several scientific and technological fields, thus facilitating the implementation of the promised impact; these include:
(1) Discovery and acquisition of relevant multimedia and multilingual online content, and the subsequent multilingual automatic speech recognition, machine translation, and analysis of the acquired textual material in the languages of interest.
(2) Acquisition, extraction, fusion, analysis, and visualisation of multimodal data obtained from diverse IoT devices and sensors.
(3) Visual understanding and extraction of useful information from multimedia data.
(4) Semantic integration, enrichment, and reasoning on the basis of the CONNEXIONs ontology used for the semantic representation of available data and extracted information.
(5) Multimodal information correlation for extracting valuable knowledge pertinent to the LEAs’ information needs out of the high volume of available data while uncovering the structure underlying the data, and for assessing potential threats.
(6) 3D reconstruction and immersive (AR and VR) technologies for improving situational awareness and investigation capabilities
(7) Operation command centre for real-time operation management and command capabilities for LEAs during law enforcement and public safety operation.
CONNEXIONs thus aimed to achieve the following impacts: (a) Provision of better tools tailored to the needs of LEA officers across the board to support them on their daily work by encompassing the entire lifecycle of law enforcement operations; (b) Prevention of more terrorist endeavours both at the investigation stage also and at the operation stage during specific missions; (c) Better identification and understanding of criminal and terrorist activities based on enhanced multimedia and multilingual content analysis for the automatic detection, classification, and tracking of objects and activities of interest; (d) Improved investigation and training capabilities through advanced next-generation technologies; and (e) Rapid crime solving that has the potential to reduce societal distress, investigative costs, and impact on victims / relatives.
The CONNEXIONs final result is an interconnected suite of advanced next-generation technologies improving the state-of-the-art in several scientific and technological fields, thus facilitating the implementation of the promised impact; these include:
(1) Discovery and acquisition of relevant multimedia and multilingual online content, and the subsequent multilingual automatic speech recognition, machine translation, and analysis of the acquired textual material in the languages of interest.
(2) Acquisition, extraction, fusion, analysis, and visualisation of multimodal data obtained from diverse IoT devices and sensors.
(3) Visual understanding and extraction of useful information from multimedia data.
(4) Semantic integration, enrichment, and reasoning on the basis of the CONNEXIONs ontology used for the semantic representation of available data and extracted information.
(5) Multimodal information correlation for extracting valuable knowledge pertinent to the LEAs’ information needs out of the high volume of available data while uncovering the structure underlying the data, and for assessing potential threats.
(6) 3D reconstruction and immersive (AR and VR) technologies for improving situational awareness and investigation capabilities
(7) Operation command centre for real-time operation management and command capabilities for LEAs during law enforcement and public safety operation.