Team Awareness Enhanced with Artificial Intelligence and Augmented Reality

Project Information

TeamAware

Grant agreement ID: 101019808

DOI

10.3030/101019808

Project closed

EC signature date 26 April 2021

Start date 1 May 2021

End date 30 April 2024

Funded under

• Secure societies - Protecting freedom and security of Europe and its citizens

Total cost

€ 6 964 702,50

EU contribution

€ 6 964 702,50

Coordinated by

SOFTWARE IMAGINATION AND VISION SRL
Romania

Periodic Reporting for period 2 - TeamAware (Team Awareness Enhanced with Artificial Intelligence and Augmented Reality)

Reporting period: 2022-05-01 to 2024-04-30

Summary of the context and overall objectives of the project

1. What is the problem /issue being addressed

First responders are the groups of people, services and organisations with specialised skills and qualifications whose
duty is to arrive first to the emergency zone, search, save and rescue operations, and perform crisis management in natural or
human-made disasters. Although first responders provide secure and safe societies by protecting the communities,
responding to the disasters and rescuing lives, they often use inefficient, weak and obsolete technologies in the
operations.
With respect to the current situation, the operational capabilities of the first responders can be dramatically boosted by
the advances in technology and engineering fields such as smart sensor systems, wearables, data processing, data fusion,
data analytics, communication infrastructure, and artificial intelligence tools. The main objective of TeamAware Project is
to develop an integrated and cost-efficient situational awareness system for first responders from different sectors with
heterogeneous and hardly interoperable sensor units including drone mounted, wearable, and external sensor systems,
existing first responder services, and operation centres. The purpose is to enhance crisis management, flexibility and
reaction capability of first responders of different sectors through real-time, fused, refined, filtered, and manageable
information by using highly-standardized augmented reality and mobile human machine interfaces.



The main objectives are:
O1: Provide strategic & operational concept to the literature, improve the policy framework
O2: Provide user requirements, risk analysis and risk assessment study to the literature


O3: Obtain a visual modeling for guidance and mapping, recognize surrounding suspicious objects, and provide a first-person view by means of head and drone-mounted EO, IR, and LIDAR sensors.
O4: Identify and determine the risks of the infrastructures within the first responder operation area.
O5: Analyse and recognize the hazardous chemical agents surrounding the first responder.
O6: Identify gunshots, explosions, human whistling and speech within the operation area.
O7: Monitor the health status, activities, motion anomalies and location of the first, responders.
O8: Collect data by involving citizens into incidents as a sensory system via social media and using city IoT sensors.

O9: Provide TeamAware Domain Ontology compatible with existing data exchange models of surveillance and sensor systems
O10: Achieve an operation centre and sensor system interoperable communication
O11: Obtain a secure, standardised, and cloudbased communication network

O12: Present detailed Common Situational Awareness Picture to an operator in the operation center and first responders in timely and functional ways
O13: Present refined and filtered Common Situational Awareness Picture enhanced with Augmented Reality to first responders in the field in a timely and manageable manner

O14: Demonstrate the first phase TeamAware system for mid-term review to assess the system functional requirements
O15: Demonstrate integrated TeamAware system solution (final demonstration)
O16: Provide fully integrated TeamAware System enhanced with augmented reality

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During this first year of the TeamAware project (M1-M12), the consortium has collaborated to manage the requirement engineering specification and to design, and follow up, a common design methodology, along the project.
Use cases initial designs, each one with each own context and goals, have been completed and important elements for the future TeamAware work (such as data flows needs, functional and non-functional requirements, technical components, processes, etc.) have been identified. All these elements have been taken into account to design the system architecture and its main technical components. The major relevant work described till now is collected in the following list of deliverables to which TeamAware project was committed in the DoA.

At this stage of the project, we just started the discussion related to the exploitation. I RP2 more information will be presented

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The major ambition of TeamAware is to provide a real-time “Common Situational Awareness Picture” enhanced ith AR HMI for first responders to ensure effective and rapid decision making, where the picture will be generated by utilising different types of interoperable sensor systems. This ambition requires improvements in different areas:
• First responder situational awareness systems
State-of-the-art: The technologies and solutions used by the first responders consist of one or multiple operation
centres and integrating various SoA sensor technologies such as electro optical surveillance systems, supporting
equipment, and other types of stationary and mobile sensor systems via cloud-based communication network. These
systems are characterised by a flexible and adaptive architecture that is able to easily grow by integrating all types of
sensors from any manufacturer and new IT modules. On a global level, the majority of first responder situational
awareness systems are based on four main trends: (1) collecting and sharing detailed surrounding event data, (2)
using new techniques to assess risks and threats, (3) employing new technology in first responder operations, and (4)
building partnerships to achieve crisis management goals.
Creation of a database containing all information during an emergency case, new methods to gather all the data, catalogue and organise
them in the database and to implement algorithms to structure these data. Besides, TeamAware architecture will
include a collaborative assistant that supports the operator in the decision-making procedure (i) Multilingual userfriendly
interface, (ii) AR/ mobile tools to help the operator to manage potential threats through clear displays to
enable quick response (iii) Fast deployable subsystems and human machine interfaces.
The collaborative situational assistant will provide intelligence and relevance to the data that will be used by the
operators. Combining and fusing the data of several sensor subsystems will deliver additional information and
metadata to create a common situational awareness picture, creating more and more reliable information for the
operator. Furthermore, the envisaged AR technology developed in scope of the TeamAware will use different devices
to present a single fused picture of all the relevant information needed by the first responders. Its value as an aid to
situational awareness will be enhanced in low visibility conditions, by providing data for points of interest, otherwise
not visible. The system will highlight fixed and dynamic threats such as hazardous gases and chemicals or fire,
smoke, explosions etc. in the vicinity. Furthermore, the location, body posture, and health will be monitored by the
first responder, team members, and the operators in the incident management centre. Preferred displayed information
will be flexible and can be filtered depending on the current task and operation.

Architecture_TeamAware