Periodic Reporting for period 2 - AUGGMED (Automated Serious Game Scenario Generator for Mixed Reality Training)
Reporting period: 2016-12-01 to 2018-05-31
The aim of AUGGMED is to develop a serious gaming platform to train law enforcement agencies, first responders and operators of critical infrastructure in their response to terrorist and organised crime attacks.
The AUGGMED platform enables both single ‘player’ and team-based training. It fosters the acquisition of emotional management, analytical thinking, problem solving and decision making skills. The game scenarios include reality-based simulations of operational environments, crowd behaviour and threats (including fire and explosions); it is delivered through a number of different methods including a conventional desktop computer game, immersive VR and Augmented Reality (AR) with multimodal interfaces.
The final AUGGMED platform delivers highly realistic training scenarios and advanced interactivity while encouraging infrastructure security staff and first responders to engage and actively participate in the training process.
AUGGMED has developed and tested three versions its prototype system:
- A conventional desktop game. This was tested in the first Pilot demonstration by firearms police officers in a number of weapons attacks in a (fictitious) UK airport in 2016.
- An immersive virtual reality version. This was tested in the second Pilot (2017) using in a 3D model of Muntaner train station (Barcelona) and featuring suspicious object and post-explosion triage scenarios, aimed at railway staff and paramedics, respectively;
- An augmented reality version. This version was set in a cruise ship terminal in the Port of Piraeus, and featuring a range of post-explosion and weapons attack scenarios. The Hellenic coast guard participated as end users in the third Pilot (March 2018).
In addition, AUGGMED has developed a haptic/thermal feedback vest which is fully integrated into the VR system and is intended to intensify the sense of realism and immersion for players.
The AUGGMED system was demonstrated at a large number of conferences and workshops, including the Security and Counter Terror Expo (London, 3-4 May 2017). In addition, the project was covered on the BBC's science programme 'Click' in 2017 (https://www.youtube.com/watch?v=DlDZWFXtYmU). A number of scientific papers have been written or are in preparation, including detailed technical work on the haptic feedback vest as well as the use of virtual reality environments in first responder training.
Exploitation efforts are ongoing and will continue beyond the end of the project.
Moreover, the AUGGMED serious game allows players to directly interact with crowd agents via hand gestures or spoken commands, thus allowing them to take control of a situation in a way that closely resembles a real-life scenario. In particular, players can interact with other players which enhances team work and/or collaboration with members of other organisations. The immersion of players into this type of realistically simulated environment will allow them to train their decision making skills and to experience (to a certain degree) emotional reactions that are inextricably linked to crisis scenarios.
In addition, AUGGMED makes use of a range of external devices (like VR headsets and thermal/haptic feedback vests) to enhance the experience of immersion for the players.
The AUGGMED serious game was developed in three stages: the first mode provides basic VR (in the form of a conventional desktop computer game), whereas the second and third mode add head-mounted devices and haptic vests to provide a fully immersive VR and/or AR environment.
Each of the three prototypes was tested in a Pilot demonstrator which included a wide range of potential end users: police firearms officers, railway staff, paramedics, coast guard and port police. The feedback received from the end users was very positive; participants in the pilots appreciated the realism of the simulation and considered the AUGGMED system to be a useful potential addition to their training regime.
Moreover, the AUGGMED consortium has undertaken a study comparing the effects of consolidation training in VR or in live role-plays. This study included 80 trainee police officers who acted out a number of suspicious object scenarios (either in VR or in a live role-play or a combination of the two). The analysis of the results is ongoing and will continue past the end of the project; the aim is to quantify the effects of consolidation training in VR and to investigate to which degree this approach can be a viable alternative to live role-play in first responder training.