CRitical Incident management training System using an Interactive Simulation environment

Project Information

CRISIS

Grant agreement ID: 242474

Project closed

Start date 1 May 2010

End date 31 October 2013

Funded under

• Specific Programme "Cooperation": Security

Total cost € 4 590 310,18

EU contribution € 3 495 612,00

Coordinated by

MIDDLESEX UNIVERSITY HIGHER EDUCATION CORPORATION
United Kingdom

Final Report Summary - CRISIS (CRitical Incident management training System using an Interactive Simulation environment)

Executive Summary:
CRISIS was conceived as a 3D immersive virtual world in which major incidents occur. Trainees are presented with a real-time evolving situation such as a train crash resulting in fires, explosions, increasing casualties and resources distributed around in a 3D virtual world. The trainee is required to view and assess the situation, formulate a plan, and organise a response by instructing his team while coordinating with other teams and higher command. Delays or incorrect actions would result in consequences such as higher death rates or greater damage. The CRISIS project started in May 2010. Despite delays due to a partner falling into bankruptcy, the project successfully closed in November 2013 in 42 months – 6 months longer than planned. The replacement partner introduced their XVR 3D world simulation technology around which CRISIS functionality was added. The key outcomes are:
1. The CRISIS interactive simulation and training environment that has delivered capabilities for training multiple front-line first response team-leaders and multi-level command, and across multiple emergency services in dealing with major or critical incidents. CRISIS comprises the FDX (Field Exercise) and CPX (Command Post Exercise) modules. CRISIS is deployed at our end-user partner locations – ISAVIA airports in Iceland, ANA airports in Portugal, and the BTP in the UK.
2. A Planner module that uses a timeline-based user interface to plan exercises and scenarios. It incorporates the Variable Uncertainty Framework for varying complexity and difficulty by re-combining scenario sub-elements. The Planner is coupled with the XVR simulation 3D world where avatars and assets would be activated by the Planner.
3. Associated with the Planner is the exercise Manager that enables instructors to control the activation of exercise injects during exercise run-time.
4. An IP-based Communications module to enable trainees to communicate with each other via the simulation world. This has the benefit of then being able to record all conversations, and simulating distance between avatars and realistic 3D soundscapes.
5. A mobile Observer tool that can be used on a tablet or iPad-type device or a smartphone-type device. This enables instructors to move around the trainees to record the instructors’ observations of the trainees’ performance, take photos of the training situation, make annotations, and assess the competency.
6. An After Action Review module that enables a review of the exercise and individual trainees’ performance almost immediately after the exercise. The AAR module shows on the exercise timeline events, communications, instructors observations and comments.
7. The Decision Support and Knowledge Management DSKM was intended to support real-time aggregation and consolidation of streaming observations and exercise events, and post-exercise data analysis. The DSKM use Topic Maps technology to create an ontology as well as a natural language interface to query the data using the Topic Maps-based Question and Answer technology. This is coupled with a combination of Esper complex events processing and Drools inference making technologies.
8. The Secure Integration Platform, SIP, comprises a tool-chain for software development, SIP Messaging middleware for message delivery, the SIP Request-reply middleware for managing persistent state of applications, and additional tools like JMS/RDF Gateway and Backend Application, and a Semantic xWiki.
9. Concepts developed within the CRISIS project, such as the competency- and performance-based training method, are being used in consultancy or research.
10. The work has been published in a number of well-regarded peer-reviewed conferences and demonstrated at several trade exhibitions and end-user type forums.

Project Context and Objectives:
SUMMARY DESCRIPTION OF PROJECT CONTEXT AND OBJECTIVES

Figure 1. Scene from CRISIS: Police stopping traffic at railroad crossing. A car had crashed into a train.
Objective A Crisis Management Simulation Training
… to develop a training program that integrates and shows how the CRISIS system … can be used for multi-level, inter- and intra-organisational crisis management training, emphasizes cue recognition, decision making, team coordination and human-interoperability.
STATUS = IMPLEMENTED AND DEPLOYED FDX AND CPX AND SUPPORTING MODULES, AT LISBON AIRPORT (ANA), REYKJAVIK AIRPORT (ISAVIA), AND BTP (LONDON)

CRISIS was conceived as a next generation, advanced video game style interactive simulation training system developed to help emergency service personnel and commanders prepare for crisis scenarios such as train and plane crashes, and terrorist attacks. Currently emergency personnel at venues such as airports must plan costly live training exercises, but the virtual reality ‘CRISIS’ system will allow them to prepare for these incidents at the push of a button. CRISIS presents emergency workers with an emergency situation in an interactive virtual world using a complex 3D view of the incident scene. Trainees navigate through the environment and build up a situation assessment of the incident which then guides their decision making process, all the time communicating with fellow trainees as the incident develops in front of them. The CRISIS software can be used by individuals or teams within the same organisation or across different organisations to train in decision making and team coordination. It can also be used to train commanders spanning multiple levels of command, in an immersive multi-player environment. Using the communications module trainees interact and communicate with fellow trainees inside the 3D world, regardless of their physical location.

Object B Exercise Planning After-Action Review and Virtual Test-bed
… to develop the capability for rapid exercise planning … and to link scenario generation and simulation … to an After Action Review system so that lessons from the exercise can be identified and learnt quickly.
STATUS = IMPLEMENTED – PLANNER AND XVR, VUF-BASED SCENARIO DESIGN AND DEVELOPMENT METHODS, OBSERVER TOOL, AAR MODULE, DSKM

CRISIS enables instructors to plan exercises where incidents presented have varying levels of complexity. Using the Variable Uncertainty Framework (Wong et al, 2011; Field et al, 2011), the scenarios can play out differently every time they are run with the system responding in different ways dependent on the trainees’ actions, with random events occurring in order to test reactions. Triggers can be set so that if a trainee fails to complete a certain action it can have a particular consequence. The occurrence of these events can be fully randomized or made to vary within a window of opportunity set by the instructor, or set to occur at a specific time. Such a range of variability enables a high level of re-playability of the simulation from a small set of events, as the VUF enables them to be combined in many different ways to create what would appear to be different situations to the trainee.

Objective C Game-based Interactive Simulation Technology for Training
… to develop a games-based interactive simulation system [with] high … interactivity [offering] … real-time cue recognition, situation assessment, problem diagnosis and planning under … uncertainty and confusion, rather than choosing between pre-defined options.
STATUS = IMPLEMENTED

Exercise scenarios are visually rich and evolve in real-time according to plans and user actions, requiring trainee to constantly assess and evaluate changing situation. The situation evolution is achieved by a combination of timing when events are activated and the use of ‘trigger zones’ where upon entry by an avatar, activate sequences of actions that can change the situation. Although not fully autonomous, this provides game-like interactivity, uncertainty and capability to generate confusion without expensive Game AI.

Objective D Human Inter-Operability and Organisational Adaptation
… to develop an understanding and framework for how decision making, team coordination and organisational adaptation occurs in crisis management at airports, … to develop more robust crisis management capabilities for dealing with unexpected events.
STATUS = IMPLEMENTED. INVESTIGATED DECISION MAKING, CHARACTERISED CRISIS SITUATIONS AND UNEXPECTED EVENTS, INCORPORATED INTO SCENARIO DEVELOPMENT METHODS

We investigated decision making and how teams adapt their plans and how such adaptation is coordinated by the team and team leaders. Surprise and startle were observed to occur in critical incident situations. Research focused on how startle affects decision making and how such situations can be re-created within the CRISIS virtual world in order to train users to deal with the consequences and complications that can arise from such surprise or changes in the situation. We developed the SPURS method – Scenario Planning for Unexpected events, Response, and Startle – a framework for planning, developing and configuring CRISIS exercise scenarios. A significant effort was also invested into identifying the skills that mattered in managing critical incidents. We identified major deficits in the existing training methods to train certain key skills; Assess alert, Assess risk, Categorise crisis, Communicate, Communicate information, Leadership, Familiarise with procedures, Formulate handling strategy, Prioritise information, Recognise strengths and weaknesses, Resist pressure, and Take control. We have referred to these 12 skill training deficits as ‘Training Gaps’ which became the focus of the training programme. We developed a training programme based on the NOTECHS (Non-Technical Skills) approach. NOTECHS was originally designed for CRM (crew resource management) training and assessing teamwork in aviation (aircraft cockpits) and later adapted for use in medicine (operating theatre teamwork). We identified behavioural markers for each ‘Training Gap’, i.e. behaviours that demonstrate good or poor practice of the competency, and have been applied in the Observer tool for recording instructors’ assessment and observations of trainee performance.

Objective E Distributed Training System
… to develop a system capable of supporting training distributed across at least four different organisations … with roles … operators to commanders …
STATUS = IMPLEMENTED; CAPABLE BUT NOT TESTED DUE TO END-USER SECURITY POLICY

The Secure Integration Platform, SIP, is the backbone and middleware that connects the different software components of the CRISIS system prototype. The SIP was designed to run over the Internet. A voice communications module was also implemented to enable trainees who might be remotely located to verbally communicate and coordinate with each other. User trials of the CRISIS training system prototype was successfully conducted over an IP-based network at each end-user site. However due to each security policy, we were unable to run the CRISIS across each end-user partner’s internal systems.

Figure 2. The CRISIS multi-player, multi-organisation, multi-command level (distributed) simulation environment during end-user trials at end-user partner ISAVIA, Iceland. Shown above are trainees representing the Transport Coordinator, Security Coordinator and the Rescue Coordinator.
Objective F Advanced Decision Technologies
… to develop advanced decision and knowledge management technologies and visualization techniques … where the information may be in multiple data formats and across multiple data sets, coupled with an advanced multi-modal interactive visualisation human-machine interface, to support decision making during CRISIS exercise planning and game-play.
STATUS = IMPLEMENTED: DSKM, AAR and OBSERVER TABLET TOOL WITH CONNECTIVTY PROVIDED BY SIP (REST-BASED PERSISTENT SERVICE GENERATOR)

The CRISIS DSKM comprises three main sub-modules and combines Topic Maps technology in a way that informs a Complex Events Processing (CEP) component to identify meaningful patterns in the stream of event data. These components are: (1) A central knowledge base for decision support and knowledge management, consisting of a Topic Maps system using an ontology of possible and planned events that can occur during the exercise, extended with inference results generated by the DSKM; (2) A real-time module running a network of Esper-algorithms carrying out filtering and aggregation of event streams that originate from simulator events and trainee actions during the CRISIS exercise; and (3) The knowledge-based reasoning module running Drools Expert, a forward-chaining rule engine based on the Rete algorithm that pattern match on the properties of the streaming data and if the preconditions are met fire the corresponding action-part of the rules to identify patterns of behaviours. The DSKM interactive visualisation provides instructors with maps and graphs of key performance indicators such as casualty numbers and transfers.

Objective G Software Engineering
… to develop a secure component-based software architecture that would enable rapid integration of existing and new components … which will … support large distributed training activities
STATUS = IMPLEMENTED SECURED INTEGRATION PLATFORM, BUT DUE TO TIME CONSTRAINTS, SECURITY MODULE OPERATIONAL, BUT NOT TESTED IN INTEGRATED SYSTEM

The Secure Integration Platform or SIP tool-chain is integrated into the Eclipse project as a set of plugins with their own actions. Users of the CRISIS middleware can model their own data messages formats or data format of persistent records in the Eclipse IDE Ecore modeller, and automatically generate as much code as possible of the target application from the model. It includes an Eclipse plugin to generate XSD files directly from Ecore files using an Ecore-to-XSD generator, which generates simpler code than Eclipse’s own plugin, in order to support effective communications. The CRISIS plugins set also provides a plugin for generating ActiveMQ-based SIP listeners and producers to simplify the development of CRISIS application on top of SIP. Moreover, there is also a REST-based Persistent Service generator for generating a full implementation of the persistent state service, an experimental JMS/RDF Gateway generator that generates a Java-based gateway application in-between JMS and RDF, and an Esper adapter generator. This has set in place a scalable architecture to support larger distributed training activities.

Figure 3. CRISIS Architecture and Data-Flow Model
Objective H User Configurability
… to develop the capability for a user with limited programming ability to configure their simulation environment (avatars, props and their behaviours) thereby allowing the same basic engine to be used to train crisis managers at different transport infrastructures, such as another airport, train station, or … football stadium.
STATUS = IMPLEMENTED.

Using the Planner and CRISIS-XVR end-users can create their own scenarios at respective locations e.g. Keflavik runway, Lisbon Airport terminal, BTP railway crossing. End-users are free to configure as many avatars and props such as fires and explosions from the XVR asset library. XVR also contains a number of other 3D environments that end-users may use. However, end users at this time cannot create their own 3d environments.

Project Results:
DESCRIPTION OF MAIN S & T RESULTS / FOREGROUND

The main S&T results from the CRISIS project are briefly described in this section and have been reported in detail in the respective deliverables.

D3.1 Revised Crisis Management User and Training Requirements Report
In this report we defined the user requirements for the CRISIS system. As the system was planned as a training system, included in this specification is a list of training specifications, including key competencies that crisis managers need to be well-trained in, and significant deficiencies in current capability to train Bronze and Silver commanders (in the BTP and their equivalents in ISAVIA (Iceland) and ANA (Portugal)). We have referred to these 12 deficiencies as the 12 Training Gaps. These Training Gaps include expert skills required to successfully to command and control crisis situations such as Assess Situation, Assess Risk, Take Control, or Resist Pressure. We used these Training Gaps to focus how the training methods and supporting systems are developed.

D3.2 Operational Concept
The Operational Concept defined how the CRISIS system would work in practice to carry out its training mission. This is where we defined how we anticipate various proposed concepts would be implemented. These concepts include the ABX ‘Airport in a BoX’ concept where the CRISIS Interactive Simulation Environment should be readily be re-used from an airport context to any other context. This has been demonstrated where the airport environments (ISAVIA and ANA) have been replaced by a totally different environment for a crash at a railway crossing environment (BTP). This is also a demonstration of the “support actions rather than roles” operational concept where the design of CRISIS should be to support the actions that are undertaken by personnel in a crisis situation, rather than specific crisis situations and specific person-roles in an organisation.

Other aspects include the alignment of command levels for system design purposes across the different end-user organisation. This is necessary to support ‘multi-user, multi-level, and multi-organisation’ training situations. In this way, the fire and rescue, medical, security and transport personnel, could use CRISIS to train across the different organisations, and include several first responder team leaders (Bronze) up to commanders and staff at the Command Post levels, e.g. the Emergency Command Post or Mobile Command Post at the airports, or the command and control room for the BTP.

Another key element of the Operational Concept is that of the Variable Uncertainty Framework, or VUF, as a ‘cheap’ alternative to increasing re-playability of the training scenarios. The VUF was conceptualised as a framework for managing complexity, certainty of an event occurring, and the number of events occurring simultaneously (Fig. 4a). Conceptually, this framework should provide a means for controlling the different ways by which events can be combined (automatically or by hand), such that to the trainee, the re-played situation appears to different to the trainee.

Figure 4a. The Variable Uncertainty Framework. By combining basic building blocks of an event in different ways, we can target different skill levels and create many variations of situations that appear different to a trainee to enhance re-playability.
Trainee response strategies are likely to be different between situations where the sub-events are spaced out with sufficient time to deal with each sub-event at a time, as compared to when sub-events occur very close to one another or overlap in time. The VUF can also create situations where the same elements are initiated in different sequence, leading to the appearance of them being different situations, even though the same event elements have been used.

Figure 4b. Controlling Variability and Complexity
Figure 4b illustrates the respective parameters that can be manipulated to create the variability of events in terms of onset time, end time, event duration, intensity of the event, and the event dependency on other events. For each of these parameters, the parameters can be influenced by controlling, for example, the duration of the time window that the onset time occurs, and a weighting about the certainty with which the onset time will occur (i.e. the event may or may not occur), and variability in terms of where in the onset time window that the event occurs (i.e. likely to occur early or late?). If, however, the training situations are required to be repeated in an exact manner, this can also be achieved by setting the variability to ‘no variation’. This makes producing scenarios cheap and fast.

‘Leaking’ information about the training situations to later trainees no longer becomes a problem as the system changes the situation by invoking the VUF. The greater advantage is that from a single set of events, numerous situations can be developed, and automatically if so desired. This is important, as the cost of developing training scenarios is expensive and time consuming.

D3.3 System Master Plan
The System Master Plan document integrates the user requirements with the training requirements in a way that defines what the system should do and how it should operate. To help us understand and define the design, we also developed a set of Reference Scenarios that were relevant to each end-user partner. These Reference Scenarios were used to drive the design and later in the process, to evaluate how well the system has catered for the end-user requirements.

Scenario Design, or how to plan and configure a scenario that will use the CRISIS combination of software components, was also investigated. Using the Reference Scenarios, the consortium investigated how Scenario Design could be conceptualised in relation to the functions that would be developed in CRISIS. Later in the project, we distinguished between spatial and temporal configuration of a scenario, as the placement of the avatars and objects in the 3D world or ‘stage’, are separate from when they occur. In later designs, the spatial configuration occurs through the XVR system, while the temporal configuration occurs through the Planner tool.

We subsequently developed a system architecture (connectivity through the Secure Integration Platform, software interface definitions, data structure definitions, security policy, example designs of the user interfaces, and expected deployment scenarios) to support the combined user and training requirements. Using an iterative development approach we developed a mock-up of the system that was used to incrementally refine the requirements and how they were translated into design.

We adopted a “Dual Path R&D Strategy”. In the first pathway, we constructed a stable prototype that can be used as a demonstrator of the basic concepts that were espoused in the original CRISIS proposal and the development effort was geared towards the deliverables as specified in the DOW. In the second pathway, we used it as a means to experiment with new ideas that emerged from WP6 Research and other work packages.

D4.1 FDX Module Implementation, and
D4.2 CPX Module Implementation
Our new partner E-Semble brought with them an existing product called XVR. It is a 3D world where avatars are controlled by human trainees who direct where they go and what they do in the 3D world in response to an un-folding situation that is managed by the simulation exercise controller or an instructor. XVR is heavily instructor driven where events evolve primarily as the result of an instructor moving avatars and vehicles and revealing hidden elements to change the state of a situation. Originally developed in a proprietary software platform, during the course of the project, our partner E-Semble ported XVR across to the Unity3D games engine.

XVR formed the basis for the FDX – the FielD eXercise module – that is intended to represent the 3D world in which the critical incident or emergency occurs and in which the trainees perceive the evolving situation, assess the situation, make decisions and direct the actions of the avatars that represent them in the 3D world. In the CRISIS project, new functionality was added to XVR in order to develop what is to become the FDX. Using a specially developed SDK, we made it possible to distinguish between efforts carried out by E-Semble on developing and maintaining their product for their existing customer-base, and efforts carried out by the CRISIS consortium to develop new capabilities through the research efforts. These additional outcomes were reported in D4.1 FDX Module Implementation report, and delivered as software.

In CRISIS, another module was developed to enable commanders or managers to participate in the command and control decision making process at an emergency command post. This is the CPX or Command Post eXercise module. It was developed as a ‘view’ into the evolving crisis situation in the 3D world of the XVR. By doing this, technically, we ensure consistency in the information being communicated by the 3D world to all users, regardless of whether they were operating the FDX or the CPX. (NOTE: If inconsistent data is required as part of the training scenario, this can be catered for separately by the instructors.)

Thus the Interactive Simulation Engine that runs the 3D world has two views: the FDX view in which the first responder and supervisors engage with, and the CPX view in which the silver or the airport emergency command staff, e.g. head of security, head of fire and rescue, will be provided views of the scene via, for example CCTV feeds, and reports from field units written up on a common “white board”, and decisions and actions written or recorded on the system simulating a commander’s operations or decision log. The commanders / managers will command the incident via voice or radio communications with the first responder team leaders or the Bronze / on-scene commander. The CPX has been delivered as software and reported in the D4.2 CPX Module Implementation report.

CRISIS software has been tested and deployed as a fully functioning laboratory prototype at our End-User Partner organisations ISAVIA, ANA and BTP. Our technical partner E-Semble had also incorporated CRISIS functionality into its commercial product, and that has recently been sold to a new customer, the London Fire Brigade (see Fire & Rescue, 1 Dec 2013, “London Fire Brigade to train with new simulation suites”).

Figure 5 - Overview of the CRISIS system modules.

FDX and CPX Architectural Overview
The CRISIS system architecture and layout is illustrated and briefly outlined in this section. Figure 5 shows the components that make up the CRISIS system. They are integrated though the Secure Integration Platform (SIP). The SIP connection has two channels: an ActiveMQ broker for publish/subscribe messages, and a Restful service for sending/retrieving data to/from the data store.

FDX: Key Functionality Created within CRISIS (Deliverable D4.1)
The software developed by the Consortium created the additional capabilities.

Exercise staff can start all CRISIS modules and processes to carry out scenario design, run a scenario and train, to evaluate training, or in administrator mode. Multiple trainees, in different roles, can move around in the virtual 3D environment, observe incident cues and analyse the scene of incident. All trainee avatars are clothed dependant on their role e.g. if the trainee is a Lisbon Airport fire fighter, then his avatar will be dressed in fire fighting gear that is recognisable as a member of the Lisbon Airport fire service. Generic as well as country and end-user specific vehicles, personnel and materials are shown in the virtual 3D world, to support trainees’ sense of immersion. Approximately 65 of such 3D assets were specifically developed in the CRISIS project.

Trainees and exercise staff (e.g. role players) can communicate with each other via the Comms Module developed by the Consortium (Fig. 6). It is a voice-over-IP application that provides capability for trainees to communicate with other trainees via voice while engaged in the 3D world. The system currently simulates real-world radio communication via one common radio channel using a ‘press-to-talk’ system. “Face-to-face” communication can also be simulated based on proximity, but was not implemented due to resource constraints. All communication within the CRISIS system is recorded and time-stamped so that they can be retrieved for subsequent training performance evaluation purposes and can also be linked to annotations made by the instructor in the Observer tool.

Figure 6. Trainee speaking with another avatar in the 3D-world, using the specially developed comms module.


Another communications capability developed within WP6 Research but not implemented in the FDX module is 3D sound within the 3D world. For example, if a truck with its engine running is located to the right of an avatar in the 3D world, the sound of the truck’s engine in the trainee’s headset would appear to come from the avatar’s right. This is an important ambient situational cue and is required in situations such as when going through rubble in search of survivors (avatars) who may cry for help. Without 3D sound, it will be impossible to use such real-world cues to make decisions about where rescue crews should be sent.

Instructors can build and change 3D virtual incidents to show a dynamic, evolving scenario and show consequences of trainee actions. The 3D virtual incident can be changed automatically based on time passed, trainee actions and/or positioning of objects (e.g. trainee avatars) in the 3D world. These automatic injects are handled by the Planner and Manager modules (see report D4.3 Simulation Design & Control Module implementation). By automating these scenario changes, the exercise staff workload is greatly reduced.

CPX: Key Functionality Created within CRISIS (Deliverable D4.2)
View 3D virtual incident with applicable avatar (supported by FDX). Multiple trainees, in different roles, can move around in the virtual 3D environment, see incident cues and analyse the scene of incident (Fig. 7). It is also possible to view the 3D world via camera’s (e.g. CCTV, heli cam or News channel feed). All trainee avatars are clothed dependant on their role (e.g. in Lisbon Airport fire fighting gear). Both generic as well as country and end-user specific vehicles, personnel, materials and incident objects are shown in the virtual 3D world, to support trainee immersion. Trainees can view victim information and triage these victims.


Figure 7. Trainee driving his virtual vehicle to the crash scene.

The scenario evolves over time, based on trainee actions (supported by FDX). The 3D virtual incident can be changed both manually and automatically based on time passed, trainee actions and/or positioning of objects (e.g. trainee avatars) in the 3D world. The automatic injects are handled by the Planner and Manager modules (see report D4.3 Simulation Design & Control Module implementation). By automating these scenario changes, the exercise staff workload is greatly reduced. Manual injects are created by the exercise staff via the XVR Facilitator interface.

Once a trainee (or role player) has entered a command post vehicle, his/her view is switched to a screen showing a shared log, whiteboard, map and video feed (Figs. 8 and 9). All trainees in this command post can write in the log and on the whiteboard and draw and write on the map. They can select which video feed is shown. The CPX Module thereby simulates together with the Comms Module all information sources generally available within a mobile command post.

CRISIS Control Centre, CCC is a standalone, central software module which manages starting and stopping of all other CRISIS modules and processes during design, run, evaluate or administrate mode. This application is developed to greatly ease up the start-up for end-users.


Figure 8. 2D top-down view of the incident scene used as a map of the incident.

Figure 9. Alternative 2-D map view of the incident scene, incorporating a log of actions and decisions, drawing functions, shared whiteboard, CCTV views of the incident.
D7.2 SIP Tool-Chain and Runtime Implementation
The CRISIS SIP or Secure Integration Platform is the middleware through which all data and processes of the interactive simulation system would be connected in a secured manner. The CRISIS SIP architecture consists of four major components, which are now described.

Component 1: The SIP Eclipse-based Tool Chain
The CRISIS SIP tool-chain is a set of tools which CRISIS application software developers use at development time for writing and/or generating their applications. The development and modelling tool-chain is based on the Eclipse open source software development and modelling environment.

The Eclipse project was started by IBM in 2001 with the aim to develop a modular Java based IDE for developing Java applications. Over the years, the project recognized the importance of modelling technologies and put a simplified version of OMG Meta Object Facility (MOF) into its core. The technology is known as Eclipse Modelling Framework (EMF). Later several third parties developed additional plugins for model manipulation and, more importantly for CRISIS, also for model-to-text transformation.

The SIP tool-chain is integrated into the Eclipse project as a set of plugins with their own actions, which are accessible from inside a CRISIS specific submenu. Users of the CRISIS middleware can model their own data messages formats or data format of persistent records in the Eclipse IDE Ecore modeller, and - using the CRISIS tool-chain – automatically generate as much code as possible of the target application from the model. It includes an Eclipse plugin to generate XSD files directly from Ecore files using an Ecore-to-XSD generator, which generates simpler code than Eclipse’s own plugin, in order to support effective communications.

The CRISIS plugins set also provides a plugin for generating ActiveMQ-based SIP listeners and producers to simplify the development of CRISIS application on top of SIP. Moreover, there is also a REST-based Persistent Service generator for generating a full implementation of the persistent state service, an experimental JMS/RDF Gateway generator that generates a Java-based gateway application in-between JMS and RDF, and an Esper adapter generator.

Component 2: The SIP Messaging Middleware
The CRISIS messaging middleware is built on top of Apache ActiveMQ and uses the common Java Message Service (JMS) API. Since the API is limited and only supports sending of Java objects, Java Strings or just plain byte arrays, we have developed a SIP layer on top of it to simplify the sending of complex objects. The SIP layer also simplifies the use of the ActiveMQ initialization API. As we need to support not only Java but also C#, the underlying object representation is XML. The developed CRISIS Eclipse tool chain supports the generation of custom listeners and producers of complex objects modelled within an Ecore-based data model

Component 3: The SIP Request-reply Middleware: REST-based Persistent Service generator
The REST-based Persistent Service generator was developed for occasions where the developer has specified a data model and wants to create a service that will store data based on the model. The generator is able to generate a full implementation of the persistent state service, which is based on specified data model, which uses the REST principle, and which provides very fine-grained access to the data properties. The generator itself is invoked by using the CRISIS > Generate REST Service action. However, the generator requires that the model is annotated with its special annotation in order to generate the service. The annotation provides some vital information for the service generation process

The CRISIS request-reply middleware uses the REST-based web services running on top of an Oracle GlassFish application server. The request-reply middleware functionality is used mainly for data persistency. Again, the user can model the desired data formats in an Eclipse Ecore model and then use the CRISIS tool-chain functionality to generate a full REST-based service. This service can then be deployed onto the GlassFish application server and used to store the application data records in a GlassFish configured database. Currently the only supported database server is PostgreSQL 9.x

The Persistent Storage Service was developed in order to fulfil requirements of the CRISIS system parts on saving various data from the system run. The JMS/RDF Backend Application is a Java-based application that transform an RDF model that is sent as a message into some JMS topic or queue, into a real RDF record in the Jena SDB database.

Component 4a Additional Tools: JMS/RDF Gateway generator
The JMS/RDF Gateway generator is an experimental generator that generates a Java-based gateway application in-between CRISIS events and events in RDF. The purpose of such a gateway is to be able to specify CRISIS events to be saved into an RDF database for a later use from the semantic web.
This allows for example a trainer, over SPARQL queries embedded into XWIKI pages, to check whether the actions of a specific responder are correct, which responders act correctly and which not, and so on. Since the RDF backed would be shared with several such gateways in the system, we have decided to write one general RDF backed that supports the SDB database and uses JMS as a transport mechanism

The generated gateway application is basically responsible for receiving a specified event from a specific JMS topic/queue, transforming the event into a set of RDF triplets (which forms a so-called RDF model), and then sending the whole RDF model into a specified JMS topic/queue. The general RDF backend application is connected to this topic/queue, picks up the received RDF model, and saves it to its SDB database. More information about the general RDF backend application can be found in section 5.3. The event type to be saved needs to be annotated using the JMSToRDFGateway annotationbetween the gateways and the RDF backend.

Component 4b Additional Tools: Esper Adapter generator
Esper is an open-source complex events processing engine. The Esper Adapter generator developed as part of CRISIS simplifies the development of Esper processing applications. The generator uses specific annotation information annotated to the Ecore model to generate a Java application that subscribes to the source JMS topic/queue and filters received events according to specified criteria using the Esper filtering engine. Events that pass successfully thorough the filtering are then published into the target JMS topic/queue. The generator also supports complex transformation of filtered events into events of a different event type. The model annotation has the name EsperAdapter and, as the case for the other annotations, it again supports several keys that drive the generation process

Component 4c Additional Tools: The Semantics XWiki
CRISIS also provides a user interface that enables users to manage and analyse the increased amount of information set around over the SIP in the form of events. The Semantics XWiki extension (based on the XWiki Open-Source Wiki tool platform) was developed during the CRISIS project in order to semantically associate information in a semantic knowledge base with events or to directly analyse events, in order to support trainers or to provide information to trainees.

Component 4d Additional Tools: Runtime: Publish-subscribe middleware
The SIP publish-subscribe middleware is based on the Java Message Service (JMS) API. It is using Apache ActiveMQ as its implementation. To ensure portability between various JMS implementations we have developed a thin SIP specific layer built on top of common JMS API. The SIP API was specifically developed to fulfil the requirements of the CRISIS project and is not a full JMS API implementation.

Component 4e Additional Tools: Securing the CRISIS platform with OpenPMF
Security in CRISIS is implemented using ObjectSecurity's OpenPMF (Open Policy Management Framework) product, which supports the definition, management of fine grain security policies in complex distributed systems. OpenPMF is a framework for enterprise wide access control. It enables a manager to protect information for confidentiality or integrity reasons. Such access control methods enables the allowing or denying of specific actions, e.g. remote invocation calls (e.g. in web services, REST, CORBA or so forth), subscribing or publishing of information (in JMS/ActiveMQ) or any other action, at different layers, including the application layer, and in a fine grained way. So, e.g. in CRISIS in a multi-organisation training, we can define that trainers of the different organisations can only see the training results of trainees of their own organisation, not of other organisations. And that only a supervisor is allowed to modify training results.

OpenPMF is based on “Model Driven Security" (MDS) which allows a user or manager to define security policies at a high level (e.g. "I want to protect this asset for confidentiality"), and the system would then generate low level, machine enforceable security configurations, e.g. access rules for different middleware. It also supports things like assessment of residual risks. Being an "Enterprise" security tool, we can manage the security efficiently in large scale by providing tools for policy definition and management, e.g. policy distribution and updates. There is no longer a need to work with a large number of configuration files. OpenPMF also support enterprise wide logging of security events, using a central console, as well as supporting multiple trust domains and communications over domain boundaries.

In CRISIS, we implemented OpenPMF for security enforcement of ActiveMQ and for the Glassfish Java application server, which is used for the implementation of the Persistent Storage. The OpenPMF Development Plug-In automatically applies or “refines” generic human understandable security models into the technically enforceable security rules models, especially in terms of access control and information flow control, for a specific environment, especially for software applications and their interactions. These technically enforceable rules are then automatically deployed to the OpenPMF Runtime Policy Repository. The Repository plays a central role in the OpenPMF Runtime Environment, which feeds the technical security models (policy rules) to the connected Policy Enforcement Points (PEPs). PEPs are runtime software entities that enforce the actual technical security model (i.e. policy rules) on the target application entity (Web Service, EJB, Web server, Database server, Firewall, Messaging Broker etc.).

In summary, OpenPMF automatically turns human-understandable, generic security policy models into matching technically enforceable, IT environment-specific security rules models, and enforces them across distributed PEPs.

The CRISIS SIP: Main Accomplishments
The main accomplishments achieved by this area of work are considered to be:
1. The leveraging of the Eclipse Modelling Framework and its associated tools to show the power of model-driven development for non-traditional and task specific systems such as CRISIS.
2. That model-driven security tools such as OpenPMF can be used to effectively secure the CRISIS system.
3. The Semantics XWiki, an extension developed to semantically associate user accessible documentation with different event types and then push this documentation for users down to the various part of the CRISIS system where users are making operational decisions.

D7.3 Exercise Planning and Inject Module
The Exercise Planning and Inject Module provides support for exercise planning and plan execution in the CRISIS training system. It is realised by three software components: (i) the Planner, (ii) the XVR simulation platform, and (iii) the Manager.

The Planner component is an end user tool used for designing an exercise by specifying in advance the temporal sequences of events and firing conditions for those events.

The XVR simulation platform is where the objects of the simulation environment – the 3D world, avatars and props such as fire trucks, fires, crashed aeroplane – are spatially pre-positioned and configured during the planning phase, and then presented by moving, hiding and showing or changing their states during the simulation run-time.

The Manager component is the run-time counterpart of the Planner. It manages the exercise plan and simultaneously monitors the exercise clock and the virtual environment maintained by the XVR simulation platform to determine whether conditions for firing events have been satisfied, in which case it initiates the corresponding changes in the virtual environment.

Figure 10: The Planner and XVR during exercise design. The Planner tool sets out the temporal sequences (foreground, with timelines. Bands (coloured rows) can be used to partition or organise the situation by events making it possible to activate or de-activate collective sets of activities. The XVR exercise design mode allows the instructor to spatially lay out in the 3D world where avatars and assets such as vehicles as well as effects such as explosions and larger fires, can be located, and hidden or showed, and trigger zones created.

Figure 11: Planning process using the XVR and Planner graphical user interfaces. Sequence explained below.
A Simple Planning Example
A straightforward planning example might involve the following steps (see Figure 11).
1. The user creates an item or group of items in XVR. In this example, there is a group named “FiretruckA”, which consists of a water tender vehicle, a crew manager, four fire fighters and a BA Officer. Initially, all personnel are inside the vehicle. This situation is represented by the Seated state.
2. The members of the “FiretruckA” group are listed in the left pane of the XVR.
3. The user can now add an inject that makes the fire crew exit the vehicle. To do this, the user move to the Planner Application and selects the New Inject command, which brings up a dialog to select the type of inject to create (Fig. 12). The user selects the Change State inject type, which brings up a wizard for configuring the inject.
4. Using the wizard, the user specifies a name for the inject, select the group to which the state change should apply (in this case “FiretruckA”) and the state that should be the new state (in this example, the Exit state). The user also specifies the triggering conditions which mandate the firing of this inject. In this example, the triggering of the inject will depend on another inject, namely the “Watertender drives into scene”. No delay has been specified, which means that the crew will exit the water tender immediately after that the water tender has arrived to the scene.
5. The new inject appears on the timeline of the plan. At this point, there are two injects in the plan. The first inject causes the group to move into location. The new inject depends on that inject and has been configured to occur immediately after the fire truck’s arriving on the scene. The checkboxes control the blocking parameter.

Figure 12: Trigger parameters of an inject.

An instructor can create a plan for events to be triggered or to occur automatically, or manually by activating injects. Firing conditions can be deterministic and probabilistic. Deterministic conditions prescribe under what precise conditions an event will be initiated – for example at a specific time point or a fixed delay after another event is fired. Probabilistic conditions are introduced to allow the same plan to be reused in multiple exercises but with variable outcomes and consequences. For example, a particular event may occur at a specific time point with a probability of 0.7 or the delay between a root event and its consequential event may be variable over a specified interval. Consecutive runs of the same plan will then lead to different outcomes based on the instantiation of random variables using a random number generator. This is an application of the Variable Uncertainty Framework VUF developed by the consortium.

Exercise plans specify the conditions under which individual assets or groups of assets change state or location. All assets states are defined in XVR and in principle the instructor may carry out all state changes manually using the XVR user interface. However, the introduction of the Planner and the Manager enables us to automate this work to decrease instructor workload, support reuse and repeatability by allowing plans to be saved and used again, and increase variability by allowing probabilistic conditions that will provide different outcomes in consecutive runs of the exercise.

D7.4 After Action Review Module
The After Action Review Module provides support for exercise evaluation in the CRISIS training system. It is realised by the Evaluator components that retrieves data about actions carried out by the trainee, communications made, and observations made by instructors, during an exercise, stored through the Secure Integration Platform, and presented to an instructor or analyst using multiple coordinated views display (Fig. 13). Unlike the present, exercise debriefing also known as the ‘after action review’, can be carried out immediately after the training exercise is completed using the CRISIS After Action Review module.

Figure 13: The Evaluator and its external interfaces to other modules.
With multiple coordinated views, the instructor can explore the data set, sort and filter the data, review data items and make annotations in order to assess how well a trainee performed, and how well the overall exercise went. Flagged situations are a particular type of data that allow the user to restore saved states of the virtual world. The evaluator realises this function in close cooperation with the XVR simulation platform.

During an exercise, data is generated in the virtual training environment, distributed through the XVR Event Portal and stored in the persistent storage of the SIP. The Evaluator retrieves those data and reconstructs a model of the exercise and presents this model to the trainees and the instructor to be explored during the AAR. The Evaluator is mainly a stand-alone tool with multiple views for various types of data, but it needs to access the XVR simulation platform if the exercise data set includes bookmarks of flagged states, because such states must be restored and explored in XVR.

The AAR module is also connected to the DSKM – the Decision Support and Knowledge Management module – through the SIP. The DSKM is an attempt to provide an automated analytics capability to speed up the trainee assessment process by finding meaningful patterns in the performance data using a combination of knowledge management ontologies and complex events processing technologies.

The AAR module automatically records all voice communications carried out through the systems inter-com system.

Figure 14: Example of the general filter function available in all text-based views. Clicking on the column header displays the filter configuration menu, where the user can select which values to include in the result set.
Text-based views support keyword search in all fields. Applying search to time fields also provides basic temporal filtering (Fig. 14). Communication data can be annotated to provide a textual representation of the contents. The module supports temporal navigation in the data set. The data from the exercise may be presented in a number of different ways to support after action review analysis and facilitate discussion.

The Map View.
The purpose of the Map View is to present an overview of the exercise area and the location of the different assets in that area (Fig. 15). This view provides a 2D representation of the virtual world as a complement to the 3D representation presented by XVR. The Map View is time aware in that it responds to the setting of the current time by displaying the locations of assets at that time point.

Figure 15. The Map View provides a two dimensional overview of the virtual world.
Time line View.
The Timeline View shows an overview of the events recorded in the exercise using a bubble view (Fig. 16). The view is dynamic and can be resized to display events in the interval selected. Each bubble corresponds to an event and the type of event and the time stamp are included in the bubble. Clicking on the bubble will display more detailed information about the event. Attachments are indicated by the paper-clip icon – for example an image attached to an observation report or an audio sequence corresponding to a communication event.

Figure 16: The Timeline View presents a time-ordered overview of the events in the exercise. This example includes observation events, communication events and flags.
Flags View.
Flags represent states of the world saved and bookmarked during an exercise. Some flags have been pre-defined during the planning of the exercise, whereas others have been introduced by an instructor during the exercise. Both types of flags appear in the Flag View (Fig. 17), but pre-planned flags carry some extra information in the Description field provided by the instructor who planned the flag. Selecting a flag and pressing the load button loads the bookmarked state in the XVR simulation platform.

Figure 17: The Flag View shows the flags created in the exercise.
Communications View.
The Communications View displays a list of all communication events (Fig. 18. The user can sort and search this list and listen to the recorded communication of a selected data item. After listening, the user may attach an interpretation to the data item in the form of a complete transcription, a summary, or a few keywords.

Figure 18: The Communications View lists the communication events occurring in the exercise.

Analysing the meaning and impact of a communication event requires information about the situation in which the transmission occurred. Methods and tools for communication analysis therefore have to preserve contextual information, manage large amounts of data, provide access to original data, and support navigation, filtering, and annotations. The Communications View displays the communication sequences that took place in the exercise.

Observations View.
Observations are short text reports provided by instructors or observers during an exercise. They provide a means of adding to the exercise data set remarks or questions that could contribute to the evaluation of an exercise. Figure 19 shows a screenshot from the Observations View. The Observations View displays a list of all observation reports events. The user can sort and search this list and review the details of a selected observation report. Observation reports can also have attachments. In principle, attachments can be anything digital that the observer can collect. In the current version, photographs are the only supported type of attachment.

Figure 19: The Observation View presents observer reports that have been submitted by observers. Attachments can be added to text reports.

Casualty Management View.
The purpose of the Casualty Management View is to provide an insight into the casualty management process. This process is distributed over space from the incident scene, through various casualty handling points set up by the responders, via the logistics required to transport patients, and to the hospitals. The process covers triage activities documented in triage classifications of individual victims to tallies of casualties at various locations. Figure 20 shows how the Triage View displays an overview of the number of casualties and their triage classification at various symbolic locations at a particular time point. In this figure, the locations are assumed to form nodes in a flow of casualties from the incident area to hospitals. The coloured bars represent casualties with different triage classifications. As would be expected, red and yellow patients are being rushed to hospitals, while green patients have to wait at the casualty assembly point.


Figure 20: The Casualty Management View (Triage) presents the number of victims and their triage tags over time. Victims flow from left to right in the view – from the accident scene (AS), to the casualty assembly point (CAS) and to hospital (HOS).

The Observer Tool
The Observer Tool module was designed for instructors or trainers to observe trainee performance during the exercise. The system was designed to run on a tablet or iPad-type computer or smartphone. Using WiFi connection, this allows observers to remain mobile since they are likely to be freely moving around the training room. The observer application has three main functions: to allow observers to make a general comment or comment about a specific trainee, to take a photo and to annotate the photo, and provide a means for instructors to efficiently make observations and assessments about trainee competencies. These competencies can be graded as being either good or poor. These observations are then imported into the Evaluator system to be used during after action review.

Competencies are different from text and photo observations in that they mark how well a trainee performs based on a set of criteria. The criteria, also known as behaviours or behavioural markers, are specified in the workgroup manager and are categorised based on the competency (or training gap) that they relate to. When the user accesses this page they are presented with the list of the behaviours. For each entry, they can specify whether the trainee has shown good performance or poor performance. These results are stored in the persistent storage and are can be assessed during after action review.

D7.5 Decision Support and Knowledge Management Module
The DSKM or Decision Support and Knowledge Management Module is intended to support exercise managers and instructors during and after the exercise. It helps them to maintain an overview of how the exercise unfolds, and see how resources are utilized and how in real-time the trainees perform their assigned tasks as the situation evolves. The DSKM also allows the instructors to flag or bookmark a time-stamped snapshot of the current state of the CRISIS-XVR simulator and supporting modules for replay and debriefing during After-Action Review or AAR.

During the exercise, the DSKM receives as input a continuous stream of events generated by the CRISIS-XVR simulation system. The events encode information, such as trainee movements, resource locations and movement, for example, movement of ambulances and personnel; items create, for example, that an area has been designated as the casualty assembly point; areas entered, and casualties being triaged. The DSKM filters, combines, and then refines this information on the basis of previously stored information and ontological background knowledge represented as Topic Maps.

At the final output stage, the DSKM presents several dynamically-updated visualizations containing aggregated statistics, which help the user in forming and maintaining an overview of the exercise. In addition, the DSKM provides data for AAR by storing its state when an instructor initiates a flag request. The stored DSKM state is mediated in an internal flag request, which is subsequently extended with CRISIS-XVR data and stored in persistent store for later access and use for debriefing by the AAR module.

The CRISIS DSKM Technology
The CRISIS DSKM comprises three main sub-modules that combines Topic Maps technology in a way that informs a Complex Events Processing (CEP) component to analyse identify meaningful patterns in the stream of event data. These components are: (1) A central knowledge base for decision support and knowledge management, consisting of a Topic Maps system using an ontology of possible and planned events that can occur during the exercise, extended with inference results generated by the DSKM; (2) A real-time module running a network of Esper-algorithms carrying out filtering and aggregation of event streams that originate from simulator events and trainee actions during the CRISIS exercise; and (3) The knowledge-based reasoning module running Drools Expert, a forward-chaining rule engine based on the Rete algorithm that pattern match on the properties of the streaming data and if the preconditions are met fire the corresponding action-part of the rules to identify patterns of behaviours.

The interface between the central knowledge base and the other two modules is implemented in TMAPI. This interface allows for both reading and modification of the central topic map. In addition to this intelligent core, the DSKM also hosts a Swing-based touch-screen GUI or graphical user interface. The DSKM utilizes a number of libraries, such as Ontopia for managing topic map data; TMAPI for seamless access to ontological and other forms of background knowledge; Drools for rule-based inference; Esper for real-time filtering of information streams; Log4J for handling textual communication with the user; and JFreeChart for plotting aggregated data.

The knowledge-management approach applied in the DSKM focuses on user-friendly querying of the above-mentioned knowledge base including inference results, using natural-language questions among other methods. In addition to this, the DSKM has an output sub-module consisting of a touch screen graphical user interface.

Complex Events Processing in CRISIS
The received input events are translated and connected as occurrences (facts) into the body of ontological background knowledge that is maintained in the form of a topic map. These facts, together with derived knowledge will serve as a basis for question answering, which allows the user to get insight into details such as which trainee did what when. As a complement to this, the Complex Event Processing module (CEP) in DSKM presents aggregated statistics and higher-level knowledge that can be derived from the basic facts.

CEP is especially useful for high-performance correlation, aggregation and filtering of events, based on simple event attributes like position or specific actions. CEP is, however, not very well suited for higher-level semantic analysis of events. For this purpose, CEP can be combined with rule-based reasoning and ontological background knowledge, which describes how the world is structured and how things work.

The basic occurrences (facts) are fed through event-type-specific channels into the real-time complex event processing module, implemented using Esper, a Java-based library for complex event processing (see http://esper.codehaus.org/). The facts are filtered, aggregated into more complex data, and refined using ontological background knowledge. The output is fed through data-type-specific output channels, each output channel feeding one chart or plot in the DSKM GUI. The individual charts and plots in the GUI subscribe to changes in the respective output channels, and therefore get automatically updated when new information is published in these channels.

The DSKM GUI
Other forms of widgets that populate the GUI include a timeline, which displays the elapsed actively-used training time; that is, training time excluding possible pauses for short intermediary debriefing during the exercise. The timeline is implemented as a swing JSlider, which subscribes to updates of the DskmTime class, more specifically to updates of the training time in this class; training time is derived from events received from CRISIS-XVR via the SIP. The exercise timeline is interactive, which means that the user can drag the timeline handle either backwards to replay previous data plots, or forwards to forecast a probable development of the exercise.

The user is offered two buttons, one above the timeline, called ‘Flag now’ and another on the top of the touch screen called ‘New note’. By design, both these buttons implement the same action, namely to create a new assessment item that is also time stamped and bookmarked with a flag, which means, among other things that a FlagRequest event is published in the SIP. The newly created assessment item is categorized according to the user’s choice, to be one of Fire fighting and securing scene, primary triage, medical treatment, transportation, communication, command and control, and finally, rescue management. The flagged assessment item that the user has created can be graded on a scale. The DSKM sums up and keeps track of the individual assessment item grades. The total sum is updated dynamically and displayed in the top right corner of the screen.

The top half of the DSKM screen hosts a number of horizontally scrollable charts and data plots. These provide the user with a quick overview of the exercise, by displaying summarized information and aggregated statistics of, for example, how resources are utilized across the crisis scene, the casualty flow from the scene to the hospitals, preventable complication and deaths, and trainee movement patterns.

The DSKM: Question Answering based on Topic Maps
The central knowledge base for the decision support and knowledge management module consists of a Topic Map system as defined in ISO/IEC 13250. The semantic event log Topic Map follows an ontology developed for supporting the use cases selected for the CRISIS project.

The Topic Map can be viewed as consisting of different sections. First, the static elements of a specific exercise as determined by a fully authored and runnable exercise plan. This is the information about the virtual-reality world and the planned events, before the exercise takes place. Second, the information directly derived from the events generated during a scenario run. Third, the additional information derived by the decision support routines processing the already existing information. The whole event log Topic Map is available for formal and natural-language queries through a graphical user interface.

As the DSKM is integrated into the full CRISIS system, an input abstraction layer tackles the transformation from a real-time on-the-wire event representation into high-level events. This transformation involves more than simple data format changes. For some event types there is an inherent impedance mismatch between the low-level and high-level versions. For example, avatar locations may be received as virtual-environment world coordinates, while the high-level events deal with an interpretation of coordinate sectors as discrete locations. This mismatch has to be bridged through complex event processing, by generating the high-level events from a collection of incoming streams containing lower-level events.

The layout of high-level events in Topic Maps format allows us to use a close mapping between the association concept in Topic Maps and sentence structure to support answering questions in natural language using the topic map as a knowledge base. This is intended as a user-friendly interface to complement the basic full-text search of the exercise log. For example, suppose that during the preparation of the after action review session, it is necessary to find out who triaged a specific victim during the exercise. The system allows the user to ask the English question “Which triager assessed victim25?” and get a graphical representation as explanation. Besides showing the relevant first responders in a graph, the user interface can also expose when each triage action took place. The reviewer can also browse the semantic logs to investigate hypotheses, such as whether the victim's condition actually changed over time, or whether a mistake was made and if this might be related to the stress level of the trainee making the triage.

D8.1 CRISIS MANAGEMENT TRAINING WITH CRISIS REPORT: THE CRISIS TRAINING METHOD
Training critical incident managers in CRISIS focuses around the principles of competency- and performance- based training. These principles ensure that:

• Training leads to in-depth understanding of the systems, tasks and teams,
• Skills are applicable in a wide variety of situations under a wide variety of conditions, i.e. good ‘transfer of training’,
• Knowledge and skills are lasting, i.e. high ‘retention levels’,
• Training is (cost) efficient

A comprehensive training design approach has been provided to develop training plans that enable competency based training for crisis management team members. The core of this approach is based on the 4C/ID model, which can be applied to simulated, live, or mixed environments. The training approach is then designed with a focus on the 12 Training Gaps.

12 Training Gaps
From a study of the training requirements involving 23 operational experts in 2011 (reported in D2.1) we identified a number of training deficiencies, or areas of competency where “much more training” was required. We referred to these deficiencies as ‘training gaps’. These training gaps represent the difference between what the operational experts considered were important and necessary to train in re-currency programmes, and what could be trained effectively using the tools and methods available. The CRISIS critical incident management training programme focused on these training gaps. The list below categorises the 12 Training Gaps according to whether they represent critical operation skills, non-technical skills and personal competencies.

(A) Critical Operational Skills
1. Assess alert
2. Categorise crisis
3. Assess risk
4. Formulate handling strategy

(B) Non-Technical Skills
5. Communicate
6. Leadership
7. Recognise strengths and weaknesses
8. Take control
9. Decision making
10. Prioritise information

(C) Personal Competencies
11. Familiarise with procedures
12. Resist pressure

Supporting Training Objectives from Planning to After Action Review
Within the CRISIS system prototype, we have also created functionality to assign sets of training scenarios and sub-scenarios to the competency to be trained. This presents the trainer with a library of scenarios that have been designed to train specific competencies. In addition, this functionality is integrated with the Observer Tool. If a training objective is, for example, ‘Leadership’, then the system will provide access to the Leadership descriptors and behavioural markers to support real-time observation and assessment of the trainee by the instructors, as well as the capability to make annotations about the trainees’ performance at that specific time and context. This functionality helps ensure that training is targeted and consistent from the setting of training objectives, to design and selection of training scenarios, to real-time assessment of performance, through to performance evaluation during the after action review.

The Crisis Multi-Level Training Approach
The CRISIS training approach supports development of a full-concept training programme for co-ordinators and commanders and includes co-ordination training for on-scene team leaders. The training is for all levels of crisis management and for all professions involved, from medics to police, from security to information managers, and ranging from individual training to intra-team training and inter-team training. Crisis management training targets the 12 Training Gaps at three levels:

• Multiple level training - This represents training inter-team aspects of crisis management, where a large number of participants are involved in a whole task (virtual or live) environment from different levels of the crisis management organisation (e.g. first responder coordinators and commanders).
• Single level training - A single level of the crisis management organisation is trained on their roles and responsibilities for a major incident. The other levels of the organisation and aspects of the incident environment are included as either actors, or constructive entities.
• Individual training - Training in the crisis management role for an individual, independent of other members of the crisis management team

CRISIS Scenario Design
Models for designing and developing training programmes such as 4C/ID do not go into detail on how specific scenarios should work out, except stating that sufficient realism and a rich context should be provided and hence operational experts need to be consulted to work out training scenarios. The CRISIS scenario design method combines (i) three factors – Relevance, Challenge, and Internal Consistency – together with (ii) the SPURS, and (iii) the VUF or Variable Uncertainty Framework into its methodology. Relevance pushes trainees past traditional thinking boundaries so that they should begin to recognise tasks that require attention. Internal consistent training scenarios should be possible to ensure realism in the line of events. Internal consistency also requires accurate information.

Real-world crisis events are never pre-scripted and unexpected events may occur at any time. This uncertainty can generate levels of anxiety about the incident among the trainees that can have either positive or negative consequences on their decision-making performance. Scenarios should also contain elements of surprise to challenge thinking as this allows trainees to perceive situations in different ways.


Table 1. Scenario design factors to initiate startle using the SPURS framework

SPURS Scenario Planning for Unexpected events, Response, and Startle is a framework developed within the project for planning, developing and configuring situations in a manner that would generate surprise and startle (Table 1). Based on a review of how horror and suspense movies create anxiety, surprise and startle. We have been guided by SPURS in how we use sound effects such as sudden noise to create fright or startle, or obscuring objects to create confusion, or setting off false alarms to create a sense of vulnerability. Then by synchronising vents with each other we can increase complexity or a sudden sense of overwhelming pressure during the decision making process.

Scenario planning based on principles of competency- and principle-based training focused on the 12 Training Gap, together with SPURS, provide a method for developing relevant and interesting scenarios. By incorporating the Variable Uncertainty Framework (its implementation in the Planner and in the XVR has been described earlier), we form another set of dimensions that an instructor can control to create un-expectedness and variation in the combination of scenario elements. Each situational element can be designed in terms of SPURS and controlled via the VUF (Planner and XVR) to time when and where these elements appear. By controlling this variability, the same situational elements can be re-used to make the scenario variations significant enough to enable re-playability.

Figure 21 is a flow-chart of a scenario developed for one of our end-user partners for a smaller-scale, part-task training that incorporates the various scenario design elements. Then to each node, variability and uncertainty can be incorporated, as is done using the timeline Planner Tool and the XVR Tool for spatial placement within the 3D world.


Figure 21. Flow chart of scenario developed a small, part-task training situation: Car crashed into train at railway crossing.

D8.1 CRISIS MANAGEMENT TRAINING WITH CRISIS REPORT: EVALUATION
The purpose of the evaluation was to determine whether CRISIS, as a 3D immersive training system, could be used to effectively train professionals involved in critical incident management.

A considerable part of this goal can be achieved by performing a regular Kirkpatrick training evaluation on a number of exercises at end-users sites. Such training evaluations are directed towards pragmatic value (Is training operationally effective? What should be improved?). For more scientific rigor, the training would need to be compared to an alternative situation (no training or standard training). Such an experimental setup would also require multiple measurements to compare the learning curves over time and to measure a transfer of training to new, untrained situations. This is extremely difficult given limited number of operational people and consequently a lack of availability of operational personnel, restrictions in time (generating learning curves takes time), liability issues (what if the experimental training leads to negative transfer?), organizational change. The baseline evaluation for the CRISIS System therefore is a standard Kirkpatrick level 1 and 2 evaluation. Where possible, attempts have been made to go beyond those levels.

Training effectiveness may be hampered when the training system is difficult to use. Usability issues may even be the prime cause for completely ineffective training. Usability analysis therefore is a core part of the CRISIS system evaluation.

Evaluation in projects such as this is to establish insight about system development progress from an operational point of view. After their hands-on experience, end-users are able to indicate which elements and functionalities of the CRISIS system prototype have been missed (not worked out) or do not function as intended. This operational perspective is important as it focuses on the most urgent, useful aspects and therefore prioritises future developments.

The methods used in the evaluation of the CRISIS system as a training platform are summarised below:

1. Training Software Experiences Questionnaire (TSEQ) (Wong, Passmore & Barnett, 2013). The TSEQ was developed for use at BTP that asked about trainees’ experiences with HYDRA (a commercial incident management training system currently in use at the BTP) and XVR, and CRISIS and XVR, and was completed after the training session had ended. Example questions explored whether the 3D view of the scene changed their assessment of the situation; and in what ways the 3D view helped them construct what was happening in the scenario.

2. Non-Technical Skills Behavioural Markers (NOTECHS) (Flin, Martin, Goeters, Hörmann, Amalberti, Valot & Nijhuis, 2003). NOTECHS was originally developed to investigate pilots’ non-technical skills, and contains four different categories – Co-operation, Leadership and Managerial Skills, Situation Awareness, Decision-Making. Each of the categories can be further divided into Elements and Behavioural Markers (BMs). BMs are described as showing examples of effective and ineffective behaviours, which, when analysed, provide information about training gaps. The behaviours were adapted based on the competencies identified earlier (D2.2) and on existing evaluation methods applied at ANA. The CRISIS markers were developed and reviewed together with the CRISIS end-users (ANA, BTP, ISAVIA), and assessment of the behavioural markers was carried out during the exercise by the observers and instructor(s) using the CRISIS Observer tool.

3. Physiological Monitoring (Pyd). BTP officers volunteered to wear a physiological monitoring device that was worn under clothing and throughout their participation in the training sessions. The Pyd measured and recorded physiological data such as heart rate, breathing rate, and skin temperature. This was used to identify and assess changes in trainees’ physiology, and was used to assess whether the CRISIS system was as stressful when compared with similar operational situations. In addition, the Physiological Monitoring Questionnaire (Smith & Adderley, 2013) was administered to collect other contextual information.

4. Training Assessment Forms. Using a 5-point Likert scale, trainees were also asked to rate their overall satisfaction for factors relating to each of the following three categories – ‘Exercise organisation and facilities’; ‘Delivery of the training by the instructor’; and ‘Application of the system for training’. Trainees were also encouraged to provide suggestions as to how CRISIS/XVR and the training could be improved, in addition to their opinions on the current training session.

5. System Usability Scale (Brooke, 1996). Participants were asked to complete the SUS questionnaire to assess system usability. Participants were asked to grade the system on a number of factors based on 5-point Likert scale. Questions include, ‘I think I would use this system frequently’, ‘I found the system unnecessarily complex’, and, ‘I found the various functions in this system were well integrated’.

6. Mechanics of Collaboration (Pinelle & Gutwin, 2008). The measurement explores basic teamwork operations, and is used to discover usability problems with users’ actions and interactions when completing shared tasks. The two main categories are Communication and Co-ordination. Communication consists of Explicit Communication (e.g. spoken and written messages), and Information Gathering (e.g. Basic Awareness). Co-ordination consists of Shared Access (e.g. obtaining a resource, and protecting work), and Transfer (synchronous and asynchronous transfer). The activities used for this research explored: explicit communication; consequential communication; co-ordination and organization of actions; monitoring in the workspace; assisting another; and protecting own work against actions of others.

7. Retention Questionnaire (Shepherd, 2013a). The RQ was developed to investigate the extent to which trainees were able to transfer the knowledge they gained from an initial training session to a subsequent training session. In order to determine the level of transfer of knowledge and/or skill that had occurred, trainees were asked before the second training session to provide base-level information about what they had learnt from a previous training session.

The Evaluation Results
The results in relation to the claims made in the original project proposal are briefly reported below:

E1. CRISIS training can be used for individual training The CRISIS System does not yet support a training session for an individual without being facilitated by role players an instructor and exercise staff. However, although the training tests focus on single-level training, individual training can be undertaken with limited support, because positions taken by trainees could be undertaken by role players.

E2. CRISIS training can be used for single command level training The results support the claim that single level training can be provided through the CRISIS system.

E3. CRISIS training can be used for multiple level training. The ISAVIA training illustrates that a two-level training (on-scene and EOC) is possible using the CRISIS System.

E4. A sequence of CRISIS training leads to a positive learning curve on competencies (for readiness training or refresher training). Due to constraints in testing possibilities, only one training session was executed at ISAVIA and ANA. While participants were positive on the trainability of the selected competencies (training objectives), no learning curve could be obtained on the basis of one training session.

E5. A sequence of CRISIS training leads to high retention (longer period in which the proficiency level of the competency is sufficient). Retention of information was shown to occur over a short period of time.

E6. A sequence of CRISIS training leads to positive transfer (to other situations than trained for). Further research with different types of scenarios is needed before this statement can be fully confirmed.

E7. Trainees and instructors can be familiarized to CRISIS system in a short period of time (half a day). Evaluations conducted suggest that both trainees and instructors were able to grasp the basics of the system (e.g. creating scenarios, learning to use the keyboard commands) in a matter of hours. Additionally, trainees and instructors needed few reminders when subsequent training sessions took place, probably because of the deep level of involvement they reported when training with the scenario.

E8. Trainees will be positiveabout the training value of CRISIS. Training systems such as Hydra presents situations using video, pictures and narratives and then asks trainees to discuss their plans. However, theydo not provide key aspects to critical incident training such as the ability to walk around the scene, and the opportunity for after action review based on every action and communication that arerecorded during the session. CRISIS participants reported that they were impressed with the system, found it useful and thought it created realistic soundscapes (3D sound), scenarios and communication styles, and welcomed the ability to add variability to the scenarios as the exercise progressed. Participants also commented that the opportunity of an on-screen, all-digital after action review was particularly helpful.

E9. Instructors will be positive on the training value of CRISIS. Results from the adaptation of the NOTECHS Behavioural Marker System (Flin et al., 2003) suggested that the instructors were positive about the training value trainees would receive from using CRISIS.

E10. Trainees will have no difficulty in using CRISIS. Feedback obtained from interviews showed a universally positive reaction towards CRISIS. Participants also explained that CRISIS could be useful for retaining skills, used as a ‘warm-up’ before real-life exercises, and training from remote, distributed locations. However, they explained that they would need assistance in starting the system, but not during use. ISAVIA participants explained that the integration of XVR and ISEE integration was unsuccessful, as it was difficult to understand the flow between them.

E11. Instructors will have no difficulty in using CRISIS. Instructors provided positive feedback with regards to training with CRISIS, particularly with regards to the fact that if a trainee does not respond to an event in the scenario, the appropriate consequences will occur. This was confirmed when the physiological data displayed a participants’ autonomic arousal changed as a function of the appearance of an unexpected event.

E12. Trainees experience sufficient realism during the scenarios. The end user feedback showed that they felt their scenarios were realistic but, more importantly, added realism to the learning experience. (42). The BTP end user scenario was comprised of two part-tasks (4C/ID model; Van Merrienboer, 1997) where highly complex skills could be learned efficiently in order to acquire competency in them. Participants also reported experiencing a level of pressure they would normally only experience during real-world crises, particularly the startle responses to the unexpected event injects.

E13. Developed support tools enhance the usability of the CRISIS training system for the instructor: Exercise Planner & Inject, Observer, DSKM, AAR and Evaluator. The exercise planner, inject, and observer tools were clearly appreciated and relieved the work of the instructor while enabling new ways of observing and adapting the training to ensure training objectives can be met. DSKM module was considered promising, but was not yet sufficiently mature for operational use. The AAR and evaluator was used in a limited way, but its great potential was recognized.


Discussion: The Evaluation Results in Relation to Theoretical Context Training concept and strategies
The training concept and strategy for this research was based on the core principles of competency- and performance-based training (Van Merrienboer & Kirschner, 2012; ICAO, 2011). Results obtained suggest that three of the four training goals have been addressed. First, both trainees and instructors were able to gain an understanding of the systems, tasks, and teams. Some technical issues did arise but these were addressed quickly, and new and simpler methods of system interactions were devised. Seconbd, participants also demonstrated short-term retention of knowledge and skills. Both concepts complement the other – the less time to learn a system suggests more time that can be dedicated to learning. Third, the application of the VUF (Field et al., 2011) provides an invaluable framework that ensures trainees may not experience the same training scenario twice, but at the same time, scenarios can also be tailored to address specific training needs. This confirms the third goal can be achieved: training is (cost) efficient (responding to the actual needs of the trainee and organization). Fourth, skills are applicable in a wide variety of situations under a wide variety of conditions – is not out of reach for CRISIS, but further research is needed to address the additional possibilities of what the system was originally intended for (Airport in a Box). However, the VUF and XVR have provided a step in the right correction.

Training design
The 4C/ID model (Van Merrienboer, 1997) was used to design competency- oriented training scenarios, where the scenarios for our airport end-users ANA and ISAVIA were whole-task, and police end-user BTP was to some extent, a part-task setup. The model is ideal for when adaptive and competency-based training is needed within an organization. The two measures addressed by the 4C/ID model provided a sound basis for CRISIS training. First, the use of the VUF to design high and low complexity scenarios allowed, to an extent, trainees to ‘prepare for the unexpected’, and forcing them to adapt their decision-making to the changes in the environment. This variability prevents trainees from adapting to a familiar routine and developing routine skills via automatic processing. Second, the part-tasks created for BTP allowed ‘emphasis manipulation’, where trainees could focus on, and train for a limited number of competencies and receive debriefings for them.

Beyond conventional training evaluation
Access was given by ISAVIA to a team of CRISIS researchers to investigate various aspects of emergency personnel performance at the full- scale airport emergency exercise held at Keflavik International Airport on the 5th of May 2012, which enabled an entirely new form of transfer study to be piloted (Shepherd, 2013b). In conventional transfer of training studies, a pre-test is given to trainees before they attend a training intervention. Some time after the training intervention, trainees attend an evaluation event at which a post-test identifies the extent to which the trainees are able to transfer their previously acquired knowledge and/or skills to situations that vary in some specific ways from those encountered during the original training session.

The Keflavik exercise involved over 300 emergency professionals in responding to a major air crash incident. However, this exercise was neither a training intervention nor an evaluation event at the level of the individual professionals involved. Rather, it was designed to test the overall efficacy of the current Airport Emergency Plan. In this respect, it is an example of the innumerable naturalistic and quasi-naturalistic settings in which the transfer of learning may be investigated, where some or all of the formal requirements for a conventional transfer evaluation are absent. In this instance, there was no single prior training event; there was no coherent cohort of trainees with a common training background; there was no possibility of setting up evaluation-specific activities during the exercise; and most of the information likely to be available was by means of retrospective, self-reporting by the participants.

The research approach designed for this event involved a retrospective survey of the knowledge and skills which individual participants felt were antecedents of their performance during the exercise. A questionnaire was completed by a third of those attending the exercise debriefing session, followed by in-depth interviews with a handful of these. The results provide considerable insights into the multiple prior learning experiences that are generalized to enable expert performance. They also provide strong support for the utility of what the study refers to as performance factor audits (PFAs) in providing richer information about performance antecedents than has hitherto been available from conventional transfer evaluation studies.

Scenario design
Participant feedback suggests that the design factors used to create the scenarios supported past research (cf. Walton, 2008; Swartz, 1991) investigating crisis management scenario training simulation design. Therefore, it is of vital importance that future research also implements these factors, in order to ensure that the scenario is relevant (applicable to the organisation’s training requirements), contains challenging situations (for decision-making and flexibility), and internal consistency (the scenario should be believable). Wack (1985) had also recommended that training scenarios should contain an element of surprise and challenge but that the level should not be too high (to avoid frustration) or low (boredom). CRISIS contained a series of unexpected events that produced startle reactions. However, the ‘difficulty level’ can be adapted if trainees experience problems with the difficulty level set by the instructors, and thereby addressing a trainee’s ‘zone of proximal development’ (Vygotsky, 1978).


REFERENCES
Brooke, J. (1996). SUS: A Quick and Dirty Usability Scale. In: P. Jordan, B. Thomas,
Field, J., Rankin, A., Pal, J. v. d., Eriksson, H., & Wong, W. (2011). Variable Uncertainty: Scenario design for training adaptive and flexible skills. Paper presented at the European Conference on Cognitive Ergonomics ECCE 2011, 24-26 August 2011.
Flin, R., Martin, L., Goeters, K-M., Hörmann, H-J., Amalberti, R., Valot, C. & Nijhuis, H. (2003). Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors & Aerospace Safety, 3(2), pp. 95-117.
Shepherd, I. D. H. (2013b) A multi-determinant approach to transfer of training research in naturalistic contexts, Paper to be submitted for journal publication.
Smith, M. & Adderley, R.A. (2013). Physiological Monitoring Questionnaire. A E Solutions,UK.
Van Merriënboer, J.J.G. & Krischner, P.A. (2012). Ten Steps to Complex Learning. A Systematic Approach to Four-Component Instructional, Design, 2nd edition. Routledge.
Wong, B.L.W. Passmore, P. & Barnett, J. (2013). The System Evaluation Questionnaire. Middlesex University, UK.

Potential Impact:
POTENTIAL IMPACT AND MAIN DISSEMINATION ACTIVITIES AND EXPLOITATION RESULTS

In this section, we summarise the impacts that we believe the project has achieved in the areas previously defined in the project’s proposal.

IMPACT AT GLOBAL AND EUROPEAN LEVEL
Improved operational readiness and preparedness.
CRISIS has developed the capability for crisis managers in airport management and transport policing to train on demand by developing a novel advanced working simulation training system that allows users across different organisations and nations to interactively experience and manage crises and security threats in simulated airport and transport network environments. This enhances their operational readiness and preparedness to respond to hazards or hostile actions.

Train on demand, train more frequently, train at different levels of the organisation.
Current training to respond to major incidents can only be carried out infrequently (once every two years at major airports), due to the cost and time needed for preparing the scenarios, coordinating the exercise, and minimising disruption to users and customers. CRISIS makes it possible to increase the frequency and type of training in an unprecedented way, by enabling training at different levels of the organisation, and training between organisations on a regular basis. Through the distributed training facility it is possible for organisations in different locations or nations to train together and thereby improve their cooperation and information sharing. Teams have the opportunity to be more practised in the different skills needed to manage a crisis or hazardous situation. This will significantly improve individual and team skill levels in procedures as well as in decision-making, cue recognition, and team coordination. All this is vital to improving operational security and developing strong organisations capable of continued operations within inter-dependent transport systems, so that they are resilient to major disturbances. Increased training frequencies also permit the dangers of unexpected emergencies and hostile actions in various transport systems to be reduced.

Training to the realistic demands of a crisis situation.
Through our research into the human element during crisis management we have developed a close collaboration and understanding of the target system users’ real and varied needs. CRISIS has investigated how current training methods and systems can be adapted to take advantage of the new interactive training environment and to address identified training gaps. In addition, the knowledge management-enhanced CITE Explorer After Action Review system provides a new capability to rapidly study the effects of a critical incidents dynamically, to learn from them, and therefore to develop and trial new ways and capabilities for countering such events. Such a capability will not only impact European crisis management organisations, but also similar organisations across the globe.

IMPACT AT A GLOBAL LEVEL: MARKET OPPORTUNITIES AND LEADERSHIP
Present limitation: Many simulation systems are not re-configurable and provide little support for after action review.
Most simulation systems and serious games developed for use in training tend to be very specific and, once delivered, can seldom be re-configured by users to expand their application or create new situations without further significant investment. Furthermore, while many systems have the capability to play back events in the 3D simulation world, they rarely have a means for showing how the user actually responded, the communications they made and how they coordinated actions with others, making after-action reviews difficult or of limited value.

CRISIS is flexible and easily reconfigurable.
Trainers can develop new scenarios using the CITE Planner tool and then easily re-configure them to produce different training experiences by making subtle adjustments to events and timings using the Variable Uncertainty Framework. Trainers can also use the Manager tool to dynamically inject variations in scenarios during a training session. Furthermore, CRISIS goes beyond the capabilities of current simulation systems in providing the CITE After Action Review Tool which allows the review of all communications and observer events immediately after training, and subsequent reconfiguration of scenarios to deliver training outcomes that are more closely matched to training expectations.

CRISIS’s user re-configurability will create new market opportunities.
CRISIS presents an ambitious concept to address the shortfalls in products on offer in the current market for crisis management training. Collectively, we have harnessed the experience and state-of-the-art technologies from partners across Europe –Denmark, Sweden, Germany, Iceland, Portugal, the Netherlands and the UK – to create a crisis management platform that is capable of user re-configurability, training-on-demand, after-action review, and exercise planning and injects management. By reviewing how planned and actual behaviours correlate, a solid and secure component-based Secure Integration provides opportunities for prototyping new concepts for specifying emergency response technology and procedures. Moreover, because CRISIS development is based on systematic user study and evaluation, and solid scientific research into decision making, team coordination, user behaviours and advance decision technologies, carried out by our partners.

CRISIS is scalable and transferable.
A key impact of CRISIS derives from its scalability and transferability. This product should open up new markets, both in Europe and around the world. For example, re-configurability extends the market to other airports and to industries as diverse as chemical processing, utilities and others. The re-configurability capability also means that local infrastructure can be re-configured to suit specific circumstances which can be kept secret from the developers, thereby protecting security. Because of the Model Driven Architecture (MDA), component-based approach, the CRISIS simulation system can be readily interfaced with other systems. This means the core engine can be transferred and embedded into other organisational systems in a variety of industries, including airports, the police, the military, businesses, and manufacturing plants. In addition, just as CRISIS can be re-configured for different environments, CRISIS can be scaled down for use in small airports, and scaled up for use in large airports by, for example, re-configuring the number of buildings, length of runways, number of fire trucks and emergency service personnel. The distributed training capability of CRISIS will also allow organisations such as the Police and Fire & Rescue services to apply CRISIS components to several different types of crisis management training beyond the air transport and transport policing applications developed initially. This CRISIS capability promotes the exchange of information between diverse users and organisations using tools and methods built into the CRISIS system.

Availability of ‘Next Users’.
A further impact of CRISIS is the availability of ‘next users’ of the technology, i.e. partners within the consortium who can take up the technology and use it to improve their own business competitiveness, or to take specific technology for use in other projects. Our industrial partner, E-semble has already incorporated CRISIS components into their XVR training software – a product that is currently in use around the world by various emergency services. For example, the CRISIS system has already been bought for training by the London Fire Brigade, and is available to their large international user base. Another Consortium partner, ANA, which manages Lisbon and several other airports in Portugal and around the world, through their Special Projects and Technology Innovation Area (a department), has expressed a keen interest in facilitating this process. Another partner, Object Security, has already deployed the secure middleware technology developed in CRISIS in another project.

Impact at an European and Global Level: Trust, Up-take and Acceptance of technology
CRISIS is reliable.
One of the key areas in developing trust by users in technology is for a technology to be experienced as reliable, consistent and capable of doing what is expected of it. When there is trust between user and the technology, we can then expect up-take and greater acceptance across the market. Another key factor in acceptance and uptake is usefulness. It must meet a need, and it must be flexible as user’s work practices change as they adapt to the new technology, and create new demands for its use. The technology needs to be capable of being modified quickly and at low cost. CRISIS provides a working system that has been tested to ensure a level of reliability that corresponds with that appropriate for its Technology Readiness Level (NB: as a research prototype CRISIS is at TRL 6; while components have been incorporated into fully operational products that have been distributed to users / customers). Testing has been undertaken at both the system level (including functionality and integration testing) and the user level (usability testing). It has also been used with all three CRISIS end-user partners (ANA, ISAVIA and BTP), involving live training sessions, and including formal evaluations of learning retention and transfer. User feedback has confirmed the usability and training effectiveness of the system, and their partners have incrementally increased their trust in the system as it has increasingly contributed to their core emergency training needs.

CRISIS is extendable.
CRISIS integrates a number of related technologies, such as VSL’s CITE Explorer system for after action review, to give the user very powerful functionality. The CITE Explorer integration opens up new and faster ways of carrying out an essential user task – the analysis of performance and lessons learnt. Advance decision technologies such as Topic Maps technology and the Protégé ontology-development tool are embedded in the Decision Support and Knowledge management Tool to assist in knowledge management, decision making and data analysis. The development of XVR for CRISIS has been combined with the existing OpenPMF and Secure Middleware, a technology developed by our partner Object Security which is also in use by the US Navy. By integrating XVR on to the Secure Integration Platform, we have created a secure environment for large scalable distributed training of users across different locations.

CRISIS is based on sound understanding of operational security demands.
Drawing on established field study methodologies for researching end-users, we have conducted investigations and analyses of user decision making and team coordination strategies. We have also studied how surprise can affect decision making, and have developed techniques for reproducing such situations within the simulation world. These have provided us with s clear understanding of the expectations, work practices, and context of the users’ work environment, in particular what traditionally occurs during major incidents and emergencies. Our partners NLR, AE Solutions, ANA and MU have worked with our end-user panel comprising ISAVIA in Iceland, ANA in Portugal, and the British Transport Police in London, to carry out these studies. From the findings, we derived the user and training requirements and operational scenarios, to ensure that what was developed is compatible with user expectations – a vital factor in ensuring our assumption of user acceptance, trust and uptake.

Wider societal implications
CRISIS will enhance inter-organisational inter-operability.
During the management of emergency incidents, many different teams may take part, including fire fighters, police, medical workers and the command and control centre staff in charge of coordinating the resources. Such teams have their own specific training needs before organizing the major incident rehearsal. The training provided by CRISIS supports information sharing between the organisations (e.g. through real-time voice communication and digital log sharing), and encourages cooperative planning to manage the crisis. In this way the CRISIS tool enables the participants to be better prepared for inter-organisational operability, adopting shared emergency management standards, and thus achieve a better team overall coordination and understanding of incident management.

CRISIS will minimise socio-economic cost of operational security preparedness.
An example of the way that CRISIS minimises socio-economic costs can be illustrated by its benefits to the Police partners within the consortium. Within the UK, the Police Service takes primacy in the management of major incidents whether or not they relate to crime. It is within this context that CRISIS is designed to be of considerable benefit. Major incident training currently consumes up to one year in preparation, has the ability to train a maximum of two senior Officers, and occurs only once every two years. Smaller training interventions normally run three times per year, and also train only a maximum of two senior Officers. A large Force may have in excess of 30 Officers that are of the required rank to take command of such incidents meaning that it is not currently practical or economical to thoroughly train everyone. The CRISIS system will help to alleviate this problem because of the low repeat-training costs and easy configurability of its scenario-based training exercises which are designed for officers operating at several levels of emergency response capability. These advantages ensure that all Officers can potentially receive initial training in managing major incidents and repeat training thereafter.

A further benefit to Policing is that officers will be able to undertake refresher training to hone their skills, not only by participating in complete training scenarios (i.e. full-task training) but also in individual sections of larger scenarios (i.e. part-task training) where it has been identified that there is a specific skills shortage.

CRISIS-developed technology can be readily adapted for non-security use.
ObjectSecurity and VLS are developing platforms and tools for model-driven and secure integration of systems in different domains, among them mission critical domains such as Air Traffic Management, and Intelligent Transportation Systems. Their development of an advanced, model-driven secure platform for information sharing and systems integration for the CRISIS project will be exploited as a basis for related products and consulting services.

Impact on Standards.
The project is in the process of contributing toward Topic Maps Standard ISO 13250, Topic Map Query Language ISO 18048.

SUMMARY NARRATIVE OF DISSEMINATION
Below we outline eleven methods for disseminating the research and development undertaken as part of the CRISIS project that has been promised in Annex 1 of the original proposal. The eleven methods are listed below, along with a commentary on how we have achieved them over the course of the project.

Publications and dissemination activities from the accompanying tables will be used to highlight the achievements. They will be referenced by their type (P = publication, and D = dissemination) and their row number (e.g. P-2).

(1) Identification of target groups and establishment of relevant contact lists.
The primary exploitation path for CRISIS is through our commercial partner, E-Semble. They are an SME that develops and builds 3D-virtual world-based training simulators, with an established customer base of over 110 customers around the world. Much of the CRISIS research have already found its way into recent releases of E-Semble flagship product, with new sales made to organisations such as the London Fire Brigade. Consortium partners will also work with E-Semble to extend their customer base, by supporting activities such as E-Semble’s International User Group Meeting in the UK, 18-19 June 2014, hosted by the Derbyshire Fire & Rescue Service.

Invitations have been received to present scientific talks and workshops, for example, at the annual South East Asian Network of Ergonomic Societies International Conference SEANES 2014 in Singapore, in December 2014, and other FP7 project meetings such as the FP7 META-CDM project meeting in Frankfurt, in November 2013.

Further expressions of interest have been received from aviation-related organisations such as the UK’s Air Accident Investigation Board as to the possibility for extending the prototype for use in training air accident investigators. Other collaborations are being discussed with the Serious Games Institute at Coventry University in the use of CRISIS platform carrying out research experiments into decision making in emergency and disaster situations, as well as opportunities to introduce the CRISIS training methods to the Romanian emergency services.

It should be noted that with the closing of the project and the budget, it is anticipated that any follow-on work will have to be funded directly by those new activities or by the partner willing to undertake the effort with the view that it will lead to further exploitation opportunities and benefits.

(2) Logo design for the project, which will reflect the project vision and give it an iconographic presence.

The CRISIS logo has the following elements to it:
• The main icon is a button to symbolise interactivity.
• Inside the button is a play icon to symbolise the use of gaming technology.
• The 3D aspect of the logo represents the use of 3D virtual worlds.
• Both the plus sign in the middle and the blue colour represent emergency response.
• The italic font and slant of the play icon represent a sense of urgency in responding to an incident.

(3) Web-portal of the project will be created. This website will include different services and sections which will aim at the different target groups of the project.
The CRISIS website, which can be found at http://www.crisis-project.eu contains the following dissemination material:
• Photos and videos of the CRISIS system in use.
• Project news, which includes project meetings, and conference and poster presentations.
• A list of peer-reviewed publications, including URLs that point to the repositories where the papers located.
• Project deliverables marked for public viewing.

(4) Events will be organized throughout of the project duration where the produced materials, activities, and technological outcomes will be presented to guests from the academic/scientific sector, the government, the cultural and tourist domain, etc.
The project concluded with a closing plenary, where external guests from both industry and related FP7 project were invited to attend. The plenary primarily consisted of a set of presentations describing both the research and technological outcomes of the project.

(5) Development and distribution of promotional material (e.g. brochures, DVD, etc.) in various forms though existing or granted resources. Informative material will principally have in view all target groups and be available for general use but also special-purpose and group-oriented material may be produced. Distribution will be done via many means but most effectively at organized presentations and workshops.
During the project two flyers (D-28 and D-33) were produced and disseminated at various exhibitions (e.g. D-6 and D-14). These flyers aimed to highlight the technological advances of the CRISIS project. The latest flyer is publically available on the CRISIS website in three languages (English, Portuguese, and Icelandic).

(6) Promotional and demonstration actions, to generate publicity and get opinions from the public: showing a movie in conferences and exhibitions, aiming to inform the public and attracting potential industrial users.
In terms of promotional actions, there was more interaction with industry than with the general public. These included several exhibitions (e.g. D-1 and D-35).

(7) Press publicity (e.g. interviews, press releases) will be sought as a complement to the participation in such events.
Throughout the project there have been a series of magazine articles (D-5, D-17 and D-31), promoting CRISIS to both industry and the wider public. There have also been two TV appearances (D-7 and D-16) and three press releases (D10, D-11 and D-12).

(8) Conference participation, where minimum 10 papers submissions through the project duration will be submitted in relevant conferences.
Throughout the duration of the project, the consortium published thirteen peer-reviewed conference papers (not including workshops). There were presented at the following conferences:
• ISCRAM the International Conference on Information Systems for Crisis Response and Management, http://www.iscram.org/
• Human Factors and Ergonomics Society Annual Meeting
• Naturalistic Decision Making bi-annual conference series
• ACM Nordic Conference on Human-Computer Interaction
• ACM European Conference on Cognitive Ergonomics
• ACM Conference on Designing Interactive Systems

(9) Journal publication, in high-impact journals.
The CRISIS consortium published three articles in the following peer-reviewed journals:
• Universal Access in the Information Society (Springer) (P-1)
• International Journal of Sociotechnology and Knowledge Development (IGI Global) (P-5)
• Universal Access in Human-Computer Interaction. Applications and Services (Springer) (P-7)

(10) Internal Research Workshops:
Two interim research findings workshop were held during the project. The first research workshop took place in Linköping, Sweden, in January 2011. The first day of the workshop included a steering committee meeting, progress update reports form all on-going work packages, final results from completed work-packages and presentation of on-going research within decision making and team adaptation. The second day was followed up by more presentations of on-going research in advanced decision technology and natural user interfaces.
To inform the consortium partners of further advancements of the various strands of on-going research, a second workshop was held in Amsterdam in September 2011. The first day of the workshop included a steering committee meeting and three research workshop sessions with presentation of the latest research findings. On the second day, version 0 of the CRISIS system was launched and demonstrated. This session was followed by a discussion about project dissemination and preparations for the mid-term review.

(11) Workshops for end-user and next-user communities:
The day before the closing plenary, the consortium met for an exploitation workshop to discuss how the software will be exploited post-project, as well as discussing how the consortium’s three end-users will adopt the software.

SUMMARY NARRATIVE OF EXPLOITATION
The Consortium’s Exploitation Workshop was held on Monday 21 Oct 2013. At that meeting we identified and worked out issues that will allow the Consortium to commercially exploit CRISIS as a whole or in parts – in terms of both product and services. The terms for exploitation and sharing of the returns from exploitation have been documented in the consortium’s Exploitation Agreement. The agreement lasts for 30 months after the official project end date, is described in detail in Deliverable 9.2 and can be summarised as follows:

(1) Foreground
The Consortium has identified a list of IP that is exploitable by partners and their ownership. This list has been organised according to whether the IP represents a tangible article such as a piece of software, or services such as the expertise and methods for developing training programmes using the NOTECHS approach. The Consortium has also established terms for using the CRISIS as a whole or its components for further research, and also for continued use by our end-user partners for training purposes.

In addition, a revenue sharing arrangement has also been established and documented in the Exploitation Agreement. It describes how benefits are to be shared, and how profits will be paid-back to the partners, and the duration of the agreement is expected to run for 3 years.

(2) Exploitation: Continued Access by End-Users
Under the Exploitation Agreement, our three end-user partners – BTP, ISA, and ANA – will enjoy continued access of the system in its current form in perpetuity. They will also continue to receive 1 year of technical and operational support at no cost to them. In the same way, partners will also enjoy the same right to continued use of the CRISIS system, as is, if employed in a not-for-profit use in other airports. However, if an upgrade required, a new commercial deal with E-Semble or other technology partners would be required.

(3) Exploitation: Continued Access for Research
Under the Exploitation Agreement, the CRISIS system and components, as is, may be exploited for research purposes at no cost to the researchers.

(4) Exploitation: CRISIS as an integrated platform
Features incorporated into XVR 2013
Support modules have been / will be connected to XVR

(5) Exploitation: CRISIS as independent software components
CRISIS may be exploited either as an integrated platform or as individual software components. These components are: the Observer and Evaluation Tool, Planner Tool, After Action Review module, the DSKM, and the SIP (Secure Integration Platform) and the Secure Wiki technology.

(6) Exploitation: CRISIS as know-how
This is much more difficult to track. Training method involving the use of related concepts such as the VUF and the framework of 4C/ID, developed to maximise the use and effectiveness of the CRISIS simulation system, may also be developed to be exploited by partners. They may adapt the know-how for other or similar systems, for both instructors and trainees.

(7) Exploitation: New markets identified
23 airports in Portugal and Portuguese-speaking countries have been identified by partner ANA as potential new users of the CRISIS system. ANA will be considering how these new airports can use the CRISIS system. ANA have also identified airport police and emergency services academies as other potential customers of the software. In ISA, the Icelanders are considering how the system as it, can be deployed at different airports in Iceland to enable distributed training, and what might be needed to adopt operationally. Amongst BTP users, training courses based on CRISIS and / or HYDRA – RailPol, Railway Accident Investigation Board.

List of Websites:

http://www.crisis-project.eu

final1-project-final-report-v26.pdf

Related documents

• final1-project-final-report-v26.pdf