Mastering the Value Function of Security Measures

Final Report Summary - VALUESEC (Mastering the Value Function of Security Measures)

Executive Summary:
The final report for the ValueSec project gives an overview of the overall results of the project. As an overall assessment it can be said that the project achieved all its goals and in certain cases went well beyond what was planned in the description of work. All deliverables of the reporting period were submitted to the European Commission / REA and all project milestones have been met. There have been some slight delays in the implementation of individual tasks and/or the submission of deliverables due to planning mistakes in the project plan, which already manifested in the first period and carried over to the second period of the project, e.g. work in WP6 and necessary re-orientation of the project’s second stakeholder workshop. However, these delays did not threaten the overall project implementation as necessary steps were taken early on in the project to soften the effects of the anticipated delays.
The project followed the approch of developing a methodological base in its first reporting period and building on this in its second reporting period with the technical development and implementation of the project’s prototype as well as its testing and validation. In all its efforts with respect to these objectives the consortium ensured a close collaboration with potential end-users and stakeholders. This led to the comfortable position that the consortium sees various viable routes for the exploitation of the project’s methodologies and tools. In general, the project and its results received positive feedback from stakeholders during different project organized events.
Project Context and Objectives:
The present document is the Final Report submitted by the ValueSec project. It will provide an overview about the deliverables produced and if necessary deviations from the Grant Agreement Annex 1, i.e. the Description of Work (DoW). The Final Report will demonstrate that the ValueSec project fulfilled all its objective during the duration of the project and in specific aspects went well beyond the objectives set out in the DoW.
The main objective of the ValueSec project was the development of a decision support methodology and accompanying software tool for policy level stakeholders in the field of security. The main characteristic of the methodology and tool is to provide an expanded cost-benefit analysis, taking into account three pillars of analysis, namely:
• A Risk Reduction Assessment (RRA)
• A (strictly monetary) Cost Benefit Assessment (CBA)
• A Qualitative Criteria Assessment (QCA)
In this, the project worked towards achieving the priorities as laid out in the FP7 Call SEC-2010.6.4-1 Cost-benefit analysis of the present and future security measures in Europe.
With the development of the methodology and tool the ValueSec consortium aimed to support policy decision makers in the field of security to make better informed decisions. The developed ValueSec methodology and the subsequent software implementation in the so-called ValueSec toolset increases the transparency of potential costs and benefits (in a wider sense) of security measures. Based on the individual and consolidated result of ValueSec’s analysis pillars, decisions makers will have a better opportunity to make the decisions based on their individual preferences. In this respect it is to be noted that ValueSec will not provide for a “total-optimizer” solution, which will provide decision makers with the “optimal solution” or “best decision”. The ValueSec tool will provide, however, for a better fundament to base a decision on.
During the implementation of the project, the consortium intensively worked together with relevant stakeholders to increase the relevance of the methodology and the tool to the intended target audience. An advisory board made up of representatives from policy, research and industry helped throughout the project to guide the implementation process and provided valuable comments on various issues. Three stakeholder workshops were carried out to actively involve stakeholders in the development process for the ValueSec toolset. The ValueSec methodology and tool was tested and validated in five individual use cases covering different decision making contexts.
For its implementation, the ValueSec project was broken down in seven work packages. The second reporting period covered activities mainly in the work packages five to seven, while work packages two to four were covered in the first reporting period. Nevertheless, work packages two and three, even though there were officially closed in the first reporting period, provided limited methodological support for the tool implementation, i.e. for work package five in the second period. Towards the end of the project all work packages were successfully closed, achieving all their objectives.

Project Results:
According to the DoW, the main objectives, following the milestones of the project, for the first half of the project were defined as the following:
• MS4 – Prototype tool available
• MS5 – End of tool tests and use case experiments
• MS6 – Final conference and evaluation of ValueSec methodologies and tools
All milestones have been accomplished during the reporting period with no significant deviations from the project plans as set out in the DoW. The only deviations to report is that even though the necessary deliverables for the accomplishment of MS5 were submitted on time, work with respect to the described activity was carried out beyond the milestone.

WP1 Project Management
Progress according to project plan. Organization of half-yearly project management meetings as well as additional meetings on needs basis. Organization of deliverables reporting. Established project structure as laid out in D1.1 including communication and cooperation rules, work exceptionally well, so that the consortium has a deep common understanding of the project objectives and how to cooperate to achieve them jointly. Submission (and subsequent acceptance) of amendment to the Grant Agreement to include additional partner, Technical University of Munich (DE), in the consortium. Organization of Final Reporting, including the set-up of a dedicated project de-briefing workshop.
WP2 Problem Analysis and Requirements
Progress according to the project plan. Work in WP led to a common understanding what decision making in a security context entails and what the common principles for decisions, the target group for the tool application and the decision rational for the ValueSec project will be. Furthermore, a structuring of a cost benefit approach has been established on which the further work in the project elaborated. All deliverables have been produced and submitted to the EC.
WP3 Theories, Methodologies, Tools
Progress according to the project plan. The work package provided a valuable understanding of which theories, methodologies and tools are available generally and among the consortium partners to cover the field of decision support in the security context. This has been accomplished by following a comprehensive assessment framework as developed in the work package. Additionally, the work package was instrumental in laying the foundation for the development of the quantitative assessment pillar, by providing an extensive assessment framework for qualitative decision criteria. All deliverables have been produced and submitted to the EC. Minor work was still done after its official closure to support the implementation of WP5, especially with respect to the refinement of a criteria catalogue and the accompanying value functions for the QCA.
WP4 Functional Design and Usability
Progress according to the project plan. WP4 brought together the theoretical foundations of WP2 and WP3 and acted as a conduit into the technical development in the project. Based on the work carried out in All deliverables have been produced and submitted to the EC.
WP5 Tool Development and Integration
Progress according to project plan. Work in the WP started but was delayed due to inconsistencies in the project planning, especially with respect to interactions with WP4. The work package developed the ValueSec toolset prototype, integrating the three assessment pillars of RRA, CBA and QCA in one assessment framework. The ValueSec toolset was implemented as a web-based solution.
WP6 Use Case and Evaluation
Progress according to project plan. Work in the WP started earlier then foreseen in the project plan, due to necessary input for the WP2 and WP3 and subsequently WP4. The work package developed use cases in five different decision making context. In total, 14 individual use case experiments were carried out and assessed, going well beyond the two planned in the DoW. Furthermore, the work package carried out the project’s second stakeholder workshop in Munich, Germany in June 2013.
WP7 Dissemination and Exploitation
Progress according to project plan. Website and public repository for deliverables established. Implementation of first public workshop successful, as well as the establishment of the project advisory board with first meeting held. Participation and dissemination of (preliminary) project results in various scientific and technical events. The work package supported the project implementation by communicating its results to external stakeholders. A number of scientific publications were prepared by different consortium members and presented at conferences and workshops. A major task for the work package was the integration of external stakeholders into the project effort. This was achieved by stakeholder engagement through the project’s advisory board as well as individual stakeholder consultations by consortium members. The work package supported the organization of project events as well as organized the ValueSec Final Conference in Brussels (BE) in December 2013.

Main Results
The main technical results of this reporting period can be summarized as follows:
• Organization of a project de-briefing workshop (WP1)
• Definition and description of a security related decision framework and decision rational for public stakeholders (WP2)
• Definition of a cost-benefit-analysis framework for decision in the security field (WP2)
• Implementation of a public stakeholder workshop (WP2)
• Comprehensive assessment of available theories, methodologies and tools for a decision support in security (WP3 and WP4)
• Definition of a methodological framework for the assessment of qualitative factors in security decisions (WP3)
• Formulation of a functional design for an extended cost-benefit decision analysis tool in a security context (WP4)
• Definition of the implementation, testing plan and technical architecture for the ValueSec toolset (WP5)
• Development prototype decision support tool for security policy stakeholders based on the ValueSec methodology (WP5)
• Testing of the ValueSec toolset and provision of an application guide (WP5)
• Definition of five major decision making contexts and subsequent use cases for the testing and validation of the ValueSec tool (WP6)
• Development and description of five specific use case in cooperation with stakeholders (WP6)
• Organization of two stakeholder workshops for the tool validation (WP6)
• Validation of the ValueSec toolset in five use cases in cooperation with stakeholders (WP6)
• Establishment of a project advisory board (WP7)
• Carrying out of 2 advisory board meeting (WP7)
• Carrying out of different stakeholder consultation meetings and participation in dissemination events (WP7)
• Publication of 19 scientific / technical papers (WP7)
• Implementation of a project website including a public repository for the dissemination of public deliverables and a private repository to coordinate work of the project partners (WP7 and WP1)
• Organization of a Final Conference (WP7)
• Development of an exploitation plan (WP7)
• Maintenance of a project website including a public repository for the dissemination of public deliverables (WP7)

Potential Impact:
Conclusions drawn on the technical and scientific work
Security stakeholders usually have to deal with closing security gaps and take precautionary measures for the future. Factors influencing their plans and decisions are usually numerous and interrelated. The main challenge to be mastered in the VS project was to decompose two very complex areas into a structure which is understandable and manageable by stakeholders:
• Analysis, description and modelling of the decision problem, and
• Methodology and tool development allowing for transparent evaluation of security measures
Both challenges were met and covered by VS. Models of typical decision problems were developed as so called Use Cases, and appropriate tools were selected via a structured evaluation of over 50 candidates. The underlying methodology and the resulting architecture materialized in the "three-pillar-solution" which was found the best decomposition of the complex security decision situations on one hand, and which provides an easy-to-understand decision support process on the other.
VS strived to and finally managed to accomplish this ambitious goal. Within the available time and resources, a high coverage of the main types of security decisions could be achieved. This was the dominating goal. On the other side, many technical details of the tools like functionality, domain coverage or user interface and reporting could not be fully developed to the desired detail. It was considered more important to satisfy the main evaluation areas RRA, CBA, QCA (horizontal view) at the cost this detail not fully modelled (vertical view). In other words: A good cross-section of tools available for varying tasks was considered more important than modelling all technical or commercial details (which may be different in new use cases).

Conclusions drawn on the project as a whole
From a technical and managerial point of view, VS over-performed. This may be said without pretention: instead of two use cases as requested in the DoW, VS developed and treated five! Not all use cases could be treated with all three tool types. Nevertheless, a total of approximately ten full experiments and four limited experiments were achieved. It would have been desirable to achieve some more progress in the overall integrated evaluation and in the generalized and common use of ontology-based database and -repository. Some analytical and practical work and progress in these two areas was achieved. Further detailing, however, had to be sacrificed to the benefit of a full tool coverage of all use cases and a large number of successful demonstrations.
During the whole period, VS was driven by an ambitious and cooperative atmosphere and high scientific and technical quality. No deficiencies in performance were realized which could have jeopardized parts of the project, let alone the whole one.

Value proposition
The vast majority of decisions within the security sector are characterized by a complex set of parameters and uncertainties. They range from quantitative factors like costs, investment budgets and benefits such as the intended reduction of damages, to highly uncertain and qualitative factors such as political objectives or societal acceptance. Funded by the European Commission’s 7th Framework Program, the ValueSec project is intended to support and enhance the decision making process with a toolset so that security measures reflect the interests of all stakeholders of the decision and of the consequences of the decision.
Decisions must be based on transparent criteria and a rigorous cost-benefit-assessment, incorporating quantitative and qualitative factors such as societal, cultural, ethical and economic implications in the overall analysis.
ValueSec creates a theoretical framework and a toolset of functionalities for planning and decision making under risk and/or uncertainty and facilitates better and more effective decision making. The ValueSec toolset proposes a toolset of functionalities combining three individual assessment pillars aiming to support rational decision-making by establishing methodologies to increase the awareness and knowledge of related factors and the transparency of the decisions-making process and of the driving decision parameters.
The ValueSec toolset is founded on three components (main pillars):
• Risk Reduction Assessment (RRA) to evaluate how good a security measure is in terms of mitigating a threat and its corresponding impact on the organization assets
• Cost-Benefit Assessment (CBA) to calculate quantitative implications of decisions on security measures, focusing on monetary costs and benefits
• Qualitative Criteria Assessment (QCA) to integrate different non-tangible decision parameters into an evaluation process, such as societal or ethical factors
ValueSec allows increasing the transparency and enabling systematic assessments of the wide array of (monetary and non-monetary) costs and benefits as well as risks associated with the security decisions. It will thus facilitate better and more transparent and effective decision making.
The Risk Reduction Assessment Module aims to provide a means for decision makers to justify their decisions by estimating/calculating the reduction of risk to be expected when the security measure is implemented and compare this to a reference case.
The Cost-Benefit Assessment Module aims at defining a practicable and straightforward cost-benefit-assessment relevant to the planned security measure. This entails estimating explicit monetary costs and benefits related to security decisions, i.e. costs of measures intended to avert or mitigate security events and those benefits which are expressible in money.
The Qualitative Criteria Assessment Module provides a toolset to analyze the, pros and cons of security measures in terms of factors which cannot be expressed in physical or monetary units, such as the societal ones. This module allows evaluating based on the revealed preferences of security policy makers with respect to these criteria.

The potentials of ValueSec were tested in five use cases during the project live. The decision making contexts in which ValueSec have been tested are the following:
• Public mass events
• Public transportation
• Aviation security
• Communal security planning, and
• Cyber security

Within these contexts, five use cases were elaborated by the consortium partners:
• Mass Event - Formula One Race Track, Valencia´s Street Circuit: Improved surveillance and detection systems
• Public Transportation - Railway Transportation Infrastructure: Improved Intrusion Detection and Damage Prevention for Passenger Trains in a Depot
• Aviation - Airport Security: New generation of Liquid Aerosol Gel (LAG) Scanners
• Communal Security - Flood Protection: Improvement of infrastructure and water management ; organizational measures to improve coping with flood events
• Cyber Security-Smart Grid attack: Improving Security of Energy Smart Grids from Targeted Viruses Attacks

Societal impact assessment in Horizon2020 using the ValueSec approach
ValueSec proposes a holistic approach to assess security measures in a policy context. However, the approach can easily be transferred to support the current priorities in the Secure Societies research program of Horizon2020 to explicitly include the assessment of societal impacts of security research. Within this priority of the research program, ValueSec is able to provide support to consortia developing proposal for submission to the program, the European Commission and third-party evaluators of research proposal as well as eventually consortia implementing security research projects.
The three-pillar-approach as developed in ValueSec provides a framework to assess the full impact of security research proposals and projects. In this respect the methodology can help to assess in how far results of a research endeavor contribute to the reduction of the risk exposure of society (RRA), in how far monetary costs and benefits are balanced (CBA) and which as well as how other relevant societal/political/environmental/… dimensions are affected. In the course of the project implementation, the naming and associated connotation of the ValueSec assessment pillars underwent distinct changes. ValueSec consciously does not use the term “societal impact assessment” or “social impact assessment” as its approach was designed from the outset to go beyond a pure “social sciences” oriented assessment of security measure but to proactively combine all relevant decision dimensions in its approach. Combining a risk-oriented, a monetary-based as well as a social sciences-oriented assessment approach yields a more complete picture of a security measure (and a security research project for this matter) than just concentrating on one of these dimensions. Moreover, it allows for the assessment of trade-offs, when assessing security measures in the sense that decision priorities (risk, money, societal factors) can be weighed differently. It is e.g. entirely possible that an innovative security measure does in fact fare worse on in the assessment of qualitative criteria/social impacts, however it might significantly improve the risk exposure of society and produce costs savings compared to the status quo. Only with this complete set of information can the measure be discussed and a societal consensus is reached to finally make an informed decision on implementation of said measure. Therefore, ValueSec, in fact, provides a far broader assessment of impacts than is commonly understood under “societal impact assessment” . What is more, ValueSec provides visual and numerical metrics to make it possible to compare alternatives based on a common set of indicators.
The following provides and indicative list in how far the ValueSec methodology is able to support the above mentioned three different stakeholders in assessing impact of security research.
• European Commission and third-party evaluators
o Setting of a clear indicator catalogue for risk factors, cost and benefit categories as well as qualitative factors to be included in the impact assessment of security research proposals and projects (as support for proposers as well as evaluators)
o Providing a structured methodological approach for a holistic impact assessment of security research
o Providing a structured approach to ex-post evaluate the impact of research projects and comparison to the ex-ante impact assessment, based on common indicators and methodology
• Consortia proposing research projects
o Provide a comprehensive impact assessment of security research projects and results based on indicative knowledge before a project
o Demonstration of “impact drivers” i.e. does the project perform exceptionally well in one specific assessment pillar
o Demonstration of expected improvements over the status quo, i.e. comparison to the reference case/state of the art and how the research results improve relevant aspects (or maybe which improvements come with specific trade-offs)
o Demonstration of relevant knowledge and/or competences in assessing the impact of the research results
o Demonstration of early stakeholder involvement (also beyond the consortium) in supporting the impact assessment
• Consortia carrying out research projects
o Methodology for continuously assess impact of project activities, e.g. through assessment of milestones
o Methodology for systematically involving stakeholder in the assessment of project activities and results
o Monitoring and follow-up on ex-ante impact assessment through continuous assessment of research activities and results
o Communication support for promoting and exploiting research results with stakeholders
o Assessment of project results using key performance/assessment indicators

The ValueSec consortium supports the further exploitation of the developed methodology and toolset for purposes of a (societal) impact assessment of security research and will discuss this with the European Commission/REA as well as interface with relevant projects and initiatives in the field, e.g. ASSERT.
List of Websites:
www.valuesec.eu

Final Report Summary - VALUESEC (Mastering the Value Function of Security Measures)

Share this page

Download