Standards Based Approach to Video Archive Search and Analysis
However, technology for technologies sake is of little value. Therefore the involvement of ethicist, legal experts and, most importantly, those users who must operate Video Archive installations and services to meet the needs of law enforcement agencies and judicial authorities, as well as those of civil protection and day-to-day organisational needs, is required. The SAVASA consortium covers each of these roles.
At its core, SAVASA will use existing reference technologies from the ICT field that have overcome the barrier of system interoperability/compatibility, i.e. between container and compression formats. The project will implement a prototype platform capable of demonstrating unified archive integration and an approach to common search and indexing. The project will also provide a set of tailored video analytics and semantic analysis tools that will provide added value to end-users, but which can also function within a legal and ethical framework. The project will provide an analysis of existing technical barriers/requirements in the standardisation of technologies and procedures, via the validation testing of a prototype platform with end users. Finally, SAVASA will implement a project structure that will ensure that RTD, Legal, Ethical and End User needs are properly balanced and addressed.
ANGEL IGLESIAS S.A. - IKUSI
Paseo Miramon 170
20009 Donostia San Sebastian
Private for-profit entities (excluding Higher or Secondary Education Establishments)
€ 607 330,10
Aitor Rodriguez (Mr.)
Sort by EU Contribution
FUNDACION CENTRO DE TECNOLOGIAS DE INTERACCION VISUAL Y COMUNICACIONES VICOMTECH
€ 339 664,74
STUDIO PROFESSIONALE ASSOCIATO A BAKER & MCKENZIE
€ 96 600
HI IBERIA INGENIERIA Y PROYECTOS SL
€ 292 346,40
DUBLIN CITY UNIVERSITY
€ 419 036,36
UNIVERSITY OF ULSTER
€ 397 602,36
INGENIERIA Y ECONOMIA DEL TRANSPORTE SME MP SA
€ 160 718
"NATIONAL CENTER FOR SCIENTIFIC RESEARCH ""DEMOKRITOS"""
€ 408 966,40
SINTEL ITALIA SPA
€ 214 180,80
MINISTERIO DEL INTERIOR
€ 229 051,15
JEFATURA CENTRAL DE TRAFICO
Grant agreement ID: 285621
1 December 2011
31 May 2014
€ 4 059 831,96
€ 3 166 266,58
ANGEL IGLESIAS S.A. - IKUSI
Final Report Summary - SAVASA (Standards Based Approach to Video Archive Search and Analysis)
This document is the Final Report of the SAVASA (Standards Based Approach to Video Archive Search and Analysis). SAVASA is supported by the SME-Security Research Call as part of the European Commission’s 7th Framework Programme and has 10 different partners, experts in the implementation, deployment and manipulation of multiple video surveillance systems.
At its core, the project uses existing reference technologies from the ICT field that have overcome the barrier of CCTV system interoperability/compatibility, i.e. between container and compression formats. The project implements a prototype platform capable of demonstrating unified archive search and retrieval based on a Cloud infrastructure and provides a set of tailored video analytics and semantic tools, ready to be implemented within the current legal and ethical framework.
This document provides along its different chapters, the information about the project context and objectives, main scientific and technological results, the potential impact, main dissemination activities and exploitation of results and the use of the of dissemination and foreground of the results created. Thus, the overall strategy for the dissemination of the project results and exploitation of the final prototype followed during the implementation of the project is reflected in the document. A list with the whole publications and dissemination activities is also included.
Project Context and Objectives:
The various technologies in video surveillance used by infrastructure operators (airports, railway, underground, etc.) have led to the installation and operation of diversified and non-interoperable video archiving systems. Thus, the exploitation of video surveillance information by law enforcement agencies in a legal framework is both technically and operationally impeded. This problem is exacerbated by the fact that diverse, oftentimes proprietary, technologies are used for the compression, indexing, storage and access of video surveillance information.
In order to overcome the current problematic, the SAVASA project proposes the creation of a video archive search platform that allows authorised users performing semantic queries over various remote and non-interoperable video archives. Access to the application is performed remotely (Cloud infrastructure) and platforms capabilities are offered as software services.
To achieve this ambitious goal, the project faces the following main objectives:
• A solution totally focused in end users’ needs. System’s requirements and architecture must be defined together with end users partners and advisors.
• A search engine that supports multi-modal video search across multiple archives addressing the problem of searching for relevant videos. Search functionalities based on the presence of objects, people, semantic concepts and scenarios, while time, location and other forms of contextual metadata about the captured video will also be used to support search.
• Interconnection between the Video Surveillance Networks (VSN) over existing network infrastructures, introducing a Video Surveillance as a Service (VSaaS) cloud service, which is specifically tailored to Video Surveillance requirements providing interoperability, centrally controlled secure remote access, unified indexing system and data storage. The VSaaS also controls the trusting and authorization processes, thus ensuring the security of each information/video transaction.
• Digital Watermarking techniques applied to guarantee the authenticity of the generated footage. The generated footage includes standard metadata that reflect for instance, time stamps, location of cameras, operator IDs, etc.
• Design and development of techniques for the detection of persons and objects in videos. The annotation results will be used to index and search videos.
• Implementation of video ontologies as a way of representing video, including: low-level video features, high-level concepts (e.g. objects/persons, actions, scenarios), inter-feature relations, inter-concept relations, and feature-concept relations.
• Reverse engineering of a select set of proprietary compression, encoding and container formats that will be integrated within current Open Source video libraries drawn from the field of multimedia.
• Privacy by Design (PbD) through integration of the legal, ethical and privacy controls or procedures defined by the Ethical Advisory Board that advises the project.
• Evaluation of system performance against principal international benchmarks in video search. Event detection tasks of SAVASA must be benchmarked against TRECVid (workshop organized by the American NIST).
To ensure that the research and development tasks meet the needs of end users, a constant integration and testing plan has been followed by the Consortium. This approach allows the platform to be developed in an iterative process. The Final Prototype Platform is the version that is functionally complete and contains all modules/components tested in previous iterations. This system will be the base for the future commercial exploitation of SAVASA.
The following paragraphs summarize the main achievements of the SAVASA project in terms of Scientific and Technical developments.
In the first place this section summarizes the core S&T results, according to the RTD objectives defined at the beginning of the project. Subsequent paragraphs structure the content of the section following the organization of the project into work packages and technical tasks. This way the technical work can be analyzed individually for each task and within the framework of its corresponding work package and the whole project development.
Pointers to the related deliverables are provided where necessary so that the reader can refer to them in order to find more details. Also, the most noteworthy scientific results are highlighted.
Core S&T results
• Focus on interoperability
o A unified archive integration layer has been created that is capable of making multiple remote or local video archiving systems available to end-users as a logically unified archive implementing SAVASA as a VSaaS (Video Surveillance as a Service).
• Multiple Archive integration
o A web services-based application that provides remote video search capabilities built upon the integration layer and that supports multi-modal video search over multiple archives in a distributed manner. This application is based on best-of-class technology that has been developed with open standards in mind.
o Provision of a verifiable chain of information from camera to operator, which dramatically reduces the operational costs of traditional custody chains to retrieve video footage as legal evidences.
• Open standards
o The integration layers of SAVASA allow the interconnection with remote, non-interoperable archive systems. It has been developed with open standards in mind, to avoid creating yet another non-interoperable system, but a framework easily accessible and extensible.
• User focused applied research
o The SAVASA platform has been finally created as a framework that permits its usage in a variety of ways, according to the different roles of operators and law enforcement agencies in their respective jurisprudences.
o The usage of ontology-based description of the content of the videos allows easy and natural extension of the search terms to handle new scenarios for specific end users.
• Ethical and privacy protection
o A coherent set of rules and recommendation of the Ethical Advisory Board that guarantees that neither legal nor ethical norms are compromised at a European level.
• Video analytics
o A set of enhanced video analytics modules that convert the large volumes of data of CCTV archives into descriptors that contain geo-spatial and time information as well as semantic annotations representing the activities of the scene. Also, integration of video semantic analysis and annotation towards permitting operators to perform searches across multiple archives based on a generalised hypothesis rather than concrete syntactic concepts.
• Contribution to standards
o The analysis of the existing standards related to the technologies involved in the project has shown the lack of standardization of procedures for encrypting video footage in CCTV system, and actions have been carried out to propose contributions to create such an standard or participate in existing standardization activities in that direction.
WP2 – Architecture and User Requirements Modelling
Task 2.3: System Architecture Design and Development Methodology (M3-M12)
The objective of this task was to obtain a global system design that provides enough flexibility to be adapted in the case of a change in user requirements during the project lifecycle.
The first architecture design of the SAVASA platform and its components was created in month 6 (D2.13 System Design and specification) giving rise to the specifications for prototype and achieving the milestone 2.1. The second and more specific version was created in month 12 (D2.14 System design specification) and it represented, with the end users’ specifications, the achievement of milestone 2.2 (see Figure 1).
Figure 1: Block diagram of the SAVASA modular architecture and data flows.
The SAVASA system was split into two parts directly related to the Offline and Online data flows. On one hand the Offline subsystem contains the modules that are involved in the generation of information from the video archives (e.g. the video analytics and semantic annotation) that is used as the basis of semantic video search, while the Online subsystem corresponds to the modules that make possible for a user to perform a semantic video search in real time.
This division is purely functional, and several modules were implemented into single SW and HW components. Indeed, due to the cloud-based orientation of the project, most of the functionalities were being deployed in the cloud which means that there are layers among the application and functional modules of the platform and the physical SW and HW components that run the system.
After having presented the architecture as a conceptual and modular view of the system, a first subdivision of the platform into modules (with identified functionalities, input/output, etc.) followed with clearly defined data flows. No specific details about implementation were given, although some preliminary options were noted for those modules whose implementation is less uncertain (details about HW and SW components are explained in deliverable D2.16 Software and Hardware requirements specification).
WP3 – Video Archive Integration, Control and Search Platform
Task 3.1: Integration and extension of open source and third party technologies (M3-M12)
NCSRD initially performed a literature survey of video (objective and subjective) quality evaluation, and integrated the ffmpeg transcoding tool into the cloud environment. NCSRD collected results about storage requirements, transcoding time and video quality of output videos. More specifically, NCSRD created a VM with typical capabilities and the effect of three parameters, i.e. frame per second, bit rate, and screen resolution, was investigated. MPEG-4 AVC/ H.264 was chosen as the encoder to be used in the SAVASA platform.
Figure 2: Video transcoding monitoring application working on the SAVASA cloud.
Vicomtech has actively contributed to this task, being one of the leading partners in the definition of the so called SAVASA Local Processing Node. Vicomtech checked how the video quality affects further video analysis tasks through running several tests performing person tracking on videos obtained for a range of encoder parameters. It has been showed, that personal tracking is not severely dependant on the quality settings of the encoder.
IKUSI has made a study about the more common CCTV systems and proprietary video formats used by end users. Mainly, different forms of input and output video formats are studied and their suitability for the implementation in the SAVASA project. This is being done in terms of international acceptance, considering internationally adopted standards, wide spread use by manufacturers, implementation support, and video quality provided by various video codecs.
The work in Task 3.1 was reported in deliverable D3.11 Integration and extension of open source third party technologies video processing and codec technologies.
Task 3.2: Global control platform development (M3-M27)
In this task, NCSRD studied the available APIs of the governor application of the cloud (built-in into OpenStack software) and has provided monitoring information to the SAVASA global controller.
In parallel, IKUSI has worked in the design and implementation of the SAVASA’s Global Control Platform. This platform is responsible of controlling the overall SAVASA solution and provides to users the necessary management interfaces. IKUSI has developed the user interface that will be used by SAVASA users (consumers, producers and the system administrator). The work focused on defining the fundamental functionalities offered by the Control Platform to the users in different scenarios and situations. The interface has been tested locally for the moment. The process of creating, sending and receiving requests (judicial reports) has been defined and implemented. Moreover, work has been done in the implementation of the Ontology tree and related events search functionality in the SMA according to the web services of HI-IBERIA.
Figure 3: Query Manager view of the SAVASA search interface.
Deliverable D3.21 Global control platform development, which reports the work in task 3.2 was closed in month 15 without any delay.
Task 3.3: Implementation of integration layers (M6-M17)
After the definition of SAVASA video formats, pool of reference videos, and software/hardware solutions adopted in task 3.1 and task 3.3 obtained further experimental results.
NCSRD implemented the final transcoding solution and collected the relevant evaluation results for importing any type of video footage to the common SAVASA platform. More specifically, the video transcoding module -converting the producer specific video data into a common video format - is extended with a new web interface and with reporting/monitoring capabilities supported by the Global Controller. In this task, emphasis is being given on the quality of transcoded videos.
During M16-M17, this task studied more thoroughly the commercial video recording and storage platforms and integration issues with the SAVASA platform. The software functionalities of several commercial surveillance products have been investigated and a list of video exporting options has been produced, exploiting the relevant SDKs. In this direction, Genetec's SDK has been obtained to allow full access to the CCTV system that has been deployed at IKUSI facilities.
The outcomes were included in a paper, entitled "The Impact of Video Transcoding Parameters on Event Detection for Surveillance Systems", presented at IEEE International Symposium on Multimedia (ISM) in California, USA. Commonly used algorithms for motion and person detection are briefly described therein, with emphasis in investigating the optimum transcoding configuration parameters. The analysis of the experimental results reveals that the existing video quality metrics are not suitable for automated systems, and that the detection of persons is affected by the reduction of bit rate and resolution, while motion detection is more sensitive to frame rate
The work carried out has been reported in deliverable D3.31 Implementation of layers for the integration of proprietary hardware and software elements.
(RP1) Task 3.4: Device, Network and Search Security, Access control and Authentication (M6-M24)
The work of the task focused on building the cloud network infrastructure and in establishing the secure remote access and the authentication within the system architecture of SAVASA. The infrastructure required for integrating into the SAVASA cloud, the Local Nodes at Producers' premises and the network configuration for allowing secure remote access to Consumers, was implemented in Task 3.4.
NCSRD established secure Local Area Network (LAN) connections between the Cloud and the Local Nodes/Consumers using Virtual Private Network (VPN) technology. NCSRD also developed a client-based and a gateway-based VPN solution. Moreover, NCSRD configured the IPSec Site-to-site VPN for the support of the SAVASA VPN infrastructure. The configuration developed supports dynamic addressing (needed if behind VPN), NAT traversal and roaming access support. In addition, NCSRD performed preliminary in-lab validation under various traffic loads with properly generated traffic.
To further enhance the security of Layer-2 VPN and encrypted IP data packets (Layer-3 VPN), Task 3.4 studied how tokens issued by Keystone and can be used by clients (Consumers) to sign their API calls. The OpenStack API endpoints are responsible to validate these requests. This task justified the need for tokens and the way that are used by OpenStack API endpoints for SAVASA cloud user verification. 23
One solution could have been to supply each single API request with a username and password. This is a straightforward solution, but an insecure one, since username and password have to be stored to environment variables. The usage of tokens addresses this shortcoming. The advantages of tokens are that are temporary and short lived. In other words, it is safer to cache them on clients than username/password pairs.
Deliverable D3.41 Network and Search Security, Access control and Authentication – version 1 reports the work carried out.
Task 3.5: Cloud computing architecture for VSaaS (M5-M26)
NCSRD has developed a fully functional cloud infrastructure (i.e. cloud controller, cloud nodes, storage) that can address the needs of the SAVASA project. For this reason, the Cloud Architecture has been upgraded to the most recent code name "Essex" for additional stability and better networking support. Afterwards, NCSRD created pre-configured "flavours" that fit SAVASA VSaaS needs, used for VM creation. At last, NCSRD created VM images to be used as instances at various producer sites and in the Cloud core.
OpenStack has been implemented. We have offered selected performance measurements for investigating the performance and the stability of the SAVASA Cloud platform. This task presented the methodology that the SAVASA platform has benchmarked. As it is clearly shown by the performance metrics, the number of virtual CPUs and the memory size of Virtual Machines are tunable parameters, which have significant impact to the overall performance. Moreover, in some cases like the sequential create, read, delete of data files there is no significant impact of upgrading VMs into larger flavors. The hardware resources that will remain can be spent to other functionalities with real impact.
Finally, during stress tests, the SAVASA platform has been operated into its limit. Knowing this limit is very useful for the end-users since the capacity of the platform can be reached. Last but not least, knowing beforehand the maximum performance of the SAVASA platform through benchmarking, the proper operation can be regularly monitored or the malfunctioning can be more easily located.
The work carried out is reported in deliverable D3.51 Cloud computing architecture for VSaaS – version 1.
Task 3.6: Secure Multiple Archive search (M10-M27)
Two use cases have been identified: a) metadata-based search, and b) semantic annotation search. The initial integration of semantic annotations into a searchable database for the TRECVid 2012 surveillance event detection is currently being assessed for test deployment into the cloud. The work carried out has been reported in deliverable D3.61 Secure Multiple Archive Search – version 1, submitted in month 15. Furthermore, DCU co-ordinated a proposal for a demonstration at the ACM International Conference on Multimedia Retrieval 2013 to show the integrated interface used for TRECVid2012 (see Task 4.7 of WP4 & D7.21). The proposal was accepted.
The multiple archive search component of the SAVASA framework has been prototyped, exposed as web services, deployed within the cloud framework, connected to the Global Control Platform interface and applied within the TRECVid evaluation framework by users from the SAVASA project partners. Core use cases for the SAVASA framework have been demonstrated and valuable feedback on the system performance has been gathered. The framework architecture – including the cloud-based deployment, standards-based data exchange and web service communication – helps to provide a search system that is adaptable and extensible. These characteristics are important to fulfil the varying user requirements identified by the project.
Figure 4: Search engine and metadata datastore modules in the SAVASA framework.
Task 3.7: Video watermarking (M10-M27)
This task tackled the design and development of algorithms for digital signature of videos, including watermarks that can turn videos into legal evidences for post-investigations carried out by Law Enforcement Agencies. The work carried out has been reported in deliverable D3.71 Digital Signature Watermarking Algorithms – version 1 submitted in month 15.
During the reporting period 2, Task 3.7 presented two options for the watermark storage and distribution. The signatures are either stored separately from the image data, in an external file; or they are included in the video stream along with the encoded video frame. In both cases, the existence of the watermark signature is not hidden. From these two options, SAVASA project preferred the first approach, i.e. saving signatures in an external file, since this approach is universal and not specific to any video codec. Moreover, separate storage of the signatures allows for remote management of the security or authenticity information, which could be provided upon request.
WP4 – Video Archive Search, Analytics and Semantic Annotation
Task 4.1 Person Identification and Analysis (M12-M27)
VICOMTECH started the development of this task in advance (despite the expected start date: M12) due to its participation in the TRECVid joint activity, in which person detectors and trackers were used as part of the modules to be integrated. Within this activity, VICOMTECH developed a particle filter method based on the Rao-Blackwellization technique and Data Association concepts, to allow the multiple detection and tracking of objects in video sequences. The use of GPU implementations allowed improved processing speeds.
Figure 5: Person detection results in some of the considered scenarios, using perspective detection, motion tracking, face detection and appearance-based detectors.
VICOMTECH worked on several algorithms for the detection and tracking of persons in video sequences especially focused on an operative tool, e.g. development of an annotation tool for database generation, implementation of CPU-GPU detectors-by-classification, training of classifiers, and calibrated detectors for enhanced performance. The status of the work is reported in deliverable D4.11 Person and object identification, tracking and localisations components – version 1, submitted in month 15, and in D4.12 Person and object identification, tracking and localisations components – version 2.
As a result of the scientific work carried out in this task, a paper regarding the effective exploitation of the perspective information of the scene was published in the MMM 2014 conference.
Task 4.2 Object Identification and Tracking (M4-M27)
DCU developed and implemented algorithms for video pre-processing, object identification and tracking with the contributions from VICOMTECH. This involved work on background subtraction, assessment of suitable feature descriptors, generalisation methods and classifier configurations. Motion trajectory based approaches were implemented using a selection of HOG/HOF/MBH descriptors, a bag-of-words based clustering approach and a RBF kernel SVM classifier. These algorithms were applied and evaluated in the first instance on the TRECVid dataset, using labelled surveillance video from an airport setting. Evaluation and improvements have continued using standard computer vision and human activity recognition datasets.
VICOMTECH defined the concept of object detection and tracking as the procedure that allows detection of any kind of object. The work has focused on providing generic tools to annotate and train classifiers that can lead to the detection of objects of interest.
DCU with the help of ULSTER and VICOMTECH co-ordinated and prepared the a submission of the publication “An information retrieval approach to identifying infrequent events in surveillance video”, to the ACM International Conference on Multimedia Retrieval 2013, based on further analysis of work from TRECVid 2012. The submission was accepted for publication.
Task 4.3 Geo-Localization of Artefacts (M12-M27)
This task started in month 12, in spite of some previous experiences of VICOMTECH in the area of GIS. Standards for the definition of exchangeable metadata file formats were studied, such as ISO 22311.
The geo-localisation results are reported in deliverable D4.11. Most significantly, the geo-localisation work has led to the definition of the camera coverage area and the implementation of a GUI tool to determine such area for any video. The tool generates geo-localisation information from the field of view of the camera that can be later included into the metadata of the videos.
Task 4.4 Privacy Masking (M12-M24)
A study on cryptographic standards used for video encryption was pursued. Only proprietary formats, used for distribution of videos on DVD, Blue-Ray or Pay Per View broadcasting systems were identified.
Therefore, a general solution for storing videos in encrypted video file format was developed. In our solution, only a simplified model was used, considering a distribution of encrypted videos to single client. The main focus was thus in valid encryption of the video stream, subject to an initialization via a single pair of asymmetric encryption keys. During the development of the encryption module, indications on good practice in video encryption from the proprietary formats were considered. Therefore, the encryption of each privacy video stream is executed in two stages. Primarily, the video stream is encrypted with a symmetric AES cipher in CBC block mode. The necessary initialization parameters and the secret key are then encrypted by the public key of the destination user. An asymmetric RSA encryption method is applied. In this format the secret information is secure to be distributed over open channels.
In the proposed solution, the encrypted video streams are embedded into one video container file, for easier distribution. Finally, in order to be able to reconstruct the original video recording from the encrypted and the obfuscated streams, a dedicated player was developed.
During our research, no open standard was identified for storage of encrypted video files. We therefore opted for demonstrating a possible secure video encryption based on publicly available libraries and open cryptographic standards. In our proposed solution, the encrypted video streams are distributed along with the main video stream (with obfuscated regions), embedded in one video container file. General available video players are not able to decipher the encrypted streams and thus show only the main obfuscated stream. Our dedicated player is able to reconstruct the original video from all streams, upon the introduction of valid distribution keys (i.e. corresponding private key of the destination user).
Figure 6: Privacy masking of certain image regions.
Task 4.5 Scenario Recognition (M7-M27)
This task investigated the available strategies for high level scenario recognition in surveillance video. where ULSTER identified potential scenarios of interest, proposed a hierarchical reference architecture which facilitates both atomic and high level scenario recognition, and developed an extensible framework for atomic (primitive) scenario recognition.
ULSTER has also evaluated and extended the list of scenarios of interest to consider hierarchical events which can incorporate outputs from single layer scenario recognition, object identification and tracking, and person identification modules. Jointly with this activity, appropriate algorithms for extension of atomic scenario recognition to hierarchical recognition have been identified. Moreover, ULSTER improved atomic scenario recognition and has participated in the identification of datasets for development and testing of SAVASA modules. For this purpose an ontology requirements specification document has been produced that presents the basis for hierarchical recognition of events. Liaise actions with HI-IBERIA have been undertaken in order to address scenario recognition extensions and ontology construction.
Figure 7: SAVASA scenario recognition framework by function.
The focus is on complex, hierarchical scenario recognition. A formal reasoning framework is introduced, which allows representing complex scenarios in terms of atomic scenarios along with spatial and temporal constraints, and then making uncertainty reasoning. RIMER, a software tool originally developed at the University of Ulster for uncertainty reasoning, is extended to implement this reasoning framework. The extended RIMER is then used for complex, hierarchical scenario recognition. The results of these tasks are reported in deliverable D4.52 Scenario Recognition – version 2.
DCU prepared a summary of suitable academic and research datasets for surveillance video for the project meeting held in Athens (11th and 12th of December 2012), and liaised with Ulster to discuss the choice of datasets for on-going development, testing and demonstration of semantic annotation components including person tracking, object detection and scenario recognition.
The main achievement is a standalone module for hierarchical scenario recognition which utilises outputs from person identification and object identification tasks. Both atomic level and hierarchical level semantic annotations are generated within this module, and outputted in agreed format (JSON notation) according to the structure defined within the SAVASA ontology. The framework has been tested through replication of previous results, as defined within deliverable D7.24 Final Platform Test Report. Additionally, the alignment of module outputs with the SAVASA ontology has been validated through collaborative testing with HI-IBERIA.
Task 4.6 Video Ontology Construction (M4-M18)
The construction of an ontology system for SAVASA and the coordination of modules integration planning and definition on integration and improvements roadmap were developed within this task, led by HI-IBERIA. The performed activities include:
• Identification of the baseline ontologies upon which to base the SAVASA ontology, which will extend these to optimize resources and maximise interoperability:
o A series of ontologies covering topics relevant for SAVASA (e.g. video scenario recognition, video formats) were identified in the SoA.
o Analysis with respect to initial requirements for the platform expressed in the project DoW and refined by use cases in WP2.
o The scope was refined by balancing effort: the resources for modelling MPEG-7 video artefacts were reduced while those for modelling use case specific features (such as scenarios useful for the end-users) were increased.
• Elicitation of the first end-user requirements for the ontology (RENFE ‘events of interest’ list was used as an input for the initial version of the ontology).
• Implementation of the ontology using well known tools (Protégé) and standards (W3C’s OWL) with preliminary coverage, reasoning and performance testing of the ontology.
• Continuation of the testing phase of the implemented ontology using well known tools (Protégé) and standards (W3C’s OWL).
Figure 8: Integration of the ontology in the SAVASA information flow.
ULSTER proposed a hierarchical ontology-based representation scheme for surveillance scenarios which include composite events as well as atomic events. ULSTER also explored in depth various approaches to recognising composite events based on their descriptions in terms of ontology, including Bayesian networks, context free grammar, and rule-based systems. It was decided that a rule-based evidential reasoning system is appropriate for SAVASA and, subsequently, this rule-based approach has been selected.
A SAVASA Video Ontology was produced in M18 and delivered as D4.61. After this was produced, it was detected that ongoing work in task 4.5 and task 4.7 could benefit from extensions to the ontology so that a new version of the ontology and the document were produced as the new D4.62 which was introduced in the official description of work via an amendment. This document contains extra explanations about the scenario recognition ontologies and also about the API available to input and query semantic elements into the triplestore of the system. This was integrated into the Final SAVASA prototype in WP7.
Task 4.7 Semantic Video Annotation (M4-M27)
This task, led by DCU, focuses on the annotation of the captured video elements using semantics such as the defined in task 4.6 Video Ontology so that elements can be stored and searched for using a consistent set of identifiers. Thus, the work is very much related with that of the video analysis tasks 4.1-4.3 the semantics and ontology study 4.6 and also with activities related to the search engine in WP3.
The principal work on this task focussed on participation in the TRECVid evaluation workshop, an international collaborative benchmarking activity co-ordinated by the US National Institute for Standards. DCU, ULSTER and VICOMTECH prepared a joint publication to be submitted applying our work to the challenging dataset of surveillance video from an airport. A number of algorithms for semantic annotation were applied and a platform built to enable interactive user evaluation where users were asked to find video segments showing particular events. As part of this process DCU, ULSTER and VICOMTECH, together with IKUSI and HI-IBERIA, visited RENFE to conduct user-based evaluations using the output, and submitted the results for the workshop. We were selected to give a talk at the workshop and ULSTER and DCU participated in the discussions and poster session. Further discussion of the outcomes is reported in D7.21.
HI-IBERIA provided a baseline ontology which shall be used for the first working versions of the video annotation. HI-IBERIA participated in the discussions leading to an adequate design of the video annotation, useful for video archive search. HI-IBERIA also provided guidelines and feedback regarding the organization of integration between the Semantic Annotation module and Video Ontology, in line with the activities of task 4.6.
DCU prepared deliverable D4.72 Semantic Annotation Components – version 2, an update of D4.71 that described work on human action recognition, the updates of the semantic annotation storage processes and the contribution to TRECVid2013 in the interactive surveillance event detection task.
The implementation and application of algorithms for semantic video annotation on the challenging TRECVid dataset was completed. The semantic annotations resulting from object identification and scenario recognition were stored as metadata and utilised within a prototype search interface that allowed users to search for specific events. As a result of the publication submitted to TRECVid, we were selected to give a talk which Suzanne Little (DCU) presented at the TRECVid 2012 workshop in Gaithersburg, US.
Updates to the video analysis module for human action recognition, prototypes for crowd behaviour and indexing of tracked persons by colour, evaluation of object recognition methods were produced. Module for storage of semantic annotations and metadata was updated, implemented, deployed on the cloud, integrated with the global control platform and tested.
WP7 – Integration and testing
Task 7.1 Integration and test of Platform Components (M3-M27)
According to the integration plan, the Consortium has integrated the final SAVASA platform in a series of prototypes, from the initial Alpha prototype, to the Beta 1 and Beta 2 prototypes, which included partial results of the research and development tasks, and eventually the final prototype. This last iteration integrates the final version of all the modules that are part of the SAVASA architecture and can be considered as a complete system that provides the functionalities defined during the entire project lifecycle. The final progress achieved for each module has been the following:
• The global control’s application (SMA) incorporates a video request mechanism between Consumers and Producers. In this application one “Local Nodes” section has been implemented in order to manage and group all the nodes. Integration with Search Engine, Video Ontology descriptor and Cloud system has finished.
• Semantic Video Analysis and Annotation tools as it provides a coherent description of the Video Surveillance concepts and relationships that are detected in the Videos and provides input for Videos labelling. The search engine and metadata store were deployed for testing through the TRECVid2013 interactive surveillance event detection task.
• Integration of the video ontology incorporates a new set of complex concepts and relationships in order to demonstrate the reasoning capabilities enables by the use of semantic search framework (new dangerous events definition using complex concepts relationship and integration with the Semantics Video Analysis and Global Control Platform module).
• The operation of SAVASA Cloud has been tested through a number of selected validation tests. All the tests have been executed from a running Essex OpenStack environment.
• VICOMTECH contributed to the local processing node with final implementation of real-time person detection, watermarking and privacy masking modules. The person detection accounts for perspective correction of videos taken by static cameras. A manual calibration tool necessary for the configuration of static cameras has also been integrated. Watermarking, for digitally signing entire videos individual frames, and privacy masking of detected regions of interest, were enhanced by cryptographic algorithms for enhanced security of produced videos. A standalone video player was developed for watermark validation and reconstruction of encrypted, privacy masked videos.
The following paragraphs summarise the prototypes built during the development of the project.
D7.11 Alpha prototype platform, submitted in month 9, includes a definition of the Integration and Testing of the SAVASA project. The alpha platform prototype is charged with the definition of the first of four formal versions. The goal of this prototype was to show existing technologies and background knowledge of the consortium in a “rough and ready” platform to be used to scope and define the goals of end user requirements and verify platform designs. This first prototype shows several independent modules: search engine & metadata, video ontology, semantic video analysis and local processing node.
D7.13 Beta 2 prototype platform has been closed. The aim of this deliverable is to present the work done in task 7.1 in order to develop Beta 2 or the third version of the SAVASA prototype. The document summarizes the progress made since the previous iteration (Beta1) for each of the modules forming the prototype. The deliverable was submitted in M21, complying with the deadline established in the DoW.
D7.14 Final prototype platform has been closed. The document shows the final version of the SAVASA platform. The Final Prototype includes and integrates all final modules defined in D2.14 System design specification-version 2, hence, the system is functionally complete and fulfils the requirements and scope posed in WP2
Task 7.2 Platform Evaluation (M13-M30)
The final SAVASA platform has been tested and validated. The main results of conclusion obtained for each module have been the following (extracted from D7.24):
• The tests carried out demonstrated that the final SMA meets the requirements defined at the beginning of the project. All the SAVASA services and functionalities can be accessed and controlled from this application. In addition, the same is compatible with the interface, security and usability constraints defined in WP2. Different use cases carried out with the platform (involving the creation/remove/edit of users and local nodes and execution of services like monitorization, search and management of requests) in order to evaluate the integrity of the functionalities have been also tested. It has been demonstrated that SAVASA meets the use cases defined at the beginning of the project. Users can interact with the platform following the procedures established and can access all the expected services and functionalities. This includes from the global management of all user accounts and local nodes, to the processes of searching and obtaining images. It is important to note that some additional features have been added to the initial services derived from the petitions and needs of the end users which have enriched the first scenarios planned in the project.
• The final version of the search engine is capable of querying video metadata and any available semantic annotation generated by the video semantic analysis or from the inference engine based on the ontology. The integration of this module with the metadata base has been widely tested through participation in TRECVid2013.
• The final ontologies allow defining the events agreed with the end users and that have been already analyzed by the semantic modules. The REST APIs constitute the entry point for other modules to interact with the ontology.
• The hierarchical scenario recognition has been evaluated with ‘opposing flow’, ‘people meet’ and ‘sterile zone entry’ recognition. Belief-Rule based Inference Method using Evidential Reasoning (RIMER) has been applied for all the cases. Then integration testing has been carried out, including person detection and sequence-based action recognition (reduction of the DCR from 1.21 to 1.07 for ObjectPut and 1.22 to 1 for Pointing events), JSON output testing (all scenario recognition module outputs are syntactically correct) and Consistency checking (evaluation of Low-level feature extraction, Atomic level person description, Atomic level action recognition and Hierarchical scenario recognition). Finally, individual scenario recognition components timings have been evaluated through Matlab’s ‘tic’ and ‘toc‘ functions. In parallel, Abnormal Crowd Behaviour and Colour-based indexing of people detections have been worked, as new capabilities that could be integrated in the future in the SAVASA solution.
• The final SAVASA cloud has been implemented based on OpenStack. The final configuration of SAVASA OpenStack Cloud is composed of one Cloud Controller and one Compute Node for Virtual Machines (VMs) deployment and one Storage Node Controller for storage support. The extension of this platform is easily achievable in order to support various loads and applications.
• The local processing node tests of the final platform evaluated the transcoding, person detection, video watermarking and privacy masking modules. Overall, the implementation correctness, data exchange formats and correct workflow of the interconnected modules has been tested. A limitation on the access to the proprietary NVR was detected which did not allow us to automatically access the recorded videos. This initial step is therefore currently simulated by manual copy of videos to specified directory. Implementation limitations on the use of the privacy masking tool have been also detected, nevertheless, the proof of concept was demonstrated successfully. Finally, recommendations and suggestion on the use of the watermarking tools have been given.
DCU participated in the TRECVid2013 semantic indexing task to provide some evaluation of a standard approach to object detection in video using frame-based feature descriptors. The large and highly-variable dataset is expected to be challenging but enabled good evaluation of the performance and scalability of classical analysis tools for general object detection. The outcomes of these evaluations will be reported in the semantic analysis test report deliverable.
DCU, ULSTER and VICOMTECH have been preparing to participate in the interactive surveillance event detection task for TRECVid2013. The aim is to utilise the integrated search engine, semantic metadata store and global control platform modules deployed on one or more virtual machines for the user interactive sessions.
Each of the Alpha prototype components has been tested as a black box, verifying the relationship between inputs and outputs of each component (Unit Testing). D7.21 Alpha prototype test report was submitted in month 12.
D7.22 Beta 1 prototype test report, submitted in M18, summarizes the tests performed on the second iteration of the SAVASA platform (Beta1 prototype). The aim of this document is validation of the different modules that are part of the SAVASA system scope before starting with the implementation of the final prototype.
D7.23 Beta 2 prototype report summarizes the principal conclusions reached after finishing with the testing and validation process of the third platform iteration (Beta 2 platform).
Finally, D7.24 Final prototype test report presents the tests carried out with the SAVASA Final Prototype. This version has served as the basis for final end user testing and for completing project dissemination and exploitation tasks.
Impact of the final platform
At present, the search, identification and delivery of images used as evidence in court is based on manual processes (video verification performed by humans, physical displacement of personnel to retrieval images, hand delivery of the videos, etc.) that hamper the work of the CCTV operators (Producers) and delay the response time of the LEAs (Consumers).
To optimize the entire process, SAVASA, offers to authorized users the possibility of using a semantic video search platform which implements different technologies like video analytics, computer vision and video ontologies in order to facilitate the detection and access of the required images. The solution is based on cloud architecture such that its services can be provided remotely. Specifically, SAVASA improves the following aspects of the current video surveillance solutions:
• All the SAVASA services and functionalities can be accessed and controlled from the SMA web application. The management of the users' accounts, local nodes or groups can be easily performed from this application. The specific environments created for Consumers and Producers allow their access to video search and retrieval functionalities, thus offering a VSaaS solution (Video Surveillance as a Service) innovative in its field.
• The search engine and metadata database module provides access to the collected metadata and semantic annotations captured or generated by other modules.
• Semantic Video Analysis and Annotation tools as it provides a coherent description of the Video Surveillance concepts and relationships that are detected in the Videos and provides input for Videos labelling.
• Implementation of video ontology incorporates a new set of complex concepts and relationships in order to demonstrate the reasoning capabilities enables by the use of semantic search framework (new dangerous events definition using complex concepts relationship and integration with the Semantics Video Analysis and Global Control Platform module).
• RESTful web services has been implemented in the all the integrations where it had sense. RESTful, in addition to define an architecture that supports any programming language, are very light, and the answers contain exactly the needed information. They also offer a good caching infrastructure via HTTP (useful to optimize the download and rendering of images, for example).
• The transferring information in the system is completely secure against disruptions by third parties.
Therefore, SAVASA's final platform may have a significant positive impact on the current video custody chain, in terms of efficiency and costs.
Considering the results associated with the final platform analysis described in the previous section, the operation of Consumers and Producers could be optimized, making their work performance more effective.
On the one hand, Producers will benefit of reduction in the need for physical displacements and/or time savings in image retrieval (SAVASA offers them mechanism to access the CCTV records remotely). Manual recovery implies the need for personnel to physically travel to another location to perform the image recovery, either to buildings, facilities or vehicles (for Railway scenario). Reduction in the need for physical displacement and/or time savings in image retrieval will have a direct impact on Producers due to savings. It should be remembered that the amount of information and data that may be collected through their video surveillance is enormous and unstructured. It follows that management of such amount of information needs the advances proposed by the SAVASA project in the areas of video analytics, privacy protection and unified video archived search.
On the other hand, Consumers need less time to search the images required by the Judicial staff. Searching remotely in different CCTV systems using semantic algorithms reduces the time needed to find the videos that facilitate the operative work of the LEAs. Communication and cooperation between Producers and Consumers will also increase through the official request system implemented in SAVASA for both kinds of users.
The final beneficiaries of these improvements will be the citizens as LEAs will resolve more efficiently and rapidly ongoing investigations. Safety feeling will increase for all the transport systems users and a reduction of the criminal activities associated with the deployment environments is expected (thefts, aggressions, vandalism, etc.). Many cases currently closed for lack of evidences may hereafter be solved.
In order to make sure that citizens right to privacy are respected, SAVASA has implemented some personal data protection mechanisms agreed with the Ethical Advisory Group. These mechanisms ensure that Producers and Consumers will always interact with the platform in a way that is compliant with the legal requirements and ethical procedures promoted by the European Commission.
Main Dissemination activities
According to the procedures defined in WP8, an overall strategy for the dissemination of the project results has been prepared and implemented by the partners. The whole dissemination event chain has been covered: project presentation to relevant European entities (Europol), the participation in workshops where intermediate results could be disseminated (TRECVid competitions), and in close to market orientated events (SICUR) where the final platform was presented to the industry.
A high number of scientific publication have been done, 17 papers were published. SAVASA has been presented in 4 continents (Europe, America, Asia and Oceania), in high level conferences like the TRECVid competition, organized by the National Institute of Standards and Technology.
All these publications have been uploaded to the project website. The website has been regularly updated with the news generated by partners. All public deliverables are available for the visitors, together with the newsletters. Apart from the website, the necessary dissemination support material has been created: brochures, posters, final video, etc. The acceptance of the dissemination material delivered to the public has been very high. An important number of entities have contacted the project consortium requesting more about the project.
Contact and collaboration with other FP7 projects of the security field (among others ADVISE, ADDPRIV or VIDEOSENSE) has been established. This close collaboration has enriched SAVASA with the knowledge and feedback coming from other projects. In the case of the ADVISE project, SAVASA participated together with them in a dissemination event (Keynote and full paper at 5th International Conference on Imaging for Crime Detection and Prevention) organized by the Kingston University in December 2013.
The spread of SAVASA will not end with the project. All the partners are involved in the future exploitation of the results so they will continue pushing in this sense. In fact, one conference has been already scheduled, SAVASA will be present within the Workshop on Video Soft Sensing that will take place during the 2-5 December 2014, in Belfast; and will share its SAVASA will share its results with other R&D projects like SLANDAIL or P-REACT.
Project results have a positive impact on the exploitation plans of all the partners that make up the Consortium. Although IKUSI is the major industrial partner of the Consortium the rest of the partners are also interested in commercializing the results generated, that should be exploited according to guidelines defined in D8.12.
Firstly, is important to remember that the objective of SAVASA was to obtain a final pilot that will validate the functionalities and scope defined during the project, but not obtaining a final product that could be directly launched in the market. In order to have a fully commercial product an industrialization and commercialization plan should be performed as is described in the SAVASA Exploitation Plan.
This final prototype includes all the features defined at the beginning of the project in order to offer the identified SAVASA services to the end users. The modules or components that are part of the solution are already defined in D2.14 “System design specification-version 2”. More specific information about this final iteration can be found in D7.14 “Final prototype platform” and D7.24 “Final prototype test report”.
The final SAVASA system has been compared with current video security commercial solutions. After analysing the characteristics of the competence can be concluded that none of products covers the set of functionalities that could be offered by a final SAVASA product. Although some solutions (BOSCH, Scati, etc.) offer the possibility of trigger alarms based on video image analysis (monitoring perimeters, people counting, strange movement, etc.), none of them has been designed as a forensic video search tool that could improve the specific requests coming from Law Enforcement Agencies. Unlike SAVASA, current products cannot be extended with relevant search parameters specified by users. Also, none of the alternatives offer the possibility of managing official requests interchange between the Consumer and Producers during the regular video custody chain. Other strengths highlighted by the competition, such as the modular structure of BOSCH, are completely covered by SAVASA. Moreover, unlike the current offer, SAVASA used exclusively not proprietary video formats, which facilitates interoperability between all stakeholders involved in the search and delivery of legal evidences.
The use of a management system based on web applications and management of information in the cloud are also some innovations exclusively addressed by our solution. Commercial solutions studied are preferably based on desk applications, and videos' processing and storage is exclusively done locally, which makes impossible to offer a system based on VSaaS or Video Surveillance as a Service, something widely asked by potential clients.
Another important advantage of the platform is the one related with the cost reduction. Taking into account that the main objective of SAVASA is to create a semantic video archive search platform, the major cost reductions will be related with the optimization of current video custody chain. The manual recovery implies the need for personnel to physically travel to another location to perform the image recovery, either to stations, buildings, workshops, etc. This is a major expense for transport entities in view of the large number of requests that they can manage over time.
The benefits of the system will be shown to the potential customers. One of the activities planned to exploit and internationalize the results is the demonstration of the product in the most important international fairs of the security sector. As usual, for some consortium partners, the presence in major trade shows and events which bring together leading industry organizations, including potential customers, is essential and therefore has become a core activity in their marketing plans. The participation of SAVASA at SICUR International Security Safety & Fire Exhibition (25th-28th, February, Madrid), is a clear example of that. This event brings together the entire safety and security industry, making up a comprehensive showcase of new developments relating to protection and prevention of people, objects and sites. SICUR is an excellent meeting-point where all representatives of this market come together.
The address of the project's website: http://www.savasa.eu/ . All visitors can contact the Consortium through email@example.com mail.
List of Websites:
Grant agreement ID: 285621
1 December 2011
31 May 2014
€ 4 059 831,96
€ 3 166 266,58
ANGEL IGLESIAS S.A. - IKUSI
Deliverables not available
Publications not available
Grant agreement ID: 285621
1 December 2011
31 May 2014
€ 4 059 831,96
€ 3 166 266,58
ANGEL IGLESIAS S.A. - IKUSI
Grant agreement ID: 285621
1 December 2011
31 May 2014
€ 4 059 831,96
€ 3 166 266,58
ANGEL IGLESIAS S.A. - IKUSI