CORDIS - Risultati della ricerca dell’UE
CORDIS

Advanced Video Surveillance archives search Engine for security applications

Final Report Summary - ADVISE (Advanced Video Surveillance archives search Engine for security applications)

Executive Summary:
In contemporary societies, video surveillance forms an integral part of incident investigation. Trails of delinquent activities may be left behind and captured/recorded by various surveillance systems. In many cases the investigation of a crime incident involves collecting and analysing video surveillance data from the crime scene.
In order for such an investigation to be successful, it is important to effectively uncover the trails and the association between suspects that can lead to the identification of accurate incriminating evidence. In this case, the investigators should be in a position of having access to a plethora of video surveillance information and the appropriate tools to perform sophisticated searches on the video archives, so as to pinpoint those video frames that reveal the evidence of criminal activities.
ADVISE aimed to design and develop an extensible framework that after negotiating all relevant legal, ethical and privacy constraints, was able to help the law enforcement authorities to fight against criminal activities, via efficient evidence mining into multiple, heterogeneous video archives. More in particular, the ADVISE aim was to analyse and geo-register surveillance video archives of different agencies, extract statistical patterns of activity and search (context-based and content-based) for specific events, people and objects through ontologies and semantic representations.
In a context where surveillance systems are continuously growing in scale, heterogeneity and capabilities, two major obstacles have to be overcome. On one hand, the variety of technical components of surveillance systems, producing video repositories with different compression formats, indexing systems, data storage formats, sources, has to be addressed. On the other hand, the legal, ethical and privacy rules that govern surveillance and the produced content have to be taken into account.
To address these two major issues, the ADVISE system is composed by two major components: the first one performing the semantically enriched, event based video analysis which offers efficient search capabilities of video archives and sophisticated result visualization, and the second one enforcing the legal, ethical and privacy constraints that apply to the exchange and processing of surveillance data.
Moreover, in order to support interoperability, the exchanged content and the associated metadata is transformed into a common format, while a dedicated ADVISE Engine has been developed to efficiently deal with each surveillance and collaborating authority’s technical and legal/ethical/privacy specificities.

The partners involved in the ADVISE project were the following:

ENGINEERING - INGEGNERIA INFORMATICA SPA (Project Coordinator) - Italy
NEUROPUBLIC A.E. PLIROFORIKIS & EPIKOINONION - Greece
CENTRE FOR RESEARCH AND TECHNOLOGY HELLAS - Greece
QUEEN MARY AND WESTFIELD COLLEGE, UNIVERSITY OF LONDON - United Kingdom
SINGULARLOGIC ANONYMOS ETAIRIA PLIROFORIAKON SYSTIMATON & EFARMOGON PLIROFORIKIS - Greece
VRIJE UNIVERSITEIT BRUSSEL - Belgium
INGENIERA DE SISTEMAS PARA LA DEFENSA DE ESPANA SA-ISDEFE* - Spain
ALMAVIVA - THE ITALIAN INNOVATION COMPANY SPA - Italy
INNOVATION ENGINEERING SRL - Italy
AYUNTAMIENTO DE MADRID - Spain

Project Coordinator: Mrs. Carmela Occhipinti
Email: carmela.occhipinti@eng.it

*Partner INGENIERA DE SISTEMAS PARA LA DEFENSA DE ESPANA SA-ISDEFE terminated its participation to the project on April 30, 2014.
Project Context and Objectives:
The ADVISE project aimed to realize a decentralized Video Archives Searching solution that meets the challenges created by the ever increasing growth of surveillance systems in scale, heterogeneity and capabilities. The main challenges, in both the technical and legal/ethical side, that ADVISE aimed to address are listed here after:
1. On the technical side, while video analysis and pattern recognition technologies are at the core of “intelligent” or “smart” surveillance, effective search of surveillance video archives requires research into searchable meta-data and semantic representation of surveillance related video features, data models for indexing and correlating diverse types of meta-data, and architectures for integrating technologies into large scale systems. A further challenge involved the incorporation of the Geographical Information System (GIS) that enables geo-registration and multi-view tracking across all the surveillance systems of peer agencies, i.e. track individuals from one camera to another, regardless of the different systems to which they belong. ADVISE aimed to address these challenges and develop a framework for video archive searching in heterogeneous repositories, enabling event identification and tracking in a multi-camera environment.
2. On the legal and ethical side, a methodology to ensure the compatibility of surveillance video analysis with legal and ethical rules were not yet defined nor precisely described. The inter-dependencies between legal arrangements, ethical rules and specifications of systems used for surveillance video search and analysis had to be thoroughly explored especially in Europe, since it has one of the strictest legislations for the protection of personal data. The ADVISE system aimed to incorporate Legal, Ethical, Privacy (LEP) rules in the design of the system, producing a LEP-compatible system. Moreover, it aimed to produce a clear methodology for the development of Privacy-by-Design systems.

Project Results:
As for its initial aim, ADVISE has designed and developed an extensible framework that after negotiating all relevant legal, ethical and privacy constraints, is able to help law enforcement authorities fight against crime and terrorism, through efficient evidence mining into multiple, heterogeneous video archives.
ADVISE analyses and geo-registers surveillance video archives from different repositories, extracts statistical patterns of activity and searches (either context or content-based) for specific events, people and objects through ontologies and semantic representations. The ADVISE system enables interoperability beyond the boundaries defined by different compression formats, indexing systems, data storage formats and access systems, offering valuable insights and help during investigations of law enforcement authorities.
The project partners have carefully designed and implemented a number of activities to achieve the following main results and generate the related foreground:
1. Modular architecture platform: An IT platform is created that provides the flexibility to update and adapt to disparate applications. The platform has the ability to replace modules or to add novel and more efficient modules. Furthermore, it can adjust to both updates of the data protection legislation (national and European) and different needs of the potential users.
2. Legal and ethical compliance: In the context of the Privacy by Design methodology, a framework for the Privacy Impact Assessment (PIA) is developed as a means of achieving the satisfactory level of the observance of ethics, privacy and data protection, subject to necessary adaptations reflecting to the nature of the project. In the words of Cavoukian, the author of the PbD concept, a PIA “can be an excellent entry point for applying the principles of Privacy by Design”. As consequence, the privacy protection is ensured by built-in mechanisms, and the semantic representation is enhanced with privacy related notions. Moreover, intrinsic and also extrinsic properties are covered.
3. Video Search: An extensible framework is implemented with video analysis and recognition algorithms, concentrating on objects, vehicles, humans and events in a spatiotemporal fashion. Additionally, a surveillance ontology schema is developed for semantic representations of events and advanced reasoning capabilities. Video metadata are indexed and categorized for improved retrieval. Finally, object/person localization is provided in combination with GIS (Geographic Information System), enabling association of segments from different sources that captured the event of interest. Video summarization, by converting the video to a stream of metadata, reflecting a 24/7 description of the footage is implemented to provide the users with a condensed representation of the video content.
4. Content Mediation: Interoperability with external systems is ensured by the framework and new heterogeneous video archives can be easily added. Video format and metadata compatibility issues are dealt with. Metadata is normalized into a common format, regardless of the origin of the different archives. The privacy, legal and ethical issues are handled per authority and per archive. Moreover, access privileges are adjusted to the user and the sensitivity of the respective material.
5. Graphical User Interface: An archive video search engine is required to have an intuitive user interface that will allow the investigators to efficiently use it. The GUI is needed to provide a dedicated interface and tools to semantically define an event through a set of event attributes from the semantic repository. Visual Analytics provided by the GUI present in a smart and analytical way the results, through user-friendly reporting systems, optimizing the report to support better understanding by the user.

Potential Impact:
ADVISE aimed to have a high technical, social and economic impact.
Concerning the technical impact, ADVISE applies the state of the art technology, advancing event-based video analysis by integrating multiple technologies and consolidating semantic representation. Furthermore, interoperability is promoted, making recommendations for interoperable standards, common practices and procedures, and taking properly into account the legal, ethical and democratic challenges of the use of video surveillance.
Moreover, from the societal point of view, ADVISE is able to increase the following:
- Social acceptance of surveillance systems based on ethical grounds: ADVISE is actively tackling Legal, Ethical and Privacy issues by design, while targeting to video archives and not real-time surveillance systems.
- Social acceptance on the grounds of effectiveness: the ADVISE system aspires to accommodate video archive search with increased capabilities that will be disseminated at a European level. This is expected to alter the conception of the public on smart surveillance system operation under realistic conditions.
- Feeling of security: the ADVISE project results will ease the work of law enforcement authorities in their fight against crime and terrorism, making citizens feel more secure.
- Crime detection: through the establishment of a federated framework, ADVISE offers a single query point to be used in incidents’ investigations and the capability of parallel, dedicated analysis of distributed, heterogeneous video archives, thus increasing the efficacy of crime solving.
Finally, from the economic stand point, ADVISE’s main impact is the following:
- Economic impact in criminal fighting: ADVISE is expected to boost the effectiveness of fighting against crime, without introducing new costs as it utilises recordings from existing legacy systems, while lowering the investigation cost, offering a significant faster and more efficient way for searching video archives.
- European Industry competitiveness in surveillance systems post-processing: ADVISE may have a significant competitive advantage in providing high-quality software for post-processing of the recorded video archives.
- European Industry competitiveness in multimedia search engines: ADVISE has the potential of greatly influencing the way video search engines behave, by defining a strong technical progress beyond the state of the art, in multimedia searching.

ADVISE dissemination activity
A summary of the most relevant dissemination activities performed during the whole project lifetime is presented below:
• 23 conferences have been participated by ADVISE representatives, including events organized by the sister projects;
• 5 papers have been produced;
• 2 ADVISE user-workshops have been organized in order to disseminate the ADVISE project and increase the awareness around its results, by promoting the opportunities represented by ADVISE for the improvement of European citizens’ security in a world of increasing vandalism and terrorist activities;
• 2 ADVISE liaison-workshops have been organized in order to stimulate a multi-sided debate with policy makers, research community and sister projects about the desirable shape of surveillance solutions deployed in European society, as well as to incite a discussion on the acceptable extent of the invasiveness of contemporary surveillance solutions;
• 4 newsletters have been published and disseminated in order to disseminate (i) the aims, objectives and main achievements aimed by ADVISE (first delivery), (ii) the 2nd ADVISE workshop organization and the ADVISE liaison workshop organization (second delivery), (iii) the results of both workshops (third delivery) while (iv) the final results of the 2nd ADVISE Workshop and the Roundtable at CPDP 2015 have been described in the final newsletter issue;
• 3 press releases have been realized and disseminated, regarding the (i) first ADVISE workshop results (first delivery), (ii) the developed pilots (second delivery) and the (iii) the approach followed in ADVISE for data protection , privacy and ethical issues in the final press release delivery.
In addition, the ADVISE partners have disseminated the project internally to their organisations and using both social channels and traditional means made available by the companies themselves, by adding reference to the ADVISE project into their own facilities (e.g. the Collaborative Room at the ENG’s premises at Pont S. Martin, Italy) or into their own institutional websites. Moreover, the ADVISE project has been disseminated with a short public description taken from the ADVISE website also in the private LinkedIn accounts of several partners.

Specific tools and communication material were used for the dissemination activity, according to different communication needs and to various event typologies. The developed communication material published on the project website at http://www.advise-project.eu/content/public-results is the following:
1 ADVISE public web site;
2 ADVISE logo;
3 ADVISE brochure to be printed and distributed in events/meetings;
4 ADVISE flyer to be sent by email or electronic means;
5 Official project presentation, describing project objectives and major results;
6 ADVISE posters to be customised and shown for specific conferences or events;
7 ADVISE system demo video;
8 ADVISE system visual tutorial.

1. Public website
All the information related to the ADVISE project is contained in the official ADVISE Web Site, that is available at the URL http://www.advise-project.eu. It was developed by the consortium with the following aims:
• Ensuring adequate presence of the ADVISE project on the web;
• Informing the general public about the aim and objectives of the ADVISE project;
• Disseminating project’s activities, public results and initiatives related to the security field;
• Creating synergies with other similar initiatives to attracting and concretely involving the relevant actors within the framework of video surveillance security and privacy sectors;
• Being the main online tool to communicate, transfer knowledge and exchange information facilitating the collaboration between the potential users and the further extensions and adoption of the project outcomes.
It includes the following diverse information to visitors:
• General information about ADVISE consortium as well as the activities carried out during the development of the project itself;
• Description of the benefits expected from the project research activities;
• Updated news and events about security, video archive search engine, video analysis, surveillance ontology, ethics and privacy, surveillance-footage archive systems;
• Link with other projects/initiatives in the video-surveillance domain;
• Logo and abstract of sister projects i.e. projects with which liaison activities have been undertaken.
The WWW server consists of a public area - accessible by all visitors - mainly with the aim of disseminating project results, as well as a password protected part, accessible only by the ADVISE partners, the EU officer and the Advisory Board members.
The project website has been constantly kept updated with content related to the ADVISE project or to material coming from other sister projects as well as with information relevant to the ADVISE sector.

2. Logo
The logo includes the name of the project, as well as a graphic realization of the concept of video surveillance (a security video-camera) related to the idea of localization into a certain area.

3. Brochure
In order to ensure the creation of paper dissemination material, a brochure was designed and distributed during the project lifetime. The ADVISE brochure had the aim to communicate clearly to public (both citizens and target companies) ADVISE’s major purposes.
The brochure contains a short description of ADVISE project, its main aims and goals and who the beneficiaries of the project are. The brochure includes also the challenges have to be approached in the project and the expected benefits of the project.

4. Flyer
A flyer template was designed with the same purpose of the brochure: it contains the highlights of the ADVISE project and its main objectives and expected results. It was developed in a vertical format to allow partners to distribute it by electronic means: email, download form websites, etc.

5. Official project presentation
In order to allow all ADVISE partners to present the ADVISE project using the same promoting material, a Power Point presentation was created. The ADVISE presentation is composed by 10 slides, introducing the project concept, partners involved, the challenges, and the expected results and/or benefits.

6. Posters
At the project beginning, a poster was designed with the same purpose of the brochure and the flyer: it contains the highlight of the ADVISE project aims and its main goals. It has been developed in a vertical format to allow partners to expose it during conferences or other events.
Moreover, specific posters were designed for the ADVISE liaison Workshop and the 2nd ADVISE workshop.

7. System demo video
In order to provide the partners with a mean for presenting the ADVISE tool working, a demo video explaining the main functionalities of the Spanish pilot was realised. The main objective of the video was to allow all ADVISE partners to present the ADVISE tool, and the Spanish pilot in particular, using the same promotional material.
The ADVISE demo video duration is about 10 minutes and it provides an overview of the user profiles and the features available in the tool, from the possibility to create an investigation and the usage of the results provided by the system. In the next Figure a screenshot of the video is presented.

8. System visual tutorial
In order to provide the general public with a mean for understanding the ADVISE tool and its main features, the ADVISE video tutorial explaining the main functionalities of the ADVISE pilot web portal (containing the Spanish and the Italian pilots) has been published in the public area of the ADVISE website.

ADVISE exploitation activity
In the first phase of the project, a preliminary exploitation plan was defined, including the roadmap of the ADVISE consortium for reaching a solid and realistic exploitation plan. The consortium realised the importance of the exploitation activities, for each individual partner and as a Consortium. For that reason, the respective exploitation actions were initiated early enough with the exploitation strategy consisting of the following steps:
1. Exploitation Strategy and Roadmap;
2. Exploitation framework including the IPR management and the identification of exploitable assets and their exploitation potential;
3. Market and business model analysis for the ADVISE system;
4. Individual exploitation plans;
5. Consortium exploitation plans, including sales strategy for the ADVISE system and a plan for the future sustainability and evolution of the ADVISE system.

During the project lifecycle, the preliminary version of the exploitation plan was enhanced with the identification of all the exploitable assets, updated partner exploitation opportunities (individual exploitation plans) and the joint exploitation plan of the ADVISE project.
At the end of the ADVISE project lifecycle, a fully tested and verified pre-commercial development has become available. After the end of the project, and based on the experience of the ADVISE testbeds, the involved partners will continue, with their own resources, towards the commercialisation of the ADVISE platform solution or sub-components.

List of Websites:

ADVISE project is available at the URL http://www.advise-project.eu.
Contact person: Mrs. Carmela Occhipinti (Project Coordinator)
Email: carmela.occhipinti@eng.it