Final Report Summary - SMARTPREVENT (Smart Video-Surveillance System to Detect and Prevent Local Crimes in Urban Areas)
Executive Summary:
The increase in urban population lead to an increase in crime. This is aggravated by illegal immigration, criminal striping, and degradation of zones. The increase of criminal activity is manifested in an increased frequency of small crimes like graffiti, theft, robbery, destruction of street furniture. This has had a big impact for local governments, citizens and businesses.
Local government agencies are trying to solve these problems. As was shown in New York, a reduction in petty crimes increases the sense of public order which produces further reduction in crime in general. These organizations often treat the problem with classical solutions such as increase in police resources, reintegration of offenders, rehabilitation of degraded areas and modern solutions such as urban video surveillance. However, these new technologies are often underused and inadequate.
The SmartPrevent project aims to enhance detection and prevention of crimes in local urban areas by exploiting the full potential of video-surveillance systems. Project develops and provides four important benefits: i) Systematic characterization of usual petty crimes in an area under automatic surveillance; ii) automatic detection of the most usual and frequent criminal activities; iii) a set of automated tools capable of alerting the appropriate responders; and iv) early prevention of crimes by prediction and early detection of crimes.
Rather than providing new methodologies or tools, SmartPrevent is focused on: a) improving already-existing methodologies by means of a set of guidelines for the use of video-surveillance systems; and b) providing a set of tools capable to improve of the existing crime detection systems. Our solution was validated by deploying a realistic prototype scenario, which will actively involve the detection and prevention for crimes in an urban areas and management of these detections by final users.
Project Context and Objectives:
The SmartPrevent project focuses on the detection and prevention of frequent petty crimes that are of high impact to local communities and citizens in urban scenarios, considered to be a low-cost video-surveillance system oriented to end-users. For this, the consortium involves a balanced number of partners with complementary background from the technical aspect to ethical and legal domain as well as the perspective of end-users.
Current video-surveillance system in urban scenarios are very limited and only consist in a presentation of the visual information captured by the visual sensors network, not oriented to end-users, that limiting thus the capacity to help and prevent the criminal activity. Furthermore, the visual-surveillance systems usually do not have any automatic process in order to store the more relevant evidences to take in account in legal punitive process of the criminals. It means that most video monitoring applications require human observer during the whole surveillance process. A key limitation comes from the human capacity that cannot attentively follow more than 10 cameras or screen for large periods of time. However, the entire workload falls on the end user of the application. Firstly, this produces a high stress level on security staff and also needs a high number of security staff that must be proportional to the number of cameras. In addition, the inherent cost of human resource make current solution clearly unsustainable in the long run.
SmartPrevent’s challenge is focused to support security staff the responsibility of ensuring an ongoing video surveillance to enhance reliability system in urban scenarios.
Indeed, SmartPrevent proposes to address this challenge by:
• Studying the characteristics of frequent criminal activities in urban scenarios in realistic scenarios, including typical variations and unanticipated criminal situations.
• Developing a low-cost adaptive video-surveillance system in order to detect and prevent the criminal activities.
• Building a video-surveillance system as punitive tools in order to store the most relevant evidences of the detected criminal activities.
In a nutshell, the SmartPrevent objectives are:
Figure 1. SmartPrevent Scientific and Technical Objectives
These objectives are achieved by project as detailed below:
·Detection of crime on persons and local businesses in urban areas. SmartPrevent's video surveillance system will exploit the full potential of the visual analytics capabilities of these systems in order to detect crimes. To do this, we will improve current technologies in video processing for obtaining a two-level semantic understanding of the scene. This includes a low level characterization, where information obtained will concern a simple semantic meaning and a high level, that by means of this information plus other needed information is added focusing on the detection of human activity, is capable of producing a meaning closer to human understanding of the facts.
· Crime prevention over persons and local businesses in urban areas: Nowadays, systems that try to detect human activities are focused on detection of such activities when being carrying out; but this approach is not necessarily valid in all applications. In the case of video surveillance, early detection of crimes is crucial, because it is often more important to detect these criminal activities before that they occur, so as to allows for the prevention of crime and not just reacting against thereof.
· Video-surveillance system oriented for end-users: Video-surveillance systems there are two key factors for effective and efficient use, the human factor and the technological factor. In the technological factor, one would consider all the hardware and software that compose the system.
On the other hand, the human factor is concerned with the system users (security staff).
Currently, in this field, system errors are mainly attributed to human errors. For this reason, the proposed system must have as main goal the generation of a set of applications that minimize human error as much as possible by means of the automatization of the end user tasks.
· Video-surveillance system as effective and efficient punitive tool that protects people's privacy: The SmartPrevent system allows for automatic detection of crimes, and therefore it can make a selective storage of only the moments when they are performing these criminal acts. Thus, we get two clear benefits, firstly, we only stored video content that contain a crime, this allows for better storage space utilization, consequently, we improve the system ability to take punitive action against offenders. Secondly, the system allows the user to remove all content from people who are not performing criminal acts and directly preserves the privacy of them.
· Temporal and spatial adaptability: We must ensure the functionality of the video-surveillance system and the portability of the proposed system to other municipalities in different European countries. In addition, we also have to ensure that new criminal activities or vandalism acts that will appear can be detected or predicted.
· Low-cost video-surveillance system: CCTV system are not expensive but the cost increase adding all related operations and personnel involved. The SmartPrevent system takes all these costs into consideration. First, the visual sensor proposed are of low cost (both device, installation and operating), and easy to maintain, etc.. Secondly, the intelligent storage and alarms system are oriented for the end user (indicated in previous objectives) to minimize the storage and use of security personnel and therefore the costs that incurred.
SmartPrevent project context and activity are arranged in seven work packages. Overall, during the 26 months, the consortium has performed all the tasks and achieved most of the results as planned where all the WPs are involved. Mainly, WP4, 5 and 6 were the activities with higher workload because of the starting of pilot phase and development of SmartPrevent system. However, non-technical activities as well as ethical issues took a large number of efforts since the Madrid government decline the pilot implementation approval. At the end, pilot phase were carried out in Kfar Saba thanks the management of EVS. This new location was aligned with the end users requirements and former pilot location, Las Rozas de Madrid. Ethical issues review was carried out for the new location in order to comply with the national rules and European Guidelines for citizen privacy.
Figure 2. Diagram for SmartPrevent
In relation with the work package scope the progress and results achieved during the whole period are summarized here below:
Definition of scenarios, criminal actions and Pilot requirements.
The final description of end user requirements were completed during the first project period in collaboration with the end user and its Local Police body. This information was discussed and improved to release a final document for the design of the software architecture and the functionalities to be developed. Moreover, these user specifications were updated with the new location of pilot phase to ensure that are valid for different scenarios.
Finally, once several scenarios and use cases were studied, Graffiti and anti-social behaviour were selected as two petty crimes to be used in order to develop the system and validate the performance during a pilot phase in Kfar Saba.
Ethical review of the data and the study of ethical issues for Pilot phase.
As a core activity of the project, privacy rights and ethical issues were taking into consideration to develop any technical stage. Ethical issues relationship with video surveillance and, in particular, the new location were carried out. At this point, Israel, where the pilot is run, do not have the same rules and normative than Europe. Cameras installation and video surveillance project in public areas are not under control of government and official approval is mandatory. In any case, the project was based on European rules and project guidelines to cover the ethical and legal issues. The Ethical Review committee was monitoring the datasets captured and validate the accordance of the pilot within ethical guidelines.
Software system for event description, detection and recognition.
The envisioned approach involved the development of a two-layer software system. The low-level scene understanding layer is in charge of gathering images from the CCTV devices for transforming it into semantic descriptors. These descriptors are the baseline data for the high-level layer, in charge of the activity detection and recognition at a topmost level of abstraction. Driven by the end-user’s requirement catalogue, an extensive review on the state of the art techniques allowed us to shape the preliminary definition of the low-level scene processing subsystem, including architecture design, its functional description, the initial version of some of the most relevant processing modules and the overall integration schema.
At the end of the project, the low-level pipeline was fully completed, which required that some components were refined and updated as a result of the first pilot integration. The high-level module was also upgraded, moving from learning by training to learning from human feedback, including transfer learning capabilities between target environments. During this stage, a strong collaborative effort was made in terms of both WP4 and WP5. The image descriptors produced by the low-level algorithms are the basic information unit in which the criminal activity detection layer relies. Given that, its integration was a key factor for the operational success of the system as a whole, so we followed an elaborated plan for continuous integration and validation, including the deployment of the final system and its backend infrastructure. As a result, the system integrating low and high level algorithms was successfully deployed in the final pilot location, running smoothly during the whole trial period. More significantly, it performed as it was developed to, providing consistent results which reflect and prove the concept and its practical implementation in the form of a software system for the prevention of petty crimes through their early detection.
Next figure shows in more details the data processing Software Pipeline at the server end plus any on-board smart camera feature extraction at the edge. This also includes the data collection tasks associated with the data acquisition component of the data processing pipeline. This Software Pipeline architecture consists of a series of video processing modules from the smart camera units to the backend server.
Figure 3. SmartPrevent Software Pipeline architecture.
Project Results:
The SmartPrevent consortium has produced several results which are valuable and applicable to citizen security through the enhancement video surveillance system and to support law enforcement bodies to fight against frequent petty crimes.
Figure 4. List of key impacts of SmartPrevent project
In particular, main achievements of SmartPrevent are:
▪ Understanding the principal characteristics of frequent criminal activities in real urban scenarios including typical variations and unanticipated criminal situations.
SmartPrevent Action 1: a total of nine petty crimes were selected and analysed. Evaluacion criteria was focused on technical and end-users requirements. Finally, two petty crimes (graffiti and Anti-social behaviour) were selected to develop the planned technical challenges.
▪ Developing a low-cost video-surveillance system in order to detect and prevent criminal activities in real time.
SmartPrevent Action 2: Project is using existing infrastructures of cameras. The solution is not dependent of cameras' features and low price cameras are adequate for the project's purpose.
▪ Implementing a video-surveillance system and validate system performance to detect petty crimes.
SmartPrevent Action 3: The SmartPrevent system is deployed in real scenario to validate system performance in terms of response time and right detection.
▪ Taking into account ethical and legal issues as part of main key point to deploy this system in public places.
SmartPrevent Action 4: European Laws were taken into account to design the system. End-users were interviewed to understand legal and ethical issues in urban scenarios with videosurveillance system. In addition, project ethical manager has supervised all technical developments.
▪ Ensuring a wide dissemination of results and inform different stakeholders (communities, governments, investors, etc), providing accurate description of the solution, their socio-economic impact and their contribution to improve the technology and capacities of relevant security thematic in European Cities.
SmartPrevent Action 5: SmartPrevent project was largely disseminated in several international conference as well as was invited by other related EU projects to present the system in workshops and conference.
▪ Fostering exploitation actions to install or to integrate the system or their main developments in existing video surveillance systems.
SmartPrevent Action 6: More than 20 actions were executed in terms of project exploitation.
The main results of the project are related with the principal objectives defined by the consortium. Its means that project development was focused on the five objectives defined within the Document of Work. At the end, the work done has covered the whole work plan structure including the seven WPs where the final system of project and the validation of results achieved and pilot performance have been carried out.
O1. Detection of crime on persons and local businesses in urban areas. SmartPrevent's video-surveillance system exploits the full potential of the visual analytics capabilities of these systems (and cameras network) in order to detect crimes. To do this, the project has improved current technologies in video processing for obtaining a two-level semantic understanding of the scene. This includes a low-level characterization, where information obtained concerns a simple semantic meaning, such as an unattended object, people detection, tracking moving objects, etc., and a high level, that by means of this information plus other needed information is added focusing on the understanding and detection of human activity, is capable of producing a meaning closer to human understanding of the facts, in this case, the detection of activities considered as criminal activities.
The activities related with this objective are located in the developments related to Low-Level Scene Understanding and Criminal Activity Understanding, Detection and Prevention. These activities are executed to improve the current computer vision technologies in order to process the scene (low-level features) and what is happening (high-level processing) The results are not dependent of the camera's features. These activities have been finished during second project period when both level, low and high, have reached the technical objectives and both layers were integrated. Regarding detection of crime, the low-level processing pipeline was completed, it was integrated within the overall processing pipeline, collaborating with Vision Semantics Ltd. for merging both low- and high-level software modules and the code was cross-validated with partners in charge of the high-level processing pipeline. At the end, it was modelled criminal activities and normal activities in the same scene in order to robustly detect and predict the occurrence of the former.
O2. Crime prevention over persons and local businesses in urban areas: Nowadays, systems that try to detect human activities are focused on detection of such activities when being carrying out; but this approach is not necessarily valid in all applications. In the case of video surveillance, early detection of crimes is crucial, because it is often more important to detect these criminal activities before that they occur, so as to allows for the prevention of crime and not just reacting against thereof. The proposed video surveillance system has as one of its ultimate goals the study of human activities prior to the execution of crimes, so that the system operators can perform this crime prevention efficiently.
The main goal of SmartPrevent Project is the crime prevention as complementary tool for the detection of criminal activity. The system was designed to understand the human behaviour when a crime is being executed. This timeline also includes a set of actions as preliminary step before the crime. As the system detects a group of behaviour and patterns to get the detection of crime with a defined threshold, the system is able to use this information to launch potential alarms about preventive actions.
O3. Video-surveillance system oriented for end-users: Video-surveillance systems there are two key factors for effective and efficient use, the human factor and the technological factor. In the technological factor one would consider all the hardware and software that compose the system. On the other hand, the human factor is concerned with the system users (security staff). Currently, in this field, system errors are mainly attributed to human errors. For this reason, the proposed system must have as main goal the generation of a set of applications that minimize human error as much as possible by means of the automatization of the end user tasks.
The system design is based on end-user know-how and its way of work to detect criminal activities. The technical design is oriented to the requirements of Law Enforcement Authorities (LEA), as end-users, where technical partners have been working closely to develop a robust detection framework to automatically identify the most discriminative characteristics of a criminal activity for distinguishing from a normal activity. The system performance; hardware and software have been evaluated during the prototype and pilot phases. In addition, false positive was a key indicator to assess the system results. Human factor like loos of concentration and high stress for video walls attention has been taking into account through the automation of crime detection. The system has focused on two petty crimes to improve the success rate of detection and implementing a detection threshold value to balance false positive vs crime detection. A learning from human feedback was developed to ensure a higher accuracy of the system.
O4. Video-surveillance system as effective and efficient punitive tool that protects people's privacy: Classically, video surveillance systems continuously store the outputs of the different cameras used by the system and this produces a large amount of data to be stored. For capacity reasons, these systems eliminate or overwrite the content automatically after a short period. In addition, these systems always present a clear conflict between privacy and security, since, by storing all the recorded video sequences, the privacy of the innocent people, who did not commit any criminal acts, is being violated. The SmartPrevent system allows for automatic detection of crimes, and therefore it can make a selective storage of only the moments when they are performing these criminal acts. Thus, we get two clear benefits, firstly, we only stored video content that contain a crime, this allows for better storage space utilization, consequently, we improve the system' ability to take punitive action against offenders. Secondly, the system allows the user to remove all content from people who are not performing criminal acts and directly preserves the privacy of them.
SmartPrevent system uses Privacy by Design methodology to meet with ethical and legal issues. A major issues during project was to get an adequate commitment between high rate of detection and citizen rights about data privacy. To deal with this approach, the cameras installed are low resolution devices where person identification is quite difficult especially when the camera network is placed three meters above the ground. Moreover, the system developed only captures video clips when a crime is detected to limit the range of time recorded. Anonymization process is defined when fully protection of identity is needed.
O5. Temporal and spatial adaptability: We must ensure the functionality of the video-surveillance system and the portability of the proposed system to other municipalities in different European countries. In addition, we also have to ensure that new criminal activities or vandalism acts that will appear can be detected or predicted. For these reasons, one of the main objectives of this project is to provide a set of mechanisms and techniques to ensure the adaptability of the developed system against these temporal and spatial changes.
The project is focused to implement a flexible system to be deployed in different locations without extra effort. The unexpected change of pilot location has proved that the system was designed in this way due to specifications and end-users requirement came from Las Rozas Local Police but the project was installed in Kfar Saba City. System’s architecture and modules were chosen to avoid any scenario dependencies. In terms of algorithms, the low-level is capable to understand the scene regardless the elements and physical environment. Moreover, the high-level algorithms has added special features like learning from human feedback to create an active learning module in order to adapt the system to any different location.
O6. Low-cost video-surveillance system: Currently, video-security systems are not excessively expensive to implement; increasingly visual sensors, screens, servers, etc. that compose the physically elements of the CCTV systems become cheaper. But the total cost of these systems is not just its installation. We must add its daily operations cost, having to take into account other factors such as security staff wages, power consumption cost, night lighting, data storage, etc.. The SmartPrevent system takes all these costs into account, in this way to be detailed in the impact section (Section 3), the measures proposed for lowering the costs. First, the visual sensor proposed are of low cost (both device, installation and operating), and easy to maintain, etc.. Furthermore, the intelligent storage and alarms system are oriented for the end user (indicated in previous objectives) to minimize the storage and use of security personnel and therefore the costs that incurred.
One of the results of end users questionnaire remarked that LEAs needs to implement this sort of solutions for cameras network with more than 100 cameras. For this, the surveillance system needs to be affordable and it should not increment the cost of current system. For this, the system is capable to operate with any existing system and the equipment requirement are not expensive. In addition, cameras used for the project are not expensive and with low maintenance and consumption. The system may be placed in common backend server with low computational cost and storage capacity is not high due to only positive criminal detection are recorded. Finally, the automatic analysis of detection and prevention remove the relationship among security staff and number of cameras deployment. A unique end-users can monitor a large video walls thanks to the alarm system.
At the end of second period, the project has reached its main objectives and is has completed the challenges defined for the project into the Document Of Work.
In terms of S&T results, SmartPrevent project has achieved significant challenges in three main domains;
- Visual Sensor network
- Scene understanding
- Criminal activity understanding
These challenges will improve the current video surveillance systems to be implemented in urban areas with a lower dependence of human factor.
From the work done about the Visual Sensor Network, the project has developed a visual sensor network with the following key points:
• The sensor analyzes the monitored scene and provide description and alerts according to the implemented scene analysis algorithms
• The sensors operate 24/7 in all weather conditions, with daylight as well as night-time with standard street illumination
• Sensor is able to interface and communicate with SmartPrevent wired/wireless network, sending an alert if communication is malfunctioning. In the case of the wireless network, the sensors must be covered by more than one access point, with a protocol for fault mitigation.
• Sensors respond to backend IT requests such as software update, configuration change, etc.
• Sensors are capable to provide frames at a spatial resolution of above 320x240, and a frame rate of over 15 fps. In addition, each sensor provides a time stamp for each frame and can be synchronized against other sensors.
• The metadata produced by the sensor on-board processing modules must conform the standard XML metadata format defined by the project system specification and shared by other processing modules that reside on the other platforms in the SmartPrevent system.
• The system only records situations in which the system detects a criminal activity. As a result, the system only records alert videos. This is an important safeguard against privacy invasive surveillance practices
For this, SmartPrevent has designed three main components:
1. WiseEye-3.0: a new version of WiseEye video sensor was designed and developed with improved processing capabilities, low light sensitivity, WIFI support, and larger physical memory – to enable additional of SmartPrevent enhanced algorithm. The WiseEye visual sensor is designed to provide:
a. Real Time video verification and notification
b. Real Time alert video clip sent to Central Station/Monitoring room
c. XML notification with video to 3rd party VMS
d. Email notification and video clip to mobile device
e. External IO signal (Dry contact)
2. WiseShield-01: Video encoder with WiseEye-3.0 SW , Linux device to be used for development aid and for performance simulation
3. Control Unit (ECU): The ECU is a device that controls and manage communicates with all WiseEye units in the site and records alerts clips. It is used as a gateway between the local WiseEye sensors and the remote control room. The ECU provides archiving and act as NVR that records all alerts video clips. In case of low bandwidth communicating line between the remote site and the control room, the ECU is used as temporary buffer between the WiseEye sensors (that do not have storage to keep alerts history) and the backend server. Each alert has its time and location stamp and is viewed as alert video clip via standard Internet media player.
Finally, the WiseEye® is upgrade with the following capabilities included, but not limited to:
▪ Accurate detection in rain, snow, dust, trees etc.
▪ Human detection range over 300 ft. (100 meters)
▪ Day/Night, all weather operation
▪ Low bandwidth real time alerts over cellular
▪ Extreme night sensitivity (0.01 lux)
▪ PoE support, dry contact input / output
▪ Dual stream H.264 and MJPEG
▪ WIFI 802.11b/g/n
▪ Low-power (1.5 Watt & 4 Watt the WIFI version)
▪ Complete web support
▪ ONVIF support
▪ Plug N’ Play
▪ Supports several commercial Video Management Systems (VMS)
• Milestone, Sureview, Bold,
▪ Provides video alerts and verification over the Ethernet by mail and by cloud applications
A key technical point of the video surveillance system is focused on the interpretation of the scene carried out by an automatically way. The following list highlights the most important actions carried out for each module implementing the low-level pipeline of the system:
• Image acquisition. The first approach for this module included a simple yet effective implementation for capturing data from the visual sensors’ network and from stored video sequences on demand. During the refinement phase, and for reasons related to the feature extraction module, we implemented an alternative approach based on the FFMPEG libraries. This provided the system with the ability of handling MPEG-based streams, which not only decodes image data from the source stream, but also additional data encoded within as part of the image, providing additional information for a richer scene description at its lower level.
• Background subtraction. It was organised the components of this module into three categories; Foreground mask, Input-frame and Background model. This module was focused on fine-tune the parameterised functions in charge of handling aspects such as the static modelling of the scene or the filtering operations performed over the source images and resulting masks.
• Shadow handling. During the foreground object segmentation, shadows are often a difficult issue to deal with. Distinguishing moving objects from shadows is a vital task because shadows can cause various problems such as shape distortion, phantoms, over/under-segmentation, false crowding, etc. The outcome from this stage is a refined copy of the foreground saliency mask obtained in the previous step, in which the regions corresponding to shadow areas are masked and merged with the negative area of the mask. In our implementation, it was chosen a colour space with better ability for splitting between chromaticity and intensity values than the RGB is capable to. As described, we selected the HSV colour space based on the previous results from Rita Cucchiara, taking advantage of the fact that HSV colour space provides separation between chromaticity and luminosity by means of how the colour model is designed. This method has been widely adopted in video-based surveillance applications, especially in those that involve segmentation for detection purposes. The main reason for choosing this solution is a good compromise between performance and computational cost. As main disadvantage of the method, the value comparisons are made at the pixel-level, which makes it sensitive to image noise. Pre and post-processing performed as part of the background subtraction stage contributes to mitigate this effect, as most of the image and mask noise is reduced by applying different filtering operations.
• Region of interest (ROI) extraction. As a result of a preliminary testing phase of our implementation for this approach, it was observed that the computational complexity of the method would cause an important bottleneck, penalizing the performance of the entire pipeline. Consequently, project worked towards the idea of minimizing the computation required for each image frame, reducing the required amount of computations from the pixel-level to the region-level. This strategy required us to implement spatio-temporal consistency mechanisms for ensuring that only the actual salient regions stand out. This improvement is very useful, especially in situations involving sudden motions, crowded environments or scenes with multiple sources of relevant action
• Person detection. This component was practically implemented during the last development phase. Initially, a person detection algorithm, based on Dollàr’s approximation was considered. Later, we discovered that this method would not be suitable for our global approach, because this technique assumes fixed person sizes. Additionally, the person to be detected should be represented by around the 50% of the image height. These assumptions were not valid for the project scenarios, so we explored a number of alternatives, but finally we decided to go with a multi-scale oriented gradients based approach. The algorithm implemented for person detection is mainly described by two sequential phases. In the first one, the feature vectors representing a collection of descriptors, known as Histograms of Oriented Gradients (HOG), are extracted from a detection window of a particular size. Then, the feature vectors are computed by a linear SVM for obtaining a binary person/non-person classification.
• Low-level features computation. Once the previously described modules were developed and validated, according to the requirements listed by final consumer for the action recognition module (WP5), the first functional implementation of this module, known as Dense Trajectories, was carried out. However, it was found several disadvantages that could affect the system performance, mainly due to computationally-demanding processes, such as computing the optical flow for the image sequences. For this, it was implemented an alternative method (known as Motion Vectors) for extracting similar feature descriptors from the scene, as we did using the previous method. In this case, the optical flow was replaced by the motion information encoded as part of the compression scheme implemented by most codecs designed for digital video streaming. Both MBHx and MBHy descriptors shown very similar values in both approaches, but, by means of the motion vectors-based one, the overall process is around two orders of magnitude faster than it was using optical flow.
The following diagram shows an overview of the individual elements composing the final implementation of the low-level scene description pipeline.
Figure 5. Overview of the final software packages of the low-level scene understanding pipeline. For each package, its top-level modules are also shown.
Finally, the core challenge of the project is related with the capacity of the system to develop a Criminal Activity Understanding module for petty crimes.
In this regard, the main objective of this module is to model criminal activities and normal activities in the same scene in order to robustly detect and predict the occurrence of the former.
Figure 6. A screen shot of the GUI of the SmartPrevent criminal activity detection module.
SmartPrevent has developed a state-of-the-art sliding window based detector, in order to deal with the scale and pose changes in activities captured in a real-world environment. The result shows that our developed method is very competitive against the state-of-the-arts. However the initial activity detection module required sufficient training data for model learning, which limits the fast deployment of the module for new visual scenes/environments. In the refined software module, this limitation is overcome by learning from human feedback based on active learning and transfer learning from source environments where labelled samples are available to target environments where no data annotation is required. These two new capabilities have been integrated into the criminal activity detection software modules and evaluated on benchmark datasets to validate their effectiveness.
Since the SmartPrevent system is to be used by a human operator who will make judgement on whether the alert generated by the system is genuine of false, it was critical that the system can learn from the human feedback and improve itself automatically and on-the-fly, so that similar mistakes can be avoided in the future. An active learning module was developed so that our activity detection model can be improved whenever human feedback is available. Our focus was on the active training sample selection criterion: assuming that getting human feedback incurs a cost because human operators’ time is limited, this criterion is used to select the most informative samples for getting human feedback so that the resulting improvement of the detection model is the most pronounced.
Project has obtained an optimised model by using as few data labels as possible. To that end, it was adopted the common uncertainty sampling strategy. More specifically, it was utilised the entropy-based active query selection algorithm, which is a natural measurement of informativeness.
Regarding the classification model, in the SmartPrevent criminal activity detection module, it was adopted the common Support Vector Machine (SVM) model as our binary classifier which gives a yes or no answer to a given video clip regarding whether it contains an instance of one type of criminal activity. It should be noted that this choice is in- dependent from our active learning method.
In terms of transfer learning method, project has presented a domain adaptation method for more effectively and rapidly exploiting a pre-trained model learning from one domain but deployed to a new unseen domain. This model allows to overcome the inherent domain shift challenge between different scenes by aligning the subspaces of two domains in a supervised fashion. This function is practically useful and valuable since it is not scalable to collect a sufficiently large set of training data for any target domains/scenes in reality, which can be over costly. It was demonstrated the efficacy of the proposed domain adaptation method on a realistic graffiti data collected in the project. In particular, the detection accuracy of a model learned in one domain is clearly improved when the model is applied to a different domain, after the unsupervised domain adaptation method is applied to align the two domains.
Finally, we have also developed a more radical method of transfer learning based on zero-shot learning. By this method, one could detect an activity which one only reads about but has never seen any visual examples before. Having this model makes the SmartPrevent system more broadly applicable.
Potential Impact:
SmartPrevent project has addressed the main priorities of the Security Research, Call 6, matching perfectly with the topic “Solution for frequent petty crimes that are high impact to local communities and citizens”, considered in the Security Work Programme 2013. For this, some of the principal impacts achieved by the project are aligned with the European strategy in Security domain.
As part of this strategic, innovative low cost solutions should be implemented to reduce crime against local communities, business and citizen. SmartPrevent system is based on current video surveillance system to improve their own intelligence in order to provide early criminal activities detection. First petty crimes selected; graffiti and anti-social behaviour are fully oriented to local governments and citizens. In addition, the system is scalable to further scenarios and petty crimes thanks to the active learning and transfer learning for fast deployment methodologies. The system was validated to demonstrate the positive impact to detect criminal activity as well as to be capable to detect in advance, prevent, the potential petty crimes.
In particular, the impact of SmartPrevent has been significant and large on the academic and industrial, and end-users communities related to crime prevention using video-surveillance systems in general, and specifically in terms of crime prevention in urban scenarios by means of a low-cost system tailor-made to meet the needs of end users.
Our impact has reached the European organizations in terms of:
▪ Providing a set of methodologies for a better use of video-surveillance system in crime prevention and detection, taking into account legal, ethical issues and better technologies, methods and algorithms.
▪ Providing a set of tools based on video-processing of video sequence and enrichment with relevant semantic information of data (tracking, definition of object, relations), so that a better and more rapid response to petty crime in urban scenarios will be enabled.
The extent of our impact outlined above is elaborated further as follows:
a. Providing a set of methodologies for a better use of video-surveillance system in crime prevention and detection.
SmartPrevent has developed several methodologies and recommendations that enables a more rapid response and an early detection of activities related with petty crimes. The work done during data acquisition and ethical issues review will act as starting point for similar project. An innovative approach to crimes prevention and detection proposed by SmartPrevent is that the recognition of suspicious activities can be used, not only to detect crime, but also to prevent it. Computer vision methodologies are able to understand scene configuration and activities that happens in this environments. Finally, the system provides a spatial and temporal adaptability to these criminal activities in order to adapt the system in each local environment in any time. The high dependency of computer vision solution to scenarios and conditions has been an important barriers to deploy smart video surveillance where human factor is only used as supervisor. For this, SmartPrevent is scalable and flexible to different infrastructure and urban scenarios.
b. Providing a set of tools based on video-processing of video sequence and enrichment with relevant semantic information of sequences, so that a better and more rapid response to crime in urban scenarios will be enabled.
The main achievements of SmartPrevent project are based on new methodologies to be used in video surveillance system to detect and prevent petty crimes. In order to strengthen this impact, a set of tools to support methodologies, as well as enabling understanding of video sequence have been developed. As a result, video-surveillance managers will be provided with real-time added-value information for a better and more efficient decision-making process. The project has focused on two main developments to understand the scene where petty crimes could occurs as well as to understand the criminal activity in an urban scenarios where several common activities are also part of the activity understanding. Firstly, a low-level pipeline is implemented to improve the scene understanding and reduce the computational cost of higher levels. This first approach is able to provide an enriched data removing not substantial information like shadows, area without interest to the upper layer where high level algorithms must discover the criminal activity with a low rate of false positive.
Thus, SmartPrevent has addressed the following areas to provide a significant impact at European level:
▪ Project has delivered technological solutions for civil protection, including protection against the risks arisen from crime and terrorist attacks. Our solution is able to be improved from an easy way to add more uses cases and criminal activities to be detected as well.
▪ SmartPrevent is focused on technologies and capabilities to enhance the effectiveness and efficiency of current systems, equipment, tools and processes and methods to improve the security of Europe. The system is interoperable and can be integrated in any Security Control Center where operators can easily to manage the system.
▪ SmartPrevent was designed taking into account the privacy rules and requirements of Law Enforcement Authorities to develop an almost plug and play system without dependency of wherever the system is deployed.
On the other hand, SmartPrevent does not only meet technical impact but social impact is a core activity of the project. At the end, the system is developed to support Police bodies to improve citizen security in urban areas and the system also must be considered as a useful and adequate tool form the citizen perspective taking into consideration privacy right and data protection.
The SmartPrevent project has intended to have a social impact, in that it is meant to identify attempted criminal activity and offer Law Enforcement agencies an affordable tool to better protect citizen against criminal acts. Thus, the project intends to give more security and sense of security to the citizens. The SmartPrevent solution will have a positive social impact by reducing the economic and social cost of petty crimes in urban areas, and this reduction in illegal activity will have significant amplified positive impacts for society and the economy making safer cities and communities. Petty crimes selected by the project have a negative impact regarding societal point of view like the graffiti actions and anti-social behaviour. In addition, a reduction in petty crime increases the sense of public order which produces a reduction of crime in general.
SmartPrevent is also focused on the crime prevention adding active learning methods to understand in advance potential petty crimes. It makes a higher social impact when a system is able to detect potential crime before occurs. The sense of public safety would grow exponentially as more automatic learning is added to the system with the expert advisory of police officers.
Moreover, SmartPrevent implemented in Police Control Center is a new data source and data repository to give an effective information about criminal activities to support police guidelines. This information may be used to improve European Security Policies.
Finally, the impact of SmartPrevent is carried out by a set of dissemination and exploitation actions, whom were led by all the partners involved to enhance a broad audience.
Numerous dissemination events have taken place besides the officially planned ones like the international events in Computer Vision (IEEE AVSS and ICDP). There are additional organizational presentations principally dealing with the relationship between members of the consortium and their respective organizations but also dissemination to the sales organizations and in general all the people dealing with innovation. A detailed list is provided in next section. In addition, the SmartPrevent consortium hosted a User Conference in Las Rozas de Madrid, Spain on 26th April 2016 (http://www.smartprevent.eu/node/56). More than 50 participants attended the conference, mainly from the Spanish end-users as well as other partners of EU projects including LASIE, FORENSOR, P-REACT, ADDPRIV and the host project SmartPrevent. They include delegates from the City Council, Spanish Local Police bodies, academic research groups, SMEs on developing commercial computer vision systems, National Police and representatives of the Ministry of Defence.
The conference was a successful event for the project dissemination and a good opportunity for presenting the SmartPrevent project results to the external users and other relevant EU projects. This final project conference also succeeded in promoting new technologies and methodologies for enhancing existing video surveillance systems designed for the prevention and detection of petty crimes in urban environments.
Regarding exploitation actions, SmartPrevent has defined a twofold approach to promote the project exploitation. Firstly, a general exploitation plan was created to push the potential actions from the whole consortium including the planned exploitation actions and social gains and economic feasibility. In a second way, each partner with interest in exploitation actions has planned and carried out specific actions to exploit the technical foreground and new knowledge adding them to their own portfolio of new solutions and services.
List of Websites:
The SmartPrevent project website (www.smartprevent.eu) was designed and created to not only have a professional look, but also provide an effective media for communicating the project objectives with the scientific and industrial communities in a wider context. More details about the project website and a brochure are given in D7.11 (Project Brochure and Project Website).
It consists of seven sections:
• Home: a summary and the project fiche on the right – full name, topic, duration, consortium – are displayed. There is a small block showing the latest news on the lower right hand side.
• Project description: the objectives are listed and the work packages are briefly described, indicating the outcomes and the leader.
• Consortium: a description of the partners together with their roles and contact information.
• Advisory board: a list of entities involved as members of the Advisory Board Group.
• News and events: up-to-date information about recent news, as well as past and forthcoming events. This section has been continuously modified with the addition of new information.
• Documents: it is made of three categories – public deliverables, publications, and dissemination material (project summary and leaflet). This is the section envisioned to allow the user to download documents and dissemination materials.
• Contact: information to get in touch with the coordinator.
• Login: this is divided into two blocks. The one on the left is intended to add/remove/edit contents in the website and the one on the right gives access to Alfresco.
As mentioned before, the section focused on the news and events displays updated information about the participation of the partners in workshops, sister projects from FP7 and H2020 and other relevant actions in the context of dissemination.
In addition, Social media is a good way of making public advances and results. Even though creating a Twitter account (@smartprevent) was not contemplated at the beginning, the consortium decided to do in order to improve the communication channels of the project. This account was used to disseminate and promote events where project has participated and also to be updated about news and trends related with video surveillance domain, ethical issues and to listen end users on security issues. .
Furthermore, the project had designed a logo during the proposal stage that was used for the whole project execution.
Finally, contact details of each partner is published on Project website in consortium tab.
The increase in urban population lead to an increase in crime. This is aggravated by illegal immigration, criminal striping, and degradation of zones. The increase of criminal activity is manifested in an increased frequency of small crimes like graffiti, theft, robbery, destruction of street furniture. This has had a big impact for local governments, citizens and businesses.
Local government agencies are trying to solve these problems. As was shown in New York, a reduction in petty crimes increases the sense of public order which produces further reduction in crime in general. These organizations often treat the problem with classical solutions such as increase in police resources, reintegration of offenders, rehabilitation of degraded areas and modern solutions such as urban video surveillance. However, these new technologies are often underused and inadequate.
The SmartPrevent project aims to enhance detection and prevention of crimes in local urban areas by exploiting the full potential of video-surveillance systems. Project develops and provides four important benefits: i) Systematic characterization of usual petty crimes in an area under automatic surveillance; ii) automatic detection of the most usual and frequent criminal activities; iii) a set of automated tools capable of alerting the appropriate responders; and iv) early prevention of crimes by prediction and early detection of crimes.
Rather than providing new methodologies or tools, SmartPrevent is focused on: a) improving already-existing methodologies by means of a set of guidelines for the use of video-surveillance systems; and b) providing a set of tools capable to improve of the existing crime detection systems. Our solution was validated by deploying a realistic prototype scenario, which will actively involve the detection and prevention for crimes in an urban areas and management of these detections by final users.
Project Context and Objectives:
The SmartPrevent project focuses on the detection and prevention of frequent petty crimes that are of high impact to local communities and citizens in urban scenarios, considered to be a low-cost video-surveillance system oriented to end-users. For this, the consortium involves a balanced number of partners with complementary background from the technical aspect to ethical and legal domain as well as the perspective of end-users.
Current video-surveillance system in urban scenarios are very limited and only consist in a presentation of the visual information captured by the visual sensors network, not oriented to end-users, that limiting thus the capacity to help and prevent the criminal activity. Furthermore, the visual-surveillance systems usually do not have any automatic process in order to store the more relevant evidences to take in account in legal punitive process of the criminals. It means that most video monitoring applications require human observer during the whole surveillance process. A key limitation comes from the human capacity that cannot attentively follow more than 10 cameras or screen for large periods of time. However, the entire workload falls on the end user of the application. Firstly, this produces a high stress level on security staff and also needs a high number of security staff that must be proportional to the number of cameras. In addition, the inherent cost of human resource make current solution clearly unsustainable in the long run.
SmartPrevent’s challenge is focused to support security staff the responsibility of ensuring an ongoing video surveillance to enhance reliability system in urban scenarios.
Indeed, SmartPrevent proposes to address this challenge by:
• Studying the characteristics of frequent criminal activities in urban scenarios in realistic scenarios, including typical variations and unanticipated criminal situations.
• Developing a low-cost adaptive video-surveillance system in order to detect and prevent the criminal activities.
• Building a video-surveillance system as punitive tools in order to store the most relevant evidences of the detected criminal activities.
In a nutshell, the SmartPrevent objectives are:
Figure 1. SmartPrevent Scientific and Technical Objectives
These objectives are achieved by project as detailed below:
·Detection of crime on persons and local businesses in urban areas. SmartPrevent's video surveillance system will exploit the full potential of the visual analytics capabilities of these systems in order to detect crimes. To do this, we will improve current technologies in video processing for obtaining a two-level semantic understanding of the scene. This includes a low level characterization, where information obtained will concern a simple semantic meaning and a high level, that by means of this information plus other needed information is added focusing on the detection of human activity, is capable of producing a meaning closer to human understanding of the facts.
· Crime prevention over persons and local businesses in urban areas: Nowadays, systems that try to detect human activities are focused on detection of such activities when being carrying out; but this approach is not necessarily valid in all applications. In the case of video surveillance, early detection of crimes is crucial, because it is often more important to detect these criminal activities before that they occur, so as to allows for the prevention of crime and not just reacting against thereof.
· Video-surveillance system oriented for end-users: Video-surveillance systems there are two key factors for effective and efficient use, the human factor and the technological factor. In the technological factor, one would consider all the hardware and software that compose the system.
On the other hand, the human factor is concerned with the system users (security staff).
Currently, in this field, system errors are mainly attributed to human errors. For this reason, the proposed system must have as main goal the generation of a set of applications that minimize human error as much as possible by means of the automatization of the end user tasks.
· Video-surveillance system as effective and efficient punitive tool that protects people's privacy: The SmartPrevent system allows for automatic detection of crimes, and therefore it can make a selective storage of only the moments when they are performing these criminal acts. Thus, we get two clear benefits, firstly, we only stored video content that contain a crime, this allows for better storage space utilization, consequently, we improve the system ability to take punitive action against offenders. Secondly, the system allows the user to remove all content from people who are not performing criminal acts and directly preserves the privacy of them.
· Temporal and spatial adaptability: We must ensure the functionality of the video-surveillance system and the portability of the proposed system to other municipalities in different European countries. In addition, we also have to ensure that new criminal activities or vandalism acts that will appear can be detected or predicted.
· Low-cost video-surveillance system: CCTV system are not expensive but the cost increase adding all related operations and personnel involved. The SmartPrevent system takes all these costs into consideration. First, the visual sensor proposed are of low cost (both device, installation and operating), and easy to maintain, etc.. Secondly, the intelligent storage and alarms system are oriented for the end user (indicated in previous objectives) to minimize the storage and use of security personnel and therefore the costs that incurred.
SmartPrevent project context and activity are arranged in seven work packages. Overall, during the 26 months, the consortium has performed all the tasks and achieved most of the results as planned where all the WPs are involved. Mainly, WP4, 5 and 6 were the activities with higher workload because of the starting of pilot phase and development of SmartPrevent system. However, non-technical activities as well as ethical issues took a large number of efforts since the Madrid government decline the pilot implementation approval. At the end, pilot phase were carried out in Kfar Saba thanks the management of EVS. This new location was aligned with the end users requirements and former pilot location, Las Rozas de Madrid. Ethical issues review was carried out for the new location in order to comply with the national rules and European Guidelines for citizen privacy.
Figure 2. Diagram for SmartPrevent
In relation with the work package scope the progress and results achieved during the whole period are summarized here below:
Definition of scenarios, criminal actions and Pilot requirements.
The final description of end user requirements were completed during the first project period in collaboration with the end user and its Local Police body. This information was discussed and improved to release a final document for the design of the software architecture and the functionalities to be developed. Moreover, these user specifications were updated with the new location of pilot phase to ensure that are valid for different scenarios.
Finally, once several scenarios and use cases were studied, Graffiti and anti-social behaviour were selected as two petty crimes to be used in order to develop the system and validate the performance during a pilot phase in Kfar Saba.
Ethical review of the data and the study of ethical issues for Pilot phase.
As a core activity of the project, privacy rights and ethical issues were taking into consideration to develop any technical stage. Ethical issues relationship with video surveillance and, in particular, the new location were carried out. At this point, Israel, where the pilot is run, do not have the same rules and normative than Europe. Cameras installation and video surveillance project in public areas are not under control of government and official approval is mandatory. In any case, the project was based on European rules and project guidelines to cover the ethical and legal issues. The Ethical Review committee was monitoring the datasets captured and validate the accordance of the pilot within ethical guidelines.
Software system for event description, detection and recognition.
The envisioned approach involved the development of a two-layer software system. The low-level scene understanding layer is in charge of gathering images from the CCTV devices for transforming it into semantic descriptors. These descriptors are the baseline data for the high-level layer, in charge of the activity detection and recognition at a topmost level of abstraction. Driven by the end-user’s requirement catalogue, an extensive review on the state of the art techniques allowed us to shape the preliminary definition of the low-level scene processing subsystem, including architecture design, its functional description, the initial version of some of the most relevant processing modules and the overall integration schema.
At the end of the project, the low-level pipeline was fully completed, which required that some components were refined and updated as a result of the first pilot integration. The high-level module was also upgraded, moving from learning by training to learning from human feedback, including transfer learning capabilities between target environments. During this stage, a strong collaborative effort was made in terms of both WP4 and WP5. The image descriptors produced by the low-level algorithms are the basic information unit in which the criminal activity detection layer relies. Given that, its integration was a key factor for the operational success of the system as a whole, so we followed an elaborated plan for continuous integration and validation, including the deployment of the final system and its backend infrastructure. As a result, the system integrating low and high level algorithms was successfully deployed in the final pilot location, running smoothly during the whole trial period. More significantly, it performed as it was developed to, providing consistent results which reflect and prove the concept and its practical implementation in the form of a software system for the prevention of petty crimes through their early detection.
Next figure shows in more details the data processing Software Pipeline at the server end plus any on-board smart camera feature extraction at the edge. This also includes the data collection tasks associated with the data acquisition component of the data processing pipeline. This Software Pipeline architecture consists of a series of video processing modules from the smart camera units to the backend server.
Figure 3. SmartPrevent Software Pipeline architecture.
Project Results:
The SmartPrevent consortium has produced several results which are valuable and applicable to citizen security through the enhancement video surveillance system and to support law enforcement bodies to fight against frequent petty crimes.
Figure 4. List of key impacts of SmartPrevent project
In particular, main achievements of SmartPrevent are:
▪ Understanding the principal characteristics of frequent criminal activities in real urban scenarios including typical variations and unanticipated criminal situations.
SmartPrevent Action 1: a total of nine petty crimes were selected and analysed. Evaluacion criteria was focused on technical and end-users requirements. Finally, two petty crimes (graffiti and Anti-social behaviour) were selected to develop the planned technical challenges.
▪ Developing a low-cost video-surveillance system in order to detect and prevent criminal activities in real time.
SmartPrevent Action 2: Project is using existing infrastructures of cameras. The solution is not dependent of cameras' features and low price cameras are adequate for the project's purpose.
▪ Implementing a video-surveillance system and validate system performance to detect petty crimes.
SmartPrevent Action 3: The SmartPrevent system is deployed in real scenario to validate system performance in terms of response time and right detection.
▪ Taking into account ethical and legal issues as part of main key point to deploy this system in public places.
SmartPrevent Action 4: European Laws were taken into account to design the system. End-users were interviewed to understand legal and ethical issues in urban scenarios with videosurveillance system. In addition, project ethical manager has supervised all technical developments.
▪ Ensuring a wide dissemination of results and inform different stakeholders (communities, governments, investors, etc), providing accurate description of the solution, their socio-economic impact and their contribution to improve the technology and capacities of relevant security thematic in European Cities.
SmartPrevent Action 5: SmartPrevent project was largely disseminated in several international conference as well as was invited by other related EU projects to present the system in workshops and conference.
▪ Fostering exploitation actions to install or to integrate the system or their main developments in existing video surveillance systems.
SmartPrevent Action 6: More than 20 actions were executed in terms of project exploitation.
The main results of the project are related with the principal objectives defined by the consortium. Its means that project development was focused on the five objectives defined within the Document of Work. At the end, the work done has covered the whole work plan structure including the seven WPs where the final system of project and the validation of results achieved and pilot performance have been carried out.
O1. Detection of crime on persons and local businesses in urban areas. SmartPrevent's video-surveillance system exploits the full potential of the visual analytics capabilities of these systems (and cameras network) in order to detect crimes. To do this, the project has improved current technologies in video processing for obtaining a two-level semantic understanding of the scene. This includes a low-level characterization, where information obtained concerns a simple semantic meaning, such as an unattended object, people detection, tracking moving objects, etc., and a high level, that by means of this information plus other needed information is added focusing on the understanding and detection of human activity, is capable of producing a meaning closer to human understanding of the facts, in this case, the detection of activities considered as criminal activities.
The activities related with this objective are located in the developments related to Low-Level Scene Understanding and Criminal Activity Understanding, Detection and Prevention. These activities are executed to improve the current computer vision technologies in order to process the scene (low-level features) and what is happening (high-level processing) The results are not dependent of the camera's features. These activities have been finished during second project period when both level, low and high, have reached the technical objectives and both layers were integrated. Regarding detection of crime, the low-level processing pipeline was completed, it was integrated within the overall processing pipeline, collaborating with Vision Semantics Ltd. for merging both low- and high-level software modules and the code was cross-validated with partners in charge of the high-level processing pipeline. At the end, it was modelled criminal activities and normal activities in the same scene in order to robustly detect and predict the occurrence of the former.
O2. Crime prevention over persons and local businesses in urban areas: Nowadays, systems that try to detect human activities are focused on detection of such activities when being carrying out; but this approach is not necessarily valid in all applications. In the case of video surveillance, early detection of crimes is crucial, because it is often more important to detect these criminal activities before that they occur, so as to allows for the prevention of crime and not just reacting against thereof. The proposed video surveillance system has as one of its ultimate goals the study of human activities prior to the execution of crimes, so that the system operators can perform this crime prevention efficiently.
The main goal of SmartPrevent Project is the crime prevention as complementary tool for the detection of criminal activity. The system was designed to understand the human behaviour when a crime is being executed. This timeline also includes a set of actions as preliminary step before the crime. As the system detects a group of behaviour and patterns to get the detection of crime with a defined threshold, the system is able to use this information to launch potential alarms about preventive actions.
O3. Video-surveillance system oriented for end-users: Video-surveillance systems there are two key factors for effective and efficient use, the human factor and the technological factor. In the technological factor one would consider all the hardware and software that compose the system. On the other hand, the human factor is concerned with the system users (security staff). Currently, in this field, system errors are mainly attributed to human errors. For this reason, the proposed system must have as main goal the generation of a set of applications that minimize human error as much as possible by means of the automatization of the end user tasks.
The system design is based on end-user know-how and its way of work to detect criminal activities. The technical design is oriented to the requirements of Law Enforcement Authorities (LEA), as end-users, where technical partners have been working closely to develop a robust detection framework to automatically identify the most discriminative characteristics of a criminal activity for distinguishing from a normal activity. The system performance; hardware and software have been evaluated during the prototype and pilot phases. In addition, false positive was a key indicator to assess the system results. Human factor like loos of concentration and high stress for video walls attention has been taking into account through the automation of crime detection. The system has focused on two petty crimes to improve the success rate of detection and implementing a detection threshold value to balance false positive vs crime detection. A learning from human feedback was developed to ensure a higher accuracy of the system.
O4. Video-surveillance system as effective and efficient punitive tool that protects people's privacy: Classically, video surveillance systems continuously store the outputs of the different cameras used by the system and this produces a large amount of data to be stored. For capacity reasons, these systems eliminate or overwrite the content automatically after a short period. In addition, these systems always present a clear conflict between privacy and security, since, by storing all the recorded video sequences, the privacy of the innocent people, who did not commit any criminal acts, is being violated. The SmartPrevent system allows for automatic detection of crimes, and therefore it can make a selective storage of only the moments when they are performing these criminal acts. Thus, we get two clear benefits, firstly, we only stored video content that contain a crime, this allows for better storage space utilization, consequently, we improve the system' ability to take punitive action against offenders. Secondly, the system allows the user to remove all content from people who are not performing criminal acts and directly preserves the privacy of them.
SmartPrevent system uses Privacy by Design methodology to meet with ethical and legal issues. A major issues during project was to get an adequate commitment between high rate of detection and citizen rights about data privacy. To deal with this approach, the cameras installed are low resolution devices where person identification is quite difficult especially when the camera network is placed three meters above the ground. Moreover, the system developed only captures video clips when a crime is detected to limit the range of time recorded. Anonymization process is defined when fully protection of identity is needed.
O5. Temporal and spatial adaptability: We must ensure the functionality of the video-surveillance system and the portability of the proposed system to other municipalities in different European countries. In addition, we also have to ensure that new criminal activities or vandalism acts that will appear can be detected or predicted. For these reasons, one of the main objectives of this project is to provide a set of mechanisms and techniques to ensure the adaptability of the developed system against these temporal and spatial changes.
The project is focused to implement a flexible system to be deployed in different locations without extra effort. The unexpected change of pilot location has proved that the system was designed in this way due to specifications and end-users requirement came from Las Rozas Local Police but the project was installed in Kfar Saba City. System’s architecture and modules were chosen to avoid any scenario dependencies. In terms of algorithms, the low-level is capable to understand the scene regardless the elements and physical environment. Moreover, the high-level algorithms has added special features like learning from human feedback to create an active learning module in order to adapt the system to any different location.
O6. Low-cost video-surveillance system: Currently, video-security systems are not excessively expensive to implement; increasingly visual sensors, screens, servers, etc. that compose the physically elements of the CCTV systems become cheaper. But the total cost of these systems is not just its installation. We must add its daily operations cost, having to take into account other factors such as security staff wages, power consumption cost, night lighting, data storage, etc.. The SmartPrevent system takes all these costs into account, in this way to be detailed in the impact section (Section 3), the measures proposed for lowering the costs. First, the visual sensor proposed are of low cost (both device, installation and operating), and easy to maintain, etc.. Furthermore, the intelligent storage and alarms system are oriented for the end user (indicated in previous objectives) to minimize the storage and use of security personnel and therefore the costs that incurred.
One of the results of end users questionnaire remarked that LEAs needs to implement this sort of solutions for cameras network with more than 100 cameras. For this, the surveillance system needs to be affordable and it should not increment the cost of current system. For this, the system is capable to operate with any existing system and the equipment requirement are not expensive. In addition, cameras used for the project are not expensive and with low maintenance and consumption. The system may be placed in common backend server with low computational cost and storage capacity is not high due to only positive criminal detection are recorded. Finally, the automatic analysis of detection and prevention remove the relationship among security staff and number of cameras deployment. A unique end-users can monitor a large video walls thanks to the alarm system.
At the end of second period, the project has reached its main objectives and is has completed the challenges defined for the project into the Document Of Work.
In terms of S&T results, SmartPrevent project has achieved significant challenges in three main domains;
- Visual Sensor network
- Scene understanding
- Criminal activity understanding
These challenges will improve the current video surveillance systems to be implemented in urban areas with a lower dependence of human factor.
From the work done about the Visual Sensor Network, the project has developed a visual sensor network with the following key points:
• The sensor analyzes the monitored scene and provide description and alerts according to the implemented scene analysis algorithms
• The sensors operate 24/7 in all weather conditions, with daylight as well as night-time with standard street illumination
• Sensor is able to interface and communicate with SmartPrevent wired/wireless network, sending an alert if communication is malfunctioning. In the case of the wireless network, the sensors must be covered by more than one access point, with a protocol for fault mitigation.
• Sensors respond to backend IT requests such as software update, configuration change, etc.
• Sensors are capable to provide frames at a spatial resolution of above 320x240, and a frame rate of over 15 fps. In addition, each sensor provides a time stamp for each frame and can be synchronized against other sensors.
• The metadata produced by the sensor on-board processing modules must conform the standard XML metadata format defined by the project system specification and shared by other processing modules that reside on the other platforms in the SmartPrevent system.
• The system only records situations in which the system detects a criminal activity. As a result, the system only records alert videos. This is an important safeguard against privacy invasive surveillance practices
For this, SmartPrevent has designed three main components:
1. WiseEye-3.0: a new version of WiseEye video sensor was designed and developed with improved processing capabilities, low light sensitivity, WIFI support, and larger physical memory – to enable additional of SmartPrevent enhanced algorithm. The WiseEye visual sensor is designed to provide:
a. Real Time video verification and notification
b. Real Time alert video clip sent to Central Station/Monitoring room
c. XML notification with video to 3rd party VMS
d. Email notification and video clip to mobile device
e. External IO signal (Dry contact)
2. WiseShield-01: Video encoder with WiseEye-3.0 SW , Linux device to be used for development aid and for performance simulation
3. Control Unit (ECU): The ECU is a device that controls and manage communicates with all WiseEye units in the site and records alerts clips. It is used as a gateway between the local WiseEye sensors and the remote control room. The ECU provides archiving and act as NVR that records all alerts video clips. In case of low bandwidth communicating line between the remote site and the control room, the ECU is used as temporary buffer between the WiseEye sensors (that do not have storage to keep alerts history) and the backend server. Each alert has its time and location stamp and is viewed as alert video clip via standard Internet media player.
Finally, the WiseEye® is upgrade with the following capabilities included, but not limited to:
▪ Accurate detection in rain, snow, dust, trees etc.
▪ Human detection range over 300 ft. (100 meters)
▪ Day/Night, all weather operation
▪ Low bandwidth real time alerts over cellular
▪ Extreme night sensitivity (0.01 lux)
▪ PoE support, dry contact input / output
▪ Dual stream H.264 and MJPEG
▪ WIFI 802.11b/g/n
▪ Low-power (1.5 Watt & 4 Watt the WIFI version)
▪ Complete web support
▪ ONVIF support
▪ Plug N’ Play
▪ Supports several commercial Video Management Systems (VMS)
• Milestone, Sureview, Bold,
▪ Provides video alerts and verification over the Ethernet by mail and by cloud applications
A key technical point of the video surveillance system is focused on the interpretation of the scene carried out by an automatically way. The following list highlights the most important actions carried out for each module implementing the low-level pipeline of the system:
• Image acquisition. The first approach for this module included a simple yet effective implementation for capturing data from the visual sensors’ network and from stored video sequences on demand. During the refinement phase, and for reasons related to the feature extraction module, we implemented an alternative approach based on the FFMPEG libraries. This provided the system with the ability of handling MPEG-based streams, which not only decodes image data from the source stream, but also additional data encoded within as part of the image, providing additional information for a richer scene description at its lower level.
• Background subtraction. It was organised the components of this module into three categories; Foreground mask, Input-frame and Background model. This module was focused on fine-tune the parameterised functions in charge of handling aspects such as the static modelling of the scene or the filtering operations performed over the source images and resulting masks.
• Shadow handling. During the foreground object segmentation, shadows are often a difficult issue to deal with. Distinguishing moving objects from shadows is a vital task because shadows can cause various problems such as shape distortion, phantoms, over/under-segmentation, false crowding, etc. The outcome from this stage is a refined copy of the foreground saliency mask obtained in the previous step, in which the regions corresponding to shadow areas are masked and merged with the negative area of the mask. In our implementation, it was chosen a colour space with better ability for splitting between chromaticity and intensity values than the RGB is capable to. As described, we selected the HSV colour space based on the previous results from Rita Cucchiara, taking advantage of the fact that HSV colour space provides separation between chromaticity and luminosity by means of how the colour model is designed. This method has been widely adopted in video-based surveillance applications, especially in those that involve segmentation for detection purposes. The main reason for choosing this solution is a good compromise between performance and computational cost. As main disadvantage of the method, the value comparisons are made at the pixel-level, which makes it sensitive to image noise. Pre and post-processing performed as part of the background subtraction stage contributes to mitigate this effect, as most of the image and mask noise is reduced by applying different filtering operations.
• Region of interest (ROI) extraction. As a result of a preliminary testing phase of our implementation for this approach, it was observed that the computational complexity of the method would cause an important bottleneck, penalizing the performance of the entire pipeline. Consequently, project worked towards the idea of minimizing the computation required for each image frame, reducing the required amount of computations from the pixel-level to the region-level. This strategy required us to implement spatio-temporal consistency mechanisms for ensuring that only the actual salient regions stand out. This improvement is very useful, especially in situations involving sudden motions, crowded environments or scenes with multiple sources of relevant action
• Person detection. This component was practically implemented during the last development phase. Initially, a person detection algorithm, based on Dollàr’s approximation was considered. Later, we discovered that this method would not be suitable for our global approach, because this technique assumes fixed person sizes. Additionally, the person to be detected should be represented by around the 50% of the image height. These assumptions were not valid for the project scenarios, so we explored a number of alternatives, but finally we decided to go with a multi-scale oriented gradients based approach. The algorithm implemented for person detection is mainly described by two sequential phases. In the first one, the feature vectors representing a collection of descriptors, known as Histograms of Oriented Gradients (HOG), are extracted from a detection window of a particular size. Then, the feature vectors are computed by a linear SVM for obtaining a binary person/non-person classification.
• Low-level features computation. Once the previously described modules were developed and validated, according to the requirements listed by final consumer for the action recognition module (WP5), the first functional implementation of this module, known as Dense Trajectories, was carried out. However, it was found several disadvantages that could affect the system performance, mainly due to computationally-demanding processes, such as computing the optical flow for the image sequences. For this, it was implemented an alternative method (known as Motion Vectors) for extracting similar feature descriptors from the scene, as we did using the previous method. In this case, the optical flow was replaced by the motion information encoded as part of the compression scheme implemented by most codecs designed for digital video streaming. Both MBHx and MBHy descriptors shown very similar values in both approaches, but, by means of the motion vectors-based one, the overall process is around two orders of magnitude faster than it was using optical flow.
The following diagram shows an overview of the individual elements composing the final implementation of the low-level scene description pipeline.
Figure 5. Overview of the final software packages of the low-level scene understanding pipeline. For each package, its top-level modules are also shown.
Finally, the core challenge of the project is related with the capacity of the system to develop a Criminal Activity Understanding module for petty crimes.
In this regard, the main objective of this module is to model criminal activities and normal activities in the same scene in order to robustly detect and predict the occurrence of the former.
Figure 6. A screen shot of the GUI of the SmartPrevent criminal activity detection module.
SmartPrevent has developed a state-of-the-art sliding window based detector, in order to deal with the scale and pose changes in activities captured in a real-world environment. The result shows that our developed method is very competitive against the state-of-the-arts. However the initial activity detection module required sufficient training data for model learning, which limits the fast deployment of the module for new visual scenes/environments. In the refined software module, this limitation is overcome by learning from human feedback based on active learning and transfer learning from source environments where labelled samples are available to target environments where no data annotation is required. These two new capabilities have been integrated into the criminal activity detection software modules and evaluated on benchmark datasets to validate their effectiveness.
Since the SmartPrevent system is to be used by a human operator who will make judgement on whether the alert generated by the system is genuine of false, it was critical that the system can learn from the human feedback and improve itself automatically and on-the-fly, so that similar mistakes can be avoided in the future. An active learning module was developed so that our activity detection model can be improved whenever human feedback is available. Our focus was on the active training sample selection criterion: assuming that getting human feedback incurs a cost because human operators’ time is limited, this criterion is used to select the most informative samples for getting human feedback so that the resulting improvement of the detection model is the most pronounced.
Project has obtained an optimised model by using as few data labels as possible. To that end, it was adopted the common uncertainty sampling strategy. More specifically, it was utilised the entropy-based active query selection algorithm, which is a natural measurement of informativeness.
Regarding the classification model, in the SmartPrevent criminal activity detection module, it was adopted the common Support Vector Machine (SVM) model as our binary classifier which gives a yes or no answer to a given video clip regarding whether it contains an instance of one type of criminal activity. It should be noted that this choice is in- dependent from our active learning method.
In terms of transfer learning method, project has presented a domain adaptation method for more effectively and rapidly exploiting a pre-trained model learning from one domain but deployed to a new unseen domain. This model allows to overcome the inherent domain shift challenge between different scenes by aligning the subspaces of two domains in a supervised fashion. This function is practically useful and valuable since it is not scalable to collect a sufficiently large set of training data for any target domains/scenes in reality, which can be over costly. It was demonstrated the efficacy of the proposed domain adaptation method on a realistic graffiti data collected in the project. In particular, the detection accuracy of a model learned in one domain is clearly improved when the model is applied to a different domain, after the unsupervised domain adaptation method is applied to align the two domains.
Finally, we have also developed a more radical method of transfer learning based on zero-shot learning. By this method, one could detect an activity which one only reads about but has never seen any visual examples before. Having this model makes the SmartPrevent system more broadly applicable.
Potential Impact:
SmartPrevent project has addressed the main priorities of the Security Research, Call 6, matching perfectly with the topic “Solution for frequent petty crimes that are high impact to local communities and citizens”, considered in the Security Work Programme 2013. For this, some of the principal impacts achieved by the project are aligned with the European strategy in Security domain.
As part of this strategic, innovative low cost solutions should be implemented to reduce crime against local communities, business and citizen. SmartPrevent system is based on current video surveillance system to improve their own intelligence in order to provide early criminal activities detection. First petty crimes selected; graffiti and anti-social behaviour are fully oriented to local governments and citizens. In addition, the system is scalable to further scenarios and petty crimes thanks to the active learning and transfer learning for fast deployment methodologies. The system was validated to demonstrate the positive impact to detect criminal activity as well as to be capable to detect in advance, prevent, the potential petty crimes.
In particular, the impact of SmartPrevent has been significant and large on the academic and industrial, and end-users communities related to crime prevention using video-surveillance systems in general, and specifically in terms of crime prevention in urban scenarios by means of a low-cost system tailor-made to meet the needs of end users.
Our impact has reached the European organizations in terms of:
▪ Providing a set of methodologies for a better use of video-surveillance system in crime prevention and detection, taking into account legal, ethical issues and better technologies, methods and algorithms.
▪ Providing a set of tools based on video-processing of video sequence and enrichment with relevant semantic information of data (tracking, definition of object, relations), so that a better and more rapid response to petty crime in urban scenarios will be enabled.
The extent of our impact outlined above is elaborated further as follows:
a. Providing a set of methodologies for a better use of video-surveillance system in crime prevention and detection.
SmartPrevent has developed several methodologies and recommendations that enables a more rapid response and an early detection of activities related with petty crimes. The work done during data acquisition and ethical issues review will act as starting point for similar project. An innovative approach to crimes prevention and detection proposed by SmartPrevent is that the recognition of suspicious activities can be used, not only to detect crime, but also to prevent it. Computer vision methodologies are able to understand scene configuration and activities that happens in this environments. Finally, the system provides a spatial and temporal adaptability to these criminal activities in order to adapt the system in each local environment in any time. The high dependency of computer vision solution to scenarios and conditions has been an important barriers to deploy smart video surveillance where human factor is only used as supervisor. For this, SmartPrevent is scalable and flexible to different infrastructure and urban scenarios.
b. Providing a set of tools based on video-processing of video sequence and enrichment with relevant semantic information of sequences, so that a better and more rapid response to crime in urban scenarios will be enabled.
The main achievements of SmartPrevent project are based on new methodologies to be used in video surveillance system to detect and prevent petty crimes. In order to strengthen this impact, a set of tools to support methodologies, as well as enabling understanding of video sequence have been developed. As a result, video-surveillance managers will be provided with real-time added-value information for a better and more efficient decision-making process. The project has focused on two main developments to understand the scene where petty crimes could occurs as well as to understand the criminal activity in an urban scenarios where several common activities are also part of the activity understanding. Firstly, a low-level pipeline is implemented to improve the scene understanding and reduce the computational cost of higher levels. This first approach is able to provide an enriched data removing not substantial information like shadows, area without interest to the upper layer where high level algorithms must discover the criminal activity with a low rate of false positive.
Thus, SmartPrevent has addressed the following areas to provide a significant impact at European level:
▪ Project has delivered technological solutions for civil protection, including protection against the risks arisen from crime and terrorist attacks. Our solution is able to be improved from an easy way to add more uses cases and criminal activities to be detected as well.
▪ SmartPrevent is focused on technologies and capabilities to enhance the effectiveness and efficiency of current systems, equipment, tools and processes and methods to improve the security of Europe. The system is interoperable and can be integrated in any Security Control Center where operators can easily to manage the system.
▪ SmartPrevent was designed taking into account the privacy rules and requirements of Law Enforcement Authorities to develop an almost plug and play system without dependency of wherever the system is deployed.
On the other hand, SmartPrevent does not only meet technical impact but social impact is a core activity of the project. At the end, the system is developed to support Police bodies to improve citizen security in urban areas and the system also must be considered as a useful and adequate tool form the citizen perspective taking into consideration privacy right and data protection.
The SmartPrevent project has intended to have a social impact, in that it is meant to identify attempted criminal activity and offer Law Enforcement agencies an affordable tool to better protect citizen against criminal acts. Thus, the project intends to give more security and sense of security to the citizens. The SmartPrevent solution will have a positive social impact by reducing the economic and social cost of petty crimes in urban areas, and this reduction in illegal activity will have significant amplified positive impacts for society and the economy making safer cities and communities. Petty crimes selected by the project have a negative impact regarding societal point of view like the graffiti actions and anti-social behaviour. In addition, a reduction in petty crime increases the sense of public order which produces a reduction of crime in general.
SmartPrevent is also focused on the crime prevention adding active learning methods to understand in advance potential petty crimes. It makes a higher social impact when a system is able to detect potential crime before occurs. The sense of public safety would grow exponentially as more automatic learning is added to the system with the expert advisory of police officers.
Moreover, SmartPrevent implemented in Police Control Center is a new data source and data repository to give an effective information about criminal activities to support police guidelines. This information may be used to improve European Security Policies.
Finally, the impact of SmartPrevent is carried out by a set of dissemination and exploitation actions, whom were led by all the partners involved to enhance a broad audience.
Numerous dissemination events have taken place besides the officially planned ones like the international events in Computer Vision (IEEE AVSS and ICDP). There are additional organizational presentations principally dealing with the relationship between members of the consortium and their respective organizations but also dissemination to the sales organizations and in general all the people dealing with innovation. A detailed list is provided in next section. In addition, the SmartPrevent consortium hosted a User Conference in Las Rozas de Madrid, Spain on 26th April 2016 (http://www.smartprevent.eu/node/56). More than 50 participants attended the conference, mainly from the Spanish end-users as well as other partners of EU projects including LASIE, FORENSOR, P-REACT, ADDPRIV and the host project SmartPrevent. They include delegates from the City Council, Spanish Local Police bodies, academic research groups, SMEs on developing commercial computer vision systems, National Police and representatives of the Ministry of Defence.
The conference was a successful event for the project dissemination and a good opportunity for presenting the SmartPrevent project results to the external users and other relevant EU projects. This final project conference also succeeded in promoting new technologies and methodologies for enhancing existing video surveillance systems designed for the prevention and detection of petty crimes in urban environments.
Regarding exploitation actions, SmartPrevent has defined a twofold approach to promote the project exploitation. Firstly, a general exploitation plan was created to push the potential actions from the whole consortium including the planned exploitation actions and social gains and economic feasibility. In a second way, each partner with interest in exploitation actions has planned and carried out specific actions to exploit the technical foreground and new knowledge adding them to their own portfolio of new solutions and services.
List of Websites:
The SmartPrevent project website (www.smartprevent.eu) was designed and created to not only have a professional look, but also provide an effective media for communicating the project objectives with the scientific and industrial communities in a wider context. More details about the project website and a brochure are given in D7.11 (Project Brochure and Project Website).
It consists of seven sections:
• Home: a summary and the project fiche on the right – full name, topic, duration, consortium – are displayed. There is a small block showing the latest news on the lower right hand side.
• Project description: the objectives are listed and the work packages are briefly described, indicating the outcomes and the leader.
• Consortium: a description of the partners together with their roles and contact information.
• Advisory board: a list of entities involved as members of the Advisory Board Group.
• News and events: up-to-date information about recent news, as well as past and forthcoming events. This section has been continuously modified with the addition of new information.
• Documents: it is made of three categories – public deliverables, publications, and dissemination material (project summary and leaflet). This is the section envisioned to allow the user to download documents and dissemination materials.
• Contact: information to get in touch with the coordinator.
• Login: this is divided into two blocks. The one on the left is intended to add/remove/edit contents in the website and the one on the right gives access to Alfresco.
As mentioned before, the section focused on the news and events displays updated information about the participation of the partners in workshops, sister projects from FP7 and H2020 and other relevant actions in the context of dissemination.
In addition, Social media is a good way of making public advances and results. Even though creating a Twitter account (@smartprevent) was not contemplated at the beginning, the consortium decided to do in order to improve the communication channels of the project. This account was used to disseminate and promote events where project has participated and also to be updated about news and trends related with video surveillance domain, ethical issues and to listen end users on security issues. .
Furthermore, the project had designed a logo during the proposal stage that was used for the whole project execution.
Finally, contact details of each partner is published on Project website in consortium tab.
