Social sensors for secUrity Assessments and Proactive EmeRgencies management

Final Report Summary - SUPER (Social sensors for secUrity Assessments and Proactive EmeRgencies management)

Executive Summary:
The SUPER project is a research and development effort that aims to create tools and technologies to support emergency management better utilise social media. In terms of outcomes, SUPER can be considered to have produced three main types of outcomes, namely: component outcomes (modules or tools that perform some function to aid in emergency management); prototype platform outcomes (the prototype software framework that provides data ingestion, storage and visualisation), and research support outcomes (e.g. datasets and initiatives that build on SUPER).
Project Context and Objectives:
The evolution of social networks in the recent years has been very fast, resulting in large user-bases comprised of hundreds of millions of content consumers and producers. Moreover, smartphones are now the most common personal devices, which are used by people of all ages and social positions. The enhanced capabilities of mobile networks and more pervasive market campaigns have changed a phone from a simple communication device between people, to mass communication devices. Social media beneficiated the increase of users as main general interconnection means between groups. Social media platforms have also outplaced many communication mediums, becoming the go-to place where people discuss topics of interest. The social impact of social media has reached a level that contends with traditional mass media as the first place as information media for a large group of people.
In recent emergencies that involve a large number of people, more direct information from involved people came from social networks. People moving to safe places use social media to provide information from their experiences and their view of the emergency. The possibility to post images and video in addition to text has given the ability to everyone to be a news reporter, and traditional medias sometimes pick information and images from social media as an information source to use in conjunction with traditional channels.
Over the last three years this evolution, predicted during project design, has exceeded expectations. Social media have become an important communication medium for society when an emergency occurs. Furthermore, emergency management organizations are now beginning to see the need to monitor and make use of social media, but are generally not yet at a technical competency level to do so.
SUPER is a joint effort of social media experts (incl. social network providers) and security experts to analyze the information needs of emergency management organizations with respect to social media, research technologies that may help satisfy these information needs and develop a holistic, integrated and privacy-friendly approach to the use of social media during emergencies. More precisely, the SUPER project aimed to develop technologies to demonstrate the possibilities of the exploitation of social media for emergency management for:
• Identifying emergency events as they occur
• Extracting useful, actionable and trustworthy information from social media
• Using social media as a communication medium and source of feedback
However, moving from a world where only a few news sources were the information disseminators to a scenario where people everywhere can report information and access information via a large number of channels and mediums introduces important challenges that needed to be studied. The exploitation of social networks that seemed a naḯve opportunity in a lifesaving activity just a few years ago is now a very difficult task that requires advanced technology to properly collect and analyze information. The capability of reaching a manageable amount of information that provides useful support for informed decision making is more difficult now with the massive use of social media in every moment of life of a multitude of people than what is was just a few years ago. Furthermore, the capability of presenting the information in an effective way, providing a readable view and situation awareness is a complex task, which requires a flexible solution. Moreover, the project had to not only account for the number of users and the technologies and devices that they use, but also an evolution of rules and regulation, as social media platforms are owned by private companies that depend from owners and agreements that involve a complex mix between laws, privacy and intellectual property.

Project Results:
The project was targeting 8 objectives that are reflected in following paragraphs (italics format), which includes feedback from the consortium about its achievement.

Objective 1: To elicit, analyse, understand and document stakeholder’s requirements associated with the exploitation of social media in emergencies. The objective will be to understand how stakeholders can leverage social media information for security operations management and security policy modelling and simulation.

During the first months of the SUPER Project the perspective of stakeholders (civil protection agencies, security forces, police forces, security systems integrators, social media platform providers) have been analysed based on requirements collection modalities (interviews, questionnaires, direct contacts, discussions, surveys). The following stakeholder requirements have been identified:
• SUPER social sensors enhance and complement the existing protocols of security assessment and emergency management in compliance with the national and European legal framework.
• SUPER social sensors have to facilitate bilateral communication between end-user agencies and public, the coordination among agencies involved in emergency management and an intelligence-led management approach.
• SUPER social sensors use big volumes of multi-channel Social Media data in order to provide reliable real-time information on hazard and damage scenarios, including event detection, event-related topical summaries, visual geo-mapping, and detailed insights through data semantic analysis.
• SUPER social sensors have to consider and adapt to the dynamic fields of Social Media-usage and emergency situations through automated processing.
• SUPER social sensors require a middleware system for flexibly integrating Social Media processing components (with plug n’ play integration of different Social Media processing components) into security and crisis management applications, which provides an intuitive platform for the end-user agencies.

Objective 2: To research and devise behavioural models that can reflect citizens’ attitude against emergencies and security events. These models will be derived through advancing background knowledge associated with the behaviour of crowds in physical contexts, as well as with the traumatic stress syndrome. Relevant knowledge from prominent emergency incidents (such as the 9/11 terrorist attack) will be exploited.

During the SUPER project a complete, systematic and analytic review of existing behavioural models associated with crowd attitude before, during and after emergency situations was performed. These findings were then combined with research and theories on communicative patterns in social media.
The above mentioned analysis led to the development of behavioural models which reflect the collective reactions of social media users to emergencies and propose a language independent categorization of user behaviour. The objective of this categorization was to compare user’s user behaviour in social media in the context of emergencies. In order to categorize the behaviour of the users, we classified them into clusters for which we can associate a different dominant behavioural patterns. Models for individual and collective behaviour in this setting will allow the understanding of online social dynamics around an event.

Objective 3: To research, design and implement advanced social networks’ processing algorithms for behavioural analysis (including sentiment analysis and topic based community tracking), as well as for identifying security incidents and emergencies (in nearly real-time). The behavioural analysis components will facilitate the understanding of relevant reactions of the public before, during and after security and emergencies incident.

New algorithms are designed, implemented and used to address the challenges of sentiment analysis, topic community detection before during and after an emergency event. After a long research in textual posts we concluded that the vicinity of n-grams can summarize valuable information to detect the topics that are discussed and the sentiment dispositions of the social network users. The n-grams graph representation model is used as a baseline with different configuration setups for topic communities’ detections and sentiment analysis. These techniques surpass the state of the art methods for the needs of an emergency event management. In addition, topological methods applied to detect communities of users that are not based on textual data, providing a more flexible model that it can work based on the social actions of users. Event detection methods are developed that uses the locality hashing technologies and word-embedding technologies. All these constitute the Super social sensors to be efficient and effective techniques.

Objective 4: To introduce a novel architecture for modular, plug n’ play integration of social networks, «social» sensors, as well as of fusion and reasoning algorithms over these sensors. Along with this architecture, an associated middleware infrastructure will be delivered. The objective is to capitalize on the «social sensor» metaphor in order to leverage the vast on-going work in the area of internet-of-things platforms and associated integration mechanisms.

Has been defined the Processing Components Network (PCN) topology and of SM processing components orchestration solution for the SUPER runtime by using a Configuration User Interface.
Mentioned components have been integrated and then refined the definition of the integrated middleware platform design. Additional components that provide secure and reliable communication (Message Bus) was integrated and the respected software components and libraries were developed.
First version has evolved based on feedbacks and provides additional features, notably
a) integration of additional social sensors and the improvement of their description
b) data collection was improved by the adoption of a message bus and the development of client software libraries,
c) the social sensor management was further refined
d) the data storage has been enhanced by additional operation in order to aid the functionality of the components.
Middleware platform facilitated plug n’ play capabilities.

Objective 5: To research and implement tools and techniques for security policies modelling and simulation, which will engage citizens in the process through measuring their interactions in relation to security events and emergencies. Policy simulation techniques will be implemented on the basis of novel applications for virtual spaces including games, debates and role play, which will allow for the elicitation of citizens’ feedback associated with emergencies. Overall, the tools to be implemented will allow for the efficient monitoring and prediction of the impact of security policies and strategies on the citizens (and the society as a whole).

Specific platforms developed and upgraded to gather citizens reactions concerning natural disasters and security events. A real time monitoring of social network pages and groups is provided by Virtual Spaces. +Spaces is also provides the capability to the end users to carried out surveys using polls and debates in order to understand the citizens opinion and reactions in aggregated and summarized representations. These platforms are used in validation scenarios providing a very useful tool of observation, understanding and data analysis of the emergency situation to the Super end users.

Objective 6: To research and provide tools and techniques for ensuring the trustworthiness of social media, through confronting/alleviating efforts for compromising social networks, including malicious rumour spreading and social networks manipulation.

During the SUPER project information trustworthiness was tackled from three main directions. Initially, two studies into information trustworthiness were undertaken with the aim of determining where rumours and false information are prevalent and how to identify them, one based on past emergency events such as terrorist attacks (D4.1) and one aligned with the SUPER end-user scenarios (D4.7). We found that rumors are in-fact quite rare occurrences during emergencies. Information disseminated during natural disasters is very trustworthy, as there is no gain to spreading false information. However, for more politically charged emergencies, such as terrorist attacks, rumors are still not common, but are more prevalent. We also observed that rumours tend to be spread via reply chains by groups of disenfranchised users. Hence, to help our users find these rumours, we implemented a discussion thread tracking technology for Twitter (D4.5). Moreover, automatic techniques to identify rumours was developed and evaluated. An automatic rumour post identification approach was developed that achieved between ~0.65 F1 and ~0.75 F1 effectiveness, depending on the event type (D4.1). Meanwhile, an approach for identifying fake images reposted from past events was also developed, based on reverse image search technology (D4.1). Additionally, we also developed a semi-automated approach that makes use of crowd workers to identify rumors quickly and with high agreement (D4.2).

Objective 7: To research techniques and a framework for the quantitative assessment of information stemming from on-line social networks and social media. To this end the project will research metrics of credibility and accuracy of the relevant information. On the basis of these metrics the project will introduce a framework for the quantitative assessment of the trustworthiness of social media infrastructures and applications.

To support quantitative assessment of information credibility and accuracy SUPER investigated the factors that make information credible during emergencies and developed technologies to automatically estimate the credibility of social media content. Through analysis of past emergency events and related works, we identified a taxonomy of credibility indicators, including textual features, author credibility, known users and references to supporting material (D4.2). Based on these indicators, we developed an automatic approach to estimate the credibility of individual social media posts in real-time (D4.2). Furthermore, through evaluation over three emergency events, we showed that this approach can accurately predict post credibility for up-to 82% of posts (D4.3). Credibility estimation is integrated into both the search and discussion thread tracking tools (D4.7).

Objective 8: To integrate the above-mentioned tools and techniques within crises management and security management systems support policy simulation, COP generation, real-time operations management, as well as both tactical and strategic planning. The integration of the various tools will result in the integrated ICT based SUPER platform, which will empower the SUPER approach to exploiting social media analysis for security and emergencies management.

Common operational Picture provide a full responding interface to the End User. It is a real time operation management interface to the platform. Through the COP is provided the access mean to all components information and the possibility, also, to access main platform management configuration interface. Through the COP all technologies provide their functionalities, providing the ability to manage information during the hot phases of emergencies and also to analyse information after events in order to decide more long term actions in a strategic view.

Objective 9: To validate and evaluate the SUPER approach in the scope of two realistic use cases concerning the use of social media in the scope earthquakes, as well as exploitation of social media for police activities. As part of this objective the project will validate that social media processing tools could become an integral part of contemporary crises management systems and strategies.

Two real pilot cases have been performed within the project, one pilot scenario was addressing emergency situation in the scope of civil protection (Italy, led by CEFU) and another one in the scope of police activities (Romania, led by IGPR). The pilots and validation results have been documented in the corresponding WP7 deliverables. The combination of the two pilot scenarios exercised all different functionalities of the overall SUPER framework. During the pilot interested external stakeholders assisting the exercise did provide valuable technical and non-technical debates and feedback to the consortium with regards of the use of social media information for crisis management. Booth, internal conclusions and external feedback have been included in deliverables submitted.

Potential Impact:
In this section, we summarize the these three types of outcomes from a high level perspective.

We divide this section into three sub-sections, one per type of outcome. Section 1 discusses the component outcomes, i.e. each of the social sensors and services that were developed to enrich social media streams and enable the end users to effectively and efficiently explore those streams. For each component we summarize what it does, why it is useful for emergency management and highlight how it expands over the state of the art. In Section 2, we describe the prototype platform outcomes. Here, we start by describing the architecture of the prototype framework developed for demonstrating the SUPER’s component outcomes, and then describe the modules of that framework that were implemented and deployed (i.e. modules for social media ingestion, data storage in the middleware and the platform GUI). Finally, Section 3 discusses research support outcomes in terms of datasets created for the community and initiatives that have resulted from SUPER.
1 Component Outcome
The first type of outcome of SUPER are the component outcomes. These are high-level software social sensors and services that ingest social media streams and either augments that stream with additional annotations (e.g. tagging sentiment or credibility) or provides a service on-top of that stream (e.g. search or direct communication with users). These social sensors and services were originally proposed as research directions in the SUPER Description Of Work. Over the course of SUPER, implementations of these social sensors and services were realized and tested to determine to what extent they were useful for supporting emergency management.

For clarity, we structure our discussion of the social sensors/services around when during an emergency these sensors/services were envisaged to be valuable. The three stages of an emergency we consider are: identification at the start of the event; response during the event; and recovery after the event.

2 Prototype Platform Outcomes
The above component outcomes are have software implementations that provide individual pieces of functionality. However, to be deployed during an event, they each need to be integrated into a wider framework that provides the social data to be processed and serves their output to the user in a usable manner. As such, the SUPER project developed a prototype framework to enable the deployment of the different component outcomes, referred to as the SUPER Framework. This framework itself is a source of project outcomes associated to the different modules that comprise it. In this section we describe these outcomes.

This section is divided into five sub-sections, which each discuss different aspects of the framework. In particular, Section 2.1 provides an overview of the SUPER architecture and data-flow within it from social media source to the user. In Section 2.2 we describe the outcomes associated to social media ingestion, i.e. how social media content is collected by the framework and formatted. Section 2.3 discusses outcomes from the development of a modular plug-and-play integration system for the different component outcomes described earlier. In Section 2.4 we describe outcomes related to data persistence within the framework and their implementation as the SUPER Middleware. Finally, we summarize the visualisation capabilities developed for SUPER within the C2HMI platform in Section 2.5.

2.1 Framework Architecture
The SUPER Architecture provides the structuring principles for the integration of social networks, social sensors and emergency management applications.The architecture comprises four different layers and clusters the various components according to these layers. These four layers, which are depicted as horizontal block in the architectural diagram (Figure 1), are described below:
● The Social Media layer, where the actual social media services e.g. Twitter, Facebook, Instagram, etc. (see lower block in Figure 1) reside.
● The Social Media Processing layer that hosts two types of components. First, the data retrieval components, i.e. the SN API and the Social Media Crawler, which connect the social media services to the SUPER framework Second, the SUPER processing components that are responsible for filtering and analysing the captured data (e.g. sentiment analysis or rumour identification) with respect to specific events. The latter form the Processing Components Network (PCN).
● The Social Sensors Data Management layer, where all of the collected data from the different deployed processing algorithms will be stored in a centralized database. This layer is responsible for providing information access and discovery of processing components (i.e. discovery of the available SUPER platform functionalities).
● The Common Operation Picture (COP), which acts as the SUPER platform user interface that will enable end-users to retrieve the requested data in an intuitive manner.

Implementation wise, the SUPER Architecture is realized in WP3, WP4, WP5 and WP6. On the one hand, the work completed in WP3 and WP4 produced the various Social Sensors (i.e. Behavior & Sentiment Analysis, Rumor Identification and Spreading Framework, Topic-based Community Detection etc.) and along with the components and the integration effort in WP5 form of the SUPER Middleware. The role of the SUPER Middleware is managing the social sensors and to gather the social media data in the Centralized Data Storage providing the services to build end user applications.
The initial Social Media Data Acquisition is performed by the social media crawlers and adaptors. The role the social media crawlers and adapters is to tap into existing social media networks, gather real-time data from them, and further, upon request, provide this stream of real-time social media to other components higher-up in the SUPER architecture (see also Figure below).

2.2 Social Media Ingestion
The detailed Data Flow follows a direction from the Social Media Layer to the social Media Processing Layer and detailed flow is shown in the figure below. The lower part are the Social Network crawlers/adapters that are responsible for creating appropriate interfaces to retrieve information from the Social Media data streams.
The Social Sensor presented in 1 are responsible for processing and annotating the incoming data streams. The software implementations of each sensor are referred to as Social Media Processing Components (SMPCs). Data Persistence Component that performs the Data Collection (using a Message Bus) and persists the annotated incoming data streams to the Centralized data storage (Data Persistence) . The Message Bus used in the PoC implementation is RabbitMQ.

Each social sensor makes use of three application programming interfaces (APIs) for operation: (i) Configuration API, (ii) Persistence API and (iii) Crawler Client API.
The social sensor pushes its processed/enriched output to a RabbitMQ mailbox in JSON format using the Persistence API. Invisible to the social sensor itself, the content of this mailbox are read by middleware Data Persistence component for subsequent reformatting and storage.
In addition to the social sensors discussed above, there is a second class of components defined within the SUPER platform, known as Social Media Processing Services. The reason for the distinction between social sensors and social media processing services is that there exist functionalities that require real-time processing of social media content for an event of interest, but instead of continuously producing output to be stored in the middleware, they provide some user-facing service. For instance, integrated search is an example of such as service. Here, social media content needs to be indexed in real-time, however the output is a service enabling the user to issue search requests.

2.3 Plug-And-Play Social Sensor/Service Integration
The SUPER Middleware Platform integrates the developed within the project social algorithms, which are implemented as autonomous components called in the SUPER terminology SMPCs (Social Media Processing Components with the SUPER Middleware Platform that persists the received social media data and provides services over it.
The plug&play aspect of the framework is achieved through the common model and semantics that all SMPCs should follow, which was presented in deliverable D5.2 and the API that each implementation of an SMPC must follow. This API dictates that the SMPC published the metadata and the configuration options of the SMPC. The sequence diagram is shown in the figure below.

In the above interaction, the SMPC, which encapsulates a specific social sensor algorithm in the form of component, publishes the SMPC information in the Centralized Data Storage, upon component initialization, using the Processing Component Service. The same sequence of operations is executed in the update and delete operations of the Processing Component Service.
The SMPC info follows, as mentioned above, the common model and the semantics of the SUPER model. Thus, the integration of new algorithms, provided they are implemented as SMPCs and respect the APIs, can be achieved without any configuration effort to the SUPER plug&play middleware platform.

2.4 SUPER Middleware
SUPER has developed within WP5 a novel plug&play Middleware, which collects, processes and manages information flows from social networks as a means of deriving the context of the emergency and accordingly provides services to end-users. The final version of the framework has been described in detail in deliverable D5.6 .
The SUPER Middleware comprises four different layers and clusters the various components according to these layers. These layers are (from bottom to top): 1) the social media layer, 2) the social media processing layer, 3) the social sensors management layer and the 4) common operational picture (COP) layer. The SUPER Middleware integrates components developed within WP3, WP4 and WP5.
The SUPER Middleware Platform is the central (middleware) layer of the SUPER architecture and has three main roles in the Integrated SUPER Framework:
● Integrate Different Social Sensors in a plug&play manner.
● Acquire and Store Social Media Data from the Social Sensors
● Provide services so that the Stored Media Data are easily accessible and queryable by the Common Operational Picture.

All the above functionality has been presented in detail in the previous sections for 2. One exception is the support for the Common Operational Picture. This goal is achieved by the Centralized Data Storage Services that the final version was presented in Deliverable D5.6. These services are provided as RESTful Web Services and they provide general and data-centric access to the social media data stored in the Centralized Data Storage.
CDS Services provide operations to get, create, update and delete events and query for social media data (in the form of binary JSON – BSON documents as the data storage is based on NOSQL database MongoDB). The complete list of the operations that are offered by the Centralized Data Storage is provided in section 3.4.3 of the deliverable D5.6 “SUPER Middleware Platform”. The BSON documents follow the modelling of the SUPER Project and could be either Events, Processing Components, Posts and Discussion Threads.

However, the integration with Multi User Visualization Tools of the Common Operational Picture is enable by the Predefined Data Retrieval Services (PDRS). These services in contrast to the CDS services (data-centric), are business-centric and provide the business functionality that the Multi User Visualization Tools (MUVT). They enable the basic functions of querying and data acquisition in order for the MUVTs to populate the User Interface with already captured social media data residing in CDS. The final version of these service is described at section 3.3.2 in deliverable D6.4

2.5 Event Visualisation via the C2HMI
The SUPER Framework provides a set of user interfaces , allowing emergency operators to obtain information insights from the social network information published and ingested and analyzed through the technologies and components described in sections 2.4. The main interface is the Common Operational Picture, which can be used for before, during and after an emergency event occurs.
Except for few tools, and thanks to the middleware architecture of the project, COP encapsulates all the functionalities developed during the project, but its own architecture allows to enrich them using external services, or new social sensors and social media processing services by simply following the APIs defined for its integration.
Apart from COP, the framework contains a tool called +Spaces that will allow policy makers to develop and promote new policies. The tool allows the assessment of new policies by interacting (using social media) with involved people; citizens, emergency operators and managers, by the implementation of experiments.
Finally, not included in the COP, a tool to follow up evolution of any information contained in Wikipedia is provided, thus allowing emergency managers to examine information being generated related to past events.
SUPER Common Operational Picture (COP) is a web based framework for command and control systems. It has been developed as real time interface for decision makers and has extended capabilities for interconnection with multiple systems.
In SUPER Project the COP represent the main interface of the platform. A single access point that allows end users to access all functions, directly from a single web interface. The COP supports both normal access and touch screen access and works as a multi user, multi profile interface allowing users to differentiate access to multiple groups and access levels.

The COP uses as access layer a reverse proxy to provide an easy configuration of access to internal and external services that COP needs to access to fulfil its role.The proxy system allows to manage access to multiple server and to obtain redundancy and high availability with multiple COP system based on name access system. It allows to use multiple COP servers or same server for multiple user entities.
Emergency management organizations use many different tools and infrastructures for environment monitoring, video surveillance systems for critical areas, tracking system for vehicles and many kind of other technical and informational system that are usually managed using different infrastructures and interfaces.
SUPER Common Operational Picture (COP) interface is an open visualization platform open for integration of data source other than SUPER platform. The integration of all data sources is an important value for decision makers because having access to all information using a single interface constitutes a rapid and comfortable support for informed decisions.
● SUPER COP can visualize and interact with Geo-localized information from End Users infrastructures. SUPER COP will provide the interconnection interface to access interactive data sources, and also other data sources can help corroborating SUPER recognized events and vice-versa.
● COP interface is a real-time interface and can visualize all information in push mode. Interactive functions also allow End Users to access other real time sources directly from COP interface, like video feeds from cameras. At the same time, real-time feeds from sensors can be shown on the interface. Environment sensors are the common application for that kind of scenario. For example, Regione Campania has a network of sensors that are integrated on the SUPER COP.
● Other End Users data sources that can be integrated are mapping sources. End Users organization maps, risk maps and other data intensive layers can be used as main or secondary map layer and as informational layer. This last type of data sources is not properly a real-time information but may support decision makers providing awareness or important information on the situation and where SUPER Events and other alarms are occurring.
● Event visualization start from other component or instruments used to detect new events or from pre-defined events. After event detection and that is enables, the operators are able to see it in event list or event search window. The events are presented in map and table view.
● From both map and table view is possible to select and access a specific event. Event visualization use the same visualization approach with map and table view on the same screen.
The COP allow the access to multiple tools for communication through social media, like the twitter client, and also the access to the configuration system (SMPC) or other integrated tools.
COP operation during an on-going emergency requires easy access to information and possibility to reduce workload to decision makers. Hence, the COP will include visualization of the following additional information:
● Different map layers
● Traffic layers
● Position of ground units
● End user’s information infrastructure connection (CCTV, sensors, etc)

3 Research Support Outcomes
In addition to the primary outcomes of SUPER in terms of social sensors/services (component outcomes) and the SUPER framework (platform outcomes), SUPER also produced a series of other outcomes that will support future research into emergency management using social media. In particular, we break down these supporting outcomes into two main types: dataset outcomes and research initiatives. Dataset outcomes are (as their name suggests) collections of (labelled) social media data that future researchers can use to train and test future algorithms and techniques for particular tasks investigated during SUPER (such as sentiment analysis or credibility estimation). Meanwhile, research initiatives are other external projects that were directly supported as part of SUPER, or which resulted from research from within SUPER. In Section 3.1 we summarise the datasets that were created during SUPER, while we discuss the research initiatives stemming from SUPER in Section 3.2.

3.1 Datasets
Datasets form the basis that modern research is built on. They provide a common scenario and ground truth under which systems of different types can be compared, as well as providing training examples that systems that utilize supervised learning can build on. For the scenarios that SUPER is concerned with, a dataset is typically a series (stream) of (social media) documents from a particular time period, with human annotated labels categorizing those documents. For example, if evaluating real-time summarization of social media during an emergency event, an associated dataset would contain a stream of social media documents to be summarized and ground truth labels for those documents specifying how suitable each of those documents are for inclusion into a summary.

Evaluation datasets were created during SUPER to evaluate many of the social sensors/services (component outcomes) developed. We provide an overview of these datasets in the table below:

Task Deliverable Description
Sentiment Analysis D3.2 These are three Twitter datasets related to different crisis-related events: the 2012 Aurora shooting, the 2012 Hurricane Isaac and the 2015 Ebro River flood.
These datasets cover two different types of crises, human-induced and natural disasters, and two languages, English and Spanish. The tweets have their subject labelled as well as any sentiment expressed (either neutral, positive or negative).
Community Identification D3.6 This is an extension to the dataset described above covering the 2012 Aurora shooting, the 2012 Hurricane Isaac and the 2015 Ebro River flood. Retrospectively for each event, general topics of discussion were identified by analysing other sources of reporting, such as associated news articles or the Wikipedia page for the event, e.g. “counts of killed/injured”, “government reaction” and “shooter mental instability”. Human annotators then manually searched the corpus of tweets finding and tagging tweets associated to each topic, creating gold standard topical communities.
Rumours D4.2 This dataset is comprised of Twitter tweets about the democracy protests that took place in Hong Kong at the end of October 2014, tweets about the extreme levels of snowfall that occurred in Buffalo, USA during November 2014; and tweets discussing the civil unrest that occurred as a result of the court verdict relating to the shooting of Michael Brown in late November 2014. Human assessors manually analyse a sample of tweets from each event, classifying them into one of 8 classes, where five of those classes represent different types of rumour.
Post Credibility D4.3 This is an extension to the Community dataset described above covering the 2012 Aurora shooting, the 2012 Hurricane Isaac and the 2015 Ebro River flood. Human assessors manually annotated the credibility of posts for each topic, assigning binary credible or not credible labels to each post.
Search D4.7 We collected six datasets corresponding to events that have different sizes (number of posts total) and burst rates (maximum number of posts per minute), ranging from around 1000 posts to over 2.4 million posts.
Trust, Sentiment Credibility D4.7 As part of pattern analysis in T4.4 we also collected a dataset comprised of Tweets and Facebook posts for the 2015 Colectiv nightclub fire. Posts collected were labelled by a police information specialist based on whether they trusted the information contained, how credible that information was, as well as sentiment expressed.

3.2 Research Initiatives
Beyond providing datasets for future researchers, SUPER also supported a number of research initiatives that aim to bring together researchers from around the world to work on tasks related to social media and emergency management. More precisely, within the duration of SUPER, the University of Glasgow partner co-organised two such initiatives under the auspices of the National Institute of Standards and Technology (NIST, US), as part of its Text Retrieval Conference (TREC) series. Furthermore, this has lead to the funding of new research initiative by NIST’s Public Safety Communications Research Division that will take place in 2018. These initiatives promote the scientific and reproducible evaluation of information retrieval systems and advance research by providing test collections/benchmarking tools and standardising the evaluation methodologies. We summarize these three research initiatives below:

TREC Temporal Summarization Initiative (2013-2015, http://www.trec-ts.org/): TREC-TS aimed to support the research and development of systems for efficiently monitoring the information associated with an disaster events over time. Specifically, it examined challenges in developing systems which can broadcast short, relevant, and reliable sentence-length updates about a developing event to the user. The TREC-TS initiative brought together 13 research groups from US, Europe and China to work on the problem of summarizing events over time, as well as produced reusable evaluation datasets and metrics that are in use today. The TREC-TS initiative was also used to evaluate the real-time summarization component of SUPER (see D3.3).

TREC Real-time Summarization Initiative (2016-Ongoing, http://trecrts.github.io/): TREC-RTS aims to enable research into systems that automatically monitor the stream of documents to keep the user up to date on topics of interest (defined by user profiles). System output can either be push notifications to the user when something important happens or an email digest at the end of each day. The 2016 edition of TREC-RTS brought together 24 research groups from 10 countries to work on this problem.

TREC Incident Streams Initiative (2018): TREC-IS is a new initiative funded by NIST’s Public Safety Communications Research Division and organised by the University of Glasgow partner that aims to promote state-of-the-art research into tooling to better support response services harness social media during emergencies. In particular, it will develop a test collection and evaluation methodology for automatic and semi-automatic filtering approaches that aim to identify and categorize information and aid requests made on social media during crisis situations. This will support the advancement the technology readiness level (TRL) of current social media crisis monitoring solutions and better support social media monitoring by PIOs and other stakeholders in the future. The TREC-IS task is to produce a series of curated feeds containing social media posts, where each feed corresponds to a particular type of information request, aid request, or report containing a particular type of information. For instance, for a flash flooding event, feeds might include, ‘requests for food/water’, ‘reports of road blockages’, and ‘evacuation requests’.

The project results have been widely disseminated at numerous international academic conferences and published as peer-reviewed papers in renowned publications:
1. John Violos, Konstantinos Tserpes, Athanasios Papaoikonomou, Magdalini Kardara, and Theodora Varvarigou. 2014. Clustering Documents using the 3-Gram Graph Representation Model. In Proceedings of the 18th Panhellenic Conference on Informatics (PCI '14). ACM, New York, NY, USA, , Article 29 , 5 pages. DOI: http://dx.doi.org/10.1145/2645791.2645812
2. Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2014. Incremental Update Summarization: Adaptive Sentence Selection based on Prevalence and Novelty. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM '14). ACM, New York, NY, USA, 301-310. DOI=http://dx.doi.org/10.1145/2661829.2661951
3. García-Gavilanes R., Kaltenbrunner A., Sáez-Trumper D., Baeza-Yates R., Aragón P., Laniado D. (2014) Who Are My Audiences? A Study of the Evolution of Target Audiences in Microblogs. In: Aiello L.M. McFarland D. (eds) Social Informatics. SocInfo 2014. Lecture Notes in Computer Science, vol 8851. Springer, Cham
4. Richard McCreadie, Saul Vargas, Craig MacDonald, Iadh Ounis, Stuart Mackie, Jarana Manotumruksa, Graham McDonald: University of Glasgow at TREC 2015: Experiments with Terrier in Contextual Suggestion, Temporal Summarisation and Dynamic Domain Tracks. TREC 2015.
5. Richard McCreadie, Karolin Kappler, Andreas Kaltenbrunner, Magdalini Kardara, Craig Macdonald, John Soldatos, and Iadh Ounis. 2015. SUPER: Towards the use of Social Sensors for Security Assessments and Proactive Management of Emergencies. In Proceedings of the 24th International Conference on World Wide Web (WWW '15 Companion). ACM, New York, NY, USA, 1217-1220. DOI: http://dx.doi.org/10.1145/2740908.2741723
6. Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2015. Crowdsourced Rumour Identification During Emergencies. In Proceedings of the 24th International Conference on World Wide Web (WWW '15 Companion). ACM, New York, NY, USA, 965-970. DOI: http://dx.doi.org/10.1145/2740908.2742573
7. Janani Kalyanam, Amin Mantrach, Diego Saez-Trumper, Hossein Vahabi, and Gert Lanckriet. 2015. Leveraging Social Context for Modeling Topic Evolution. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '15). ACM, New York, NY, USA, 517-526. DOI: http://dx.doi.org/10.1145/2783258.2783319
8. Athanasios Papaoikonomou, Magdalini Kardara, and Theodora Varvarigou. 2015. Trust Inference in Online Social Networks. In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015 (ASONAM '15), Jian Pei, Fabrizio Silvestri, and Jie Tang (Eds.). ACM, New York, NY, USA, 600-604. DOI: http://dx.doi.org/10.1145/2808797.2809418
9. Freire, Ana, Manca, Matteo, Saez-Trumper, Diego, Laniado, David, Bordino, Ilaria, Gullo, Francesco, AND Kaltenbrunner, Andreas. "Graph-Based Breaking News Detection on Wikipedia" International AAAI Conference on Web and Social Media 2016. Available at: https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/view/13198.
10. John Violos, Konstantinos Tserpes, Evangelos Psomakelis, Konstantinos Psychas, and Theodora Varvarigou. 2016. Sentiment Analysis using Word-Graphs. In Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics (WIMS '16), Rajendra Akerkar, Michel Plantié, Sylvie Ranwez, Sébastien Harispe, Anne Laurent, Patrice Bellot, Jacky Montmain, and François Trousset (Eds.). ACM, New York, NY, USA, , Article 22 , 9 pages. DOI: https://doi.org/10.1145/2912845.2912863
11. Comparing Overall and Targeted Sentiments in Social Media during Crises. Saúl Vargas, Richard McCreadie, Craig Macdonald, Iadh Ounis. In Proceedings of 10th international AAAI conference on Web and social media (ICWSM) 2016.
12. Sean Moran, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2016. Enhancing First Story Detection using Word Embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR '16). ACM, New York, NY, USA, 821-824. DOI: https://doi.org/10.1145/2911451.2914719
13. Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2016. EAIMS: Emergency Analysis Identification and Management System. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR '16). ACM, New York, NY, USA, 1101-1104. DOI: https://doi.org/10.1145/2911451.2911460
14. Emergency Identification and Analysis with EAIMS by Richard Mccreadie, Craig Macdonald and Iadh Ounis. In Proceedings of the fourth international workshop on The Social Web for Disaster Management, held at CIKM 2016, Indianapolis, USA. arXiv:1610.04002. https://arxiv.org/abs/1610.04002
15. Mauro Biafore and the SUPER-FP7 Team. Leveraging social media for flood emergency management: an experience in Campania region (southern Italy). Geophysical Research Abstracts Vol. 19, EGU2017-10238-1, 2017 EGU General Assembly 2017 © Author(s) 2017. CC Attribution 3.0 License.
16. Karin Sim Smith, Richard McCreadie, Craig Macdonald and Iadh Ounis. Analyzing Disproportionate Reaction via Comparative Multilingual Targeted Sentiment in Twitter. In Proceedings of Advances in Social Network Analysis and Mining (ASONAM), 2017
For the SUPER project the following journal submissions are currently pending official response or are under preparation:
1. ACM Transactions on Information Systems (TOIS): Explicit Diversification of Event Aspects for Temporal Summarization. Richard McCreadie, Rodrygo Santos, Craig Macdonald, Iadh Ounis. ACM Transactions on Information Systems (http://tois.acm.org/) is one of the top two journals of the Information Sciences field.
2. Journal of the Association for Information Science and Technology (JASIST): Comparing Overall and Targeted Sentiments in Social Media during Crises. Saul Vargas, Richard McCreadie, Craig Macdonald, Iadh Ounis. JASIST JASIST is the other of the two top journals within the Information Sciences field. (http://onlinelibrary.wiley.com/journal/10.1002/%28ISSN%292330-1643)
3. IEEE Intelligent Systems Magazine: Specified Event detection using the N-Gram-Graphs representation model. John Violos, Konstantinos Tserpes, Evangelos Psomakelis, Konstantinos Psychas, and Theodora Varvarigou. (http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=9670)
10.2 Industry workshops and stakeholder events
Apart from the research dissemination at scientific conferences and publications the SUPER consortium has been very active engaging with industry and end-user stakeholders by participating or organizing specific events.
• Stand at Eurosatory 2014, 20th June 2014, Paris (France)
• Stand at NCT CBRNe Europe, 2-3 September, 2014 – Leipzig (Germany)
• Attending PSOPHIA Final event, 16th September, 2014 – Madrid (Spain)
• Stand at Broadband World Forum 2014, 21-23/10/2014, Amsterdam (Nederlands)
• Presentation and Demo at SICSA DEMOfest 2014, 30th of October 2014, Edinburgh, UK.
• Attending REWARD Final workshop, 23-24/10/2014, Naples (Italy)
• Stand at IoT Solutions World Congress 2015, 16-18/09/2015, Barcelona (Spain)
• Poster at Researchers Night, 25th September, 2015, Athens, Greece
• Stand at European Utility Week, 03-05/11/2015, Vienna, Austria
• Presentation and demo at SICSA DemoFest 2015, November 19th, 2015 in Edinburgh, UK
• Stand and Mobile World Congress, 22-25 February, 2016, Barcelona, Spain
• Stand at South Summit, 5-7 October, 2016, Madrid, Spain
• Stand at IoT solutions Worlds Congress, 25-27 October, 2016, Barcelona, Spain
• Presentation and Demo at SICSA DemoFest 2016, November 11th, 2016 in Glasgow, UK
• Stand at Smart City Expo, 15-17 November,2016, Barcelona, Spain
• Stand at World Congress, 27 February - 2 March 2017, Barcelona, Spain.
• Presentation at Glasgow Information Festival 2017, 6 April 2017, Glasgow, UK
• Keynote presentation at Social Media for Emergency Response and Preparedness workshop, co-located with ECIR 2017 (European Conference on Information Retrieval), 9 April 2017, Aberdeen, UK
The project has also been picked up by the research*eu results magazine. Together with the editorial team of the magazine, an article entitled “Mastering the power of social media for better emergency response” has been prepared. Its online version has been published on CORDIS (http://cordis.europa.eu/result/rcn/197459_en.html) and the print version will be published in the upcoming issue.
End-users have been closely involved in the developments of the projects. SUPER has organized a number of events involving the End-User Advisory Board (EUAB) and other relevant stakeholders in order to present and discuss the current developments.
• EUAB Board Meeting, November 21st, 2014, Rome, Italy
• External Stakeholder Conference, February 26th, Sorrento, Italy
• EUAB Board Meeting, June 10th, 2016, Rome
• Internal Stakeholder Meeting, June 4th-8th, 2016 Brasov, Romania
• SUPER Seminars, November 16th, 2016, Sorrento, Italy
• EUAB Board Meeting, November 24th 2016, Naples, Italy
• Public Stakeholder Conference, November 24th, 2016, Sorrento, Italy
• Stakeholder Conference, March 3rd, 2017, Bucharest, Romania

List of Websites:
http://www.super-fp7.eu

Final Report Summary - SUPER (Social sensors for secUrity Assessments and Proactive EmeRgencies management)

Diese Seite teilen

Herunterladen