Final Report Summary - SLANDAIL (Security System for language and image analysis)
Executive Summary:
Citizens regularly communicate through social media computing systems, particularly in times of rapid upheaval. During major emergencies or events the sharing of information (for example, infrastructural warnings, details of threat levels, and transportation advice) becomes an urgent necessity. As a result, a large volume of information relevant to emergency response is being generated in real time across social computing networks in a variety of new forms.
These developments mean that hierarchical models of communication between institutions and citizens are being challenged, with information now flowing in multiple directions through what have become known as “backchannel communications” (Cardoso 2008, Lindsay 2016). In situations of natural disaster, the scale and speed of events can threaten to overwhelm the capacities of emergency managers. By exploring the potential of modern computing systems to incorporate real-time feedback from the public, and by viewing the citizenry as a powerful, self-organizing, and collectively intelligent force (as Palen, Anderson et al. 2010 have influentially argued we should), we can add new tools to those currently available in disaster response. However, technical constraints as well as the complex ethical and legal issues surrounding data collection mean that emergency managers are currently unable to aggregate and systematically utilise this information.
SLANDAIL navigated the complexity of computing, ethics, the law and communications to create an aggregated and integrated system. Operational emergency managers were introduced to legal and ethical principles, while academic researchers developed modules in tandem with SMEs working in data analytics, emergency management and communications. The complexity of roles of active partners was carefully managed throughout the project, and disparate areas of research were carefully combined to create a working Emergency Management System (Figure 2).
At the time of SLANDAIL’s inception, the integration of social media analysis into disaster response processes was rare. SLANDAIL addressed this problem by equipping emergency managers with analytical tools to enhance response and planning capabilities during natural disasters, as well as with communications guidelines and legal and ethical principles for harvesting data and protecting citizens. The project combined expertise and resources in social media, security, ethics, law, emergency management, communication, text analytics, and image analytics, to deliver a multi-disciplinary solution to a challenging, real-world problem.
The system uses location-based geospatial information combined with processed social media data. The overall objective, in the context of security under the FP7 Framework, is for the project to help increase the security of individuals and groups living in areas affected by natural disasters by adding social media to sources of information available to emergency managers.
The project has been designed in response to requirements drawn up by control room teams that manage disasters and emergency events. End Users of the system have been closely involved in the design, testing and evaluation of each strand of the project. SLANDAIL partners collaborated to build and test a prototype system with due regard to ethical and factual data provenance. The SLANDAIL System aims to remove the burden of search and interpretation of the latent information contained in social media data from End Users.
Finally, SLANDAIL followed a Commercialisation and Exploitation Plan to further develop research outputs.
2.2. Glossary
Title Definition
SLANDAIL System An integrated system using emergency management and social media
SLANDAIL Social Media Monitor A platform that monitors social media live using advanced text and image analysis tools developed through project research
SLANDAIL Licence Agreement A licence agreement for use of social media data in emergency management systems that takes into account research in four areas of law: Data protection, copyright, IP and Human Rights
SLANDAIL End Users Emergency management teams and control room operators, from police and army to civil protection agencies, with operational experience in managing major natural and man-made disasters
2.3. Objectives
# Objective Achievement
1 COLLATE and REVIEW the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property Emergency Managers were involved throughout system design and testing. Research completed in the first six months outlined the usability requirements for End User beneficiaries. Test data collated as part of specific emergency related case studies was used to perform system tests.
2 CREATE protocols to protect the rights of the citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens The project created an ethical framework for using data in an emergency and followed this with preliminary legal guidelines and recommendations. This culminated in the development of an innovative model licence agreement for a social media analysis system for disaster response.
3 BUILD AND TEST a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management Built and tested a working prototype that combines emergency management systems with social media. As the project developed, state-of-the-art methods (text, image, speech/non-verbal communication) were introduced, tested and evaluated, leading to the creation and testing of an integrated system.
2.4. Achievements
Approaches to exploiting social computing systems to enhance the management of public crises have been co-evolving organically alongside the media themselves (Hughes and Palen 2009, Shklovski, Burke et al. 2010, Acar and Muraki 2011). SLANDAIL integrates these approaches and methodological reflections and explores how they can best be directed towards the development of a system for using social media in a disaster management context, focusing on improving communication during disasters and addressing the ethical issues involved. The project navigated the complexities of analytics, emergency management, ethics and communication. We successfully deployed, tested and evaluated a Social Media Monitor that analyses social media data in real-time, using advanced analysis in text and image. This was integrated with an Emergency Management System that is supported by a Licence Agreement with communication functionality to enhance End User capabilities.
Text analytics: Existing methods of text mining have been designed for formally written texts. Social media communications tend to be characterised by different structural and linguistic properties. SLANDAIL worked to develop methods of text analysis that would be effective in identifying and extracting relevant information from spontaneous communications in social computing systems.
Terminology (disaster-specific terms) and ontology (a defined set of concepts in emergency management) were identified as key research areas. These contributed to the design and build of the SLANDAIL Social Media Monitor and localisation of the SLANDAIL system to include all three project languages (English, German and Italian). Our methods of language processing and of terminology management can be exploited during operations to facilitate communications when time is limited and stress may limit operators’ ability to communicate effectively.SLANDAIL combined models of and investigations on the language of affect (Martin and White 2005), sentiment analysis (Ahmad 2011), disaster communications (Haddow and Haddow 2014) and the language of social media (Zappavigna 2012) and adapted these to the study of the language of affect in formal and social media in English, German and Italian.
Image analytics: Images have to be processed in a timely manner and as close to real time as possible. SLANDAIL research involved studying images and videos, often of inferior quality, made and posted by citizens affected by disasters. We deployed an image recognition system for detection of images related to flooding and events related to flooding. The results of our controlled experiments using degraded images indicate that the SLANDAIL system can still produce acceptable results when all images are degraded, giving it a broad application in practice. We have produced a system in which the outputs of text analysis are integrated with image analyses at the feature level to enhance the informational content. Furthermore, we tested a video analysis system for shaky camera footage in disaster zone areas, evaluated by End Users.
Integration of an emergency management system (EMS) and social media monitor: At the outset of the project, two separate systems existed that each performed complex tasks – CID’s Topic Analyst and DataPiano’s Sistema Informativo Gestione Emergenze (SIGE). Topic Analyst, a tool for the monitoring and analysis of textual data, was capable of processing documents in volume from the internet and providing in-depth analysis. SIGE, an Emergency Management System (EMS) and the base program used to build the SLANDAIL System, was capable of retrieving information on ongoing disasters and assisting emergency management communication by accessing databases of key personnel. However, a system that integrated both capabilities, as well as incorporating image analytical capabilities, had the potential to deliver numerous benefits, including real-time information and “eyes on the ground” information from citizens.
Early on in the project, we loosely coupled these existing systems to prove feasibility. Due to limitations in German law and contractual agreements concerning CID, we were unable to use live data from their system. So, we employed and integrated a Social Media Monitor in line with End User needs to enhance emergency management control room capabilities. Using multi-modal aggregation we tested a method for increasing decision-making capabilities using different signals of information, including disaster-zone information aggregated with social media data.
Ethical and societal awareness: Data gleaned from social computing systems is characterised by an inherent tension between mass and personal communication, since the informal and local nature of much of the content can sit uneasily with the global, open-access nature of the platforms on which it is shared. SLANDAIL sought to balance the needs of emergency managers and first responders for an effective and flexible system with the rights of citizens to privacy and data protection.
SLANDAIL examined the societal impact of social media in a security context and outlined an ethical framework for use in disaster management. This framework sets out specific measures to be taken in the technical design, licencing, administration, governance and use of the system. Given that there are various controversies on government surveillance relating to the use of social media (Bertot, Jaeger et al. 2012, Bekkers, Edwards et al. 2013), existing disaster management literature needs clarity as to the ethical validity of collating large amounts of personal data when people are at their most vulnerable.
Legal research culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. This agreement enabled the project to pursue the objective of ensuring the ethical harvesting and analysis of social media data to improve disaster management outcomes. It will provide a foundation for future projects and represents a novel contribution to the research landscape of the ethics of disaster management practice, disaster response law, human rights law and data protection in exceptional circumstances.
Communication: Some emergency management experts refer to social media as a kind of psychological first aid and as a system for ensuring community cohesion (Taylor, Wells et al. 2012). Social networks can be key tools in building real networks and fostering community resilience (National Academies of Sciences and Medicine 2017). Our case studies demonstrated the importance in emergency management of delivering clear, authoritative information to citizens in a tonally appropriate manner – namely, a manner that generates trust. End User beneficiaries identified the need to have model messages available that can be swiftly adapted to incident-specific situations. SLANDAIL developed a method to analyse, translate and produce messages for emergency management, and implemented these through an integrated communication system.
Emergency management systems reviewed by SLANDAIL exist in one language only (usually English) and no provision is made for porting them into other languages. The SLANDAIL prototype’s Graphical User Interface (GUI) has been localised in three languages, with the potential to port the system of analysis for use in other languages. Graphical tiles are used so that communication via text is reduced to a minimum and translation into another language can be carried out quickly and in a cost-effective way.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our research includes exploring the use of facial expression analysis software to obtain real-time data on speakers’ facial expressions. SLANDAIL has developed systems to be integrated in the system prototype as APIs (Application Programming Interfaces), which can be used as communication training tools for emergency operators and first responders. iMotions, a system with this capability, was integrated in to the SLANDAIL System.
Project Context and Objectives:
Citizens regularly communicate through social media computing systems, particularly in times of rapid upheaval. During major emergencies or events the sharing of information (for example, infrastructural warnings, details of threat levels, and transportation advice) becomes an urgent necessity. As a result, a large volume of information relevant to emergency response is being generated in real time across social computing networks in a variety of new forms.
These developments mean that hierarchical models of communication between institutions and citizens are being challenged, with information now flowing in multiple directions through what have become known as “backchannel communications” (Cardoso 2008, Lindsay 2016). In situations of natural disaster, the scale and speed of events can threaten to overwhelm the capacities of emergency managers. By exploring the potential of modern computing systems to incorporate real-time feedback from the public, and by viewing the citizenry as a powerful, self-organizing, and collectively intelligent force (as Palen, Anderson et al. 2010 have influentially argued we should), we can add new tools to those currently available in disaster response. However, technical constraints as well as the complex ethical and legal issues surrounding data collection mean that emergency managers are currently unable to aggregate and systematically utilise this information.
SLANDAIL navigated the complexity of computing, ethics, the law and communications to create an aggregated and integrated system. Operational emergency managers were introduced to legal and ethical principles, while academic researchers developed modules in tandem with SMEs working in data analytics, emergency management and communications. The complexity of roles of active partners was carefully managed throughout the project, and disparate areas of research were carefully combined to create a working Emergency Management System (Figure 2).
At the time of SLANDAIL’s inception, the integration of social media analysis into disaster response processes was rare. SLANDAIL addressed this problem by equipping emergency managers with analytical tools to enhance response and planning capabilities during natural disasters, as well as with communications guidelines and legal and ethical principles for harvesting data and protecting citizens. The project combined expertise and resources in social media, security, ethics, law, emergency management, communication, text analytics, and image analytics, to deliver a multi-disciplinary solution to a challenging, real-world problem.
The system uses location-based geospatial information combined with processed social media data. The overall objective, in the context of security under the FP7 Framework, is for the project to help increase the security of individuals and groups living in areas affected by natural disasters by adding social media to sources of information available to emergency managers.
The project has been designed in response to requirements drawn up by control room teams that manage disasters and emergency events. End Users of the system have been closely involved in the design, testing and evaluation of each strand of the project. SLANDAIL partners collaborated to build and test a prototype system with due regard to ethical and factual data provenance. The SLANDAIL System aims to remove the burden of search and interpretation of the latent information contained in social media data from End Users.
Finally, SLANDAIL followed a Commercialisation and Exploitation Plan to further develop research outputs.
2.2. Glossary
Title Definition
SLANDAIL System An integrated system using emergency management and social media
SLANDAIL Social Media Monitor A platform that monitors social media live using advanced text and image analysis tools developed through project research
SLANDAIL Licence Agreement A licence agreement for use of social media data in emergency management systems that takes into account research in four areas of law: Data protection, copyright, IP and Human Rights
SLANDAIL End Users Emergency management teams and control room operators, from police and army to civil protection agencies, with operational experience in managing major natural and man-made disasters
2.3. Objectives
# Objective Achievement
1 COLLATE and REVIEW the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property Emergency Managers were involved throughout system design and testing. Research completed in the first six months outlined the usability requirements for End User beneficiaries. Test data collated as part of specific emergency related case studies was used to perform system tests.
2 CREATE protocols to protect the rights of the citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens The project created an ethical framework for using data in an emergency and followed this with preliminary legal guidelines and recommendations. This culminated in the development of an innovative model licence agreement for a social media analysis system for disaster response.
3 BUILD AND TEST a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management Built and tested a working prototype that combines emergency management systems with social media. As the project developed, state-of-the-art methods (text, image, speech/non-verbal communication) were introduced, tested and evaluated, leading to the creation and testing of an integrated system.
2.4. Achievements
Approaches to exploiting social computing systems to enhance the management of public crises have been co-evolving organically alongside the media themselves (Hughes and Palen 2009, Shklovski, Burke et al. 2010, Acar and Muraki 2011). SLANDAIL integrates these approaches and methodological reflections and explores how they can best be directed towards the development of a system for using social media in a disaster management context, focusing on improving communication during disasters and addressing the ethical issues involved. The project navigated the complexities of analytics, emergency management, ethics and communication. We successfully deployed, tested and evaluated a Social Media Monitor that analyses social media data in real-time, using advanced analysis in text and image. This was integrated with an Emergency Management System that is supported by a Licence Agreement with communication functionality to enhance End User capabilities.
Text analytics: Existing methods of text mining have been designed for formally written texts. Social media communications tend to be characterised by different structural and linguistic properties. SLANDAIL worked to develop methods of text analysis that would be effective in identifying and extracting relevant information from spontaneous communications in social computing systems.
Terminology (disaster-specific terms) and ontology (a defined set of concepts in emergency management) were identified as key research areas. These contributed to the design and build of the SLANDAIL Social Media Monitor and localisation of the SLANDAIL system to include all three project languages (English, German and Italian). Our methods of language processing and of terminology management can be exploited during operations to facilitate communications when time is limited and stress may limit operators’ ability to communicate effectively.SLANDAIL combined models of and investigations on the language of affect (Martin and White 2005), sentiment analysis (Ahmad 2011), disaster communications (Haddow and Haddow 2014) and the language of social media (Zappavigna 2012) and adapted these to the study of the language of affect in formal and social media in English, German and Italian.
Image analytics: Images have to be processed in a timely manner and as close to real time as possible. SLANDAIL research involved studying images and videos, often of inferior quality, made and posted by citizens affected by disasters. We deployed an image recognition system for detection of images related to flooding and events related to flooding. The results of our controlled experiments using degraded images indicate that the SLANDAIL system can still produce acceptable results when all images are degraded, giving it a broad application in practice. We have produced a system in which the outputs of text analysis are integrated with image analyses at the feature level to enhance the informational content. Furthermore, we tested a video analysis system for shaky camera footage in disaster zone areas, evaluated by End Users.
Integration of an emergency management system (EMS) and social media monitor: At the outset of the project, two separate systems existed that each performed complex tasks – CID’s Topic Analyst and DataPiano’s Sistema Informativo Gestione Emergenze (SIGE). Topic Analyst, a tool for the monitoring and analysis of textual data, was capable of processing documents in volume from the internet and providing in-depth analysis. SIGE, an Emergency Management System (EMS) and the base program used to build the SLANDAIL System, was capable of retrieving information on ongoing disasters and assisting emergency management communication by accessing databases of key personnel. However, a system that integrated both capabilities, as well as incorporating image analytical capabilities, had the potential to deliver numerous benefits, including real-time information and “eyes on the ground” information from citizens.
Early on in the project, we loosely coupled these existing systems to prove feasibility. Due to limitations in German law and contractual agreements concerning CID, we were unable to use live data from their system. So, we employed and integrated a Social Media Monitor in line with End User needs to enhance emergency management control room capabilities. Using multi-modal aggregation we tested a method for increasing decision-making capabilities using different signals of information, including disaster-zone information aggregated with social media data.
Ethical and societal awareness: Data gleaned from social computing systems is characterised by an inherent tension between mass and personal communication, since the informal and local nature of much of the content can sit uneasily with the global, open-access nature of the platforms on which it is shared. SLANDAIL sought to balance the needs of emergency managers and first responders for an effective and flexible system with the rights of citizens to privacy and data protection.
SLANDAIL examined the societal impact of social media in a security context and outlined an ethical framework for use in disaster management. This framework sets out specific measures to be taken in the technical design, licencing, administration, governance and use of the system. Given that there are various controversies on government surveillance relating to the use of social media (Bertot, Jaeger et al. 2012, Bekkers, Edwards et al. 2013), existing disaster management literature needs clarity as to the ethical validity of collating large amounts of personal data when people are at their most vulnerable.
Legal research culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. This agreement enabled the project to pursue the objective of ensuring the ethical harvesting and analysis of social media data to improve disaster management outcomes. It will provide a foundation for future projects and represents a novel contribution to the research landscape of the ethics of disaster management practice, disaster response law, human rights law and data protection in exceptional circumstances.
Communication: Some emergency management experts refer to social media as a kind of psychological first aid and as a system for ensuring community cohesion (Taylor, Wells et al. 2012). Social networks can be key tools in building real networks and fostering community resilience (National Academies of Sciences and Medicine 2017). Our case studies demonstrated the importance in emergency management of delivering clear, authoritative information to citizens in a tonally appropriate manner – namely, a manner that generates trust. End User beneficiaries identified the need to have model messages available that can be swiftly adapted to incident-specific situations. SLANDAIL developed a method to analyse, translate and produce messages for emergency management, and implemented these through an integrated communication system.
Emergency management systems reviewed by SLANDAIL exist in one language only (usually English) and no provision is made for porting them into other languages. The SLANDAIL prototype’s Graphical User Interface (GUI) has been localised in three languages, with the potential to port the system of analysis for use in other languages. Graphical tiles are used so that communication via text is reduced to a minimum and translation into another language can be carried out quickly and in a cost-effective way.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our research includes exploring the use of facial expression analysis software to obtain real-time data on speakers’ facial expressions. SLANDAIL has developed systems to be integrated in the system prototype as APIs (Application Programming Interfaces), which can be used as communication training tools for emergency operators and first responders. iMotions, a system with this capability, was integrated in to the SLANDAIL System.
Project Results:
3. Description of the Main S&T Results/Foregrounds
In SLANDAIL, academics and SMEs worked together with emergency operatives to demonstrate a cost-effective and ethically robust way to integrate information from social media into an emergency management system. Our goal was to make maximum ethical use of the information obtainable from social computing systems to enhance the performance of emergency management systems and reduce the burden of interpretation on End Users. SLANDAIL partners collaborated to model a system that delivers consistent and relevant messages to disaster management personnel working in three languages (English, German and Italian) with accompanying guidelines for preserving data confidentiality and ensuring factual provenance of information.
The core of the research and development on this project is summarised in four main strands:
1) Emergency Management: What do emergency managers do in a disaster? What technologies do they use? How can social media data improve their operations?
2) Ethical Analysis and Legal Research: Is it legal to use social media? Are there any ethical boundaries for using personal data in an emergency?
3) Analytics Research: How can text and image be automatically aggregated and analysed to help emergency management teams?
4) Control Room Technology: How can social media be integrated into technical systems to improve emergency response? Can communications between emergency managers and the public be improved?
Figure 4: The four areas of SLANDAIL research developed in tandem with one another during the project
The following sections present an account of the work undertaken and results achieved under four headings. The first section, EMERGENCY MANAGEMENT: Requirements and Technologies (3.1) describes our analysis of the operational environment in which End User beneficiaries respond to disasters. The following section, ETHICAL ANALYSIS AND LEGAL RESEARCH: Society and data (3.2) examines the tension between the demands upon the different stakeholders involved in disaster response – the requirements upon emergency managers to protect life and property, alongside the need of citizens for privacy and confidentiality – and describes the measures taken by SLANDAIL to mitigate this tension and ensure the system would be ethically robust.
ANALYTICS RESEARCH: Advances in Social Media Analytics (3.3) outlines the work undertaken to develop the technical components of the system. It describes our work on advancing and adapting text and image processing techniques to the analysis of social media data, our research on speech and non-verbal communications, and our development of methods to aggregate these techniques to deliver relevant information. Finally, CONTROL ROOM TECHNOLOGY: Building a Social Media-Enabled Emergency Response System (3.4) describes the final stages of SLANDAIL’s work to deliver an integrated system to analyse data extracted from social computing systems in an ethical manner. This section outlines the methods used to consolidate and integrate our research and present a simple, consistent and user-friendly system that allows emergency managers to use social media to increase their situational awareness of an emergency. This reduces the existing burden of information processing upon emergency managers and enables them to respond more effectively to threats.
3.1. EMERGENCY MANAGEMENT: Requirements and Technologies
3.1.1. Introduction
SLANDAIL research was directed at increasing the security of citizens by enhancing the capabilities of emergency managers to respond in an informed and effective way to natural disasters. It was essential for the project to involve emergency managers from the outset, and our research during the first six months of the project focused on analysis of existing disaster response practices. Our work established the basis of the subsequent research by ascertaining the requirements and real-world operations of our End User beneficiaries – PSNI, GARDA, and BLS, with input from Protezione Civile, via DataPiano.
The acquisition of End User requirements, case studies and technical specifications allowed us to fulfil the first of our objectives: to collate and review the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property. It placed the project on a practical and empirical footing, informing our academic and technical research with an understanding of the situational contexts in which the system would function. It also clarified the tension (a tension which would later culminate in the creation of an innovative Licence Agreement) between, on the one hand, the ethical questions surrounding mass data collection and the privacy of citizens, and on the other, the need of first responders for a system that would deliver real-time data in an effective and relevant manner.
End Users led an analysis of disaster response studies to discover common patterns in the management of crises and to establish the extent to which social media analysis is integrated into emergency response practices. These case studies focused on flooding, a problem common to a range of natural disasters, to enable an effective comparison of disaster management protocols and operational environments. The case study outcomes demonstrated the need for more effective analysis of social media data as well as providing an outline of the phases and communicative demands of a crisis.
Our analysis included a technological assessment of End User systems. We found that current systems would be compatible with a social media monitor and identified the importance of a flexible, web-based system that would be adaptable to disaster response practices. This analysis also revealed the value to End Users of having communication templates and guidelines to communicate information to citizens during times of crisis, and highlighted the importance of building trust in advance of emergency situations.
End User involvement continued throughout the project in the form of meetings, software demonstrations and software evaluations. Feedback was assimilated into each successive stage of software development. End User feedback was also incorporated into research on the ethical and social impact of social media analysis. End Users highlighted the duty of care to protect life and property first during an emergency. Emergency managers’ primary focus is the safety of citizens, and their key concern in using data is in the recovery period when subsequent investigations are carried out. At this stage, they wish to ensure there is a hierarchy of responsibility for decisions made.
Meetings between emergency managers, legal experts and academics were key to identifying the tension between this obligation on the part of End Users and the need to protect the privacy of citizens. This tension informed our research into the relevant ethical, practical and statutory issues and development of the legal framework within which the system could operate.
Figure 5: SLANDAIL’s first 6-month plenary meeting was held at the Titanic Centre in Belfast in October 2014. The meeting was organised by members of the PSNI and was attended by representatives from all partners.
Attendance from all partners included (l-r, back row-front row): ROW 1: Jose Maria Zavala Perez (CIES), Sebastian Wustmann (CID), Eamon O’Loughlin (GARDA), Henrik Selsøe Sørensen (Strategic Advisory Board); ROW 2: Bryan Scotney (Ulster), Stephen Kelly, Shane Finan, Damian Jackson (TCD); ROW 3: Sonya Coleman, Min Jing (Ulster), Rebecca Bury (Stillwater), Gemma Galdon Clavell, Sadhbh McCarthy, Gary Doyle (CIES), Alexander Stumpfegger (CID), Maria Grazia Busà (UNIPD), Jon Henry (PSNI); ROW 4: Raffaella Panizzon (UNIPD), Victoria Macarthur (Pintail), Brian Halligan (GARDA), Khurshid Ahmad (TCD), Una Williamson (PSNI), Gerhard Heyer (INFAI), Daniel Zschuckelt (BLS), Enrico Musacchio (DataPiano), Sabine Gründer-Fahrer (INFAI), Francesco Russo (DataPiano), Carlo Aldrovandi (TCD)
3.1.2. Disaster Management Case Studies
At the outset of SLANDAIL, each End User developed a case study in their own catchment area to demonstrate real life scenarios in which social media was or was not used in disaster or emergency management. GARDA detailed an incident of widespread flooding on the M11, a major arterial route between Dublin city and the East coast of Ireland. BLS covered flooding in June 2013, when a string of rivers in the area of Saxony broke their banks. PSNI detailed the serious threat of coastal flooding during the winter of 2013/14 which had the potential to seriously impact a built-up area of Belfast city. This archive of disaster management case studies (D1.1) facilitated the sharing of information between SLANDAIL partners. Overall, these studies revealed that there was no consistent use of social media in disaster management by the agencies. End Users recognised that social computing systems could play an important role in terms of resource deployment, communication and decision making.
Subsequently, SLANDAIL reviewed social media usage in crisis management, focusing on specific instances of flood disaster management in the USA, Italy, and Germany, in order to understand the operational environment and identify potential development. Each of these instances showed different levels and sophistication of usage, and the review served as an analytical tool to illustrate the ways in which social computing systems have helped emergency services in the past, and identify where social media use could be improved upon.
It emerged that emergency services are hampered by the lack of a clear and efficient method of aggregating Twitter or Facebook streams to form concise and direct messages. It also became clear that in times of crisis the demand for information rises dramatically, and users’ ability to source credible, reliable information decreases. The issue of trust again came to the fore as an essential factor in communication between emergency services and the public.
3.1.3. Review of Technological systems
We reviewed the technological systems and capabilities of each of the End Users, using case studies and questionnaires to ascertain the technologies used by End Users. Two police forces and one army reserve (PSNI, BLS, and GARDA) as well as the Protezione Civile of Veneto Region (responsible for coordinating the work of civil protection agencies in the Veneto region), provided information on the technologies they use during the management of emergency situations. This review encompassed all the components of the End User beneficiaries’ IT equipment as well as extending the analysis to IT use during emergency management by other significant US and International organisations (like FEMA and the Red Cross) to ascertain common capabilities and practices.
We determined that all existing End User resources were compatible with design requirements, but that some integration of existing structures would be necessary. SLANDAIL was developed to answer End Users’ needs by using multi-platform software to ensure adaptability and enable the re-use of existing hardware. We designed the Emergency Management System to be highly configurable to minimise customisation needs. We found that social media tended to be used to manage the earliest phases of an emergency on an ad-hoc, manual basis, and we identified potential for it to be further integrated into existing operations. We subsequently integrated the Emergency Management and Social Media Monitor (3.4.2). The review highlighted the importance of clear and fast communication during emergencies, as well as the need to establish a trusting relationship with citizens. This gave the project team a broader picture of civil technologies and communication media used by End Users, and facilitated advances in research and technology development.
3.1.4. Information Sharing and Public Communications
SLANDAIL determined how communication occurs during emergencies and how information is exchanged and interpreted by the public and other stakeholders. The approach included a workshop hosted at the LREC Conference in Reykjavik in May 2014, analyses of video footage of official communications during crises, and the development of guidelines for effective disaster communications reflecting best practice in professional communications. The resulting analysis (documented in D1.4) identified gaps in information exchange processes and enabled an overview of how communication occurs during emergencies (Busà and Brugnerotto 2014, Musacchio 2014). This aided the technical development and assessment of the SLANDAIL prototype and helped End Users to improve their communications by building trust with stakeholders before disasters. Trust emerged as a key issue in this research (Fennell 2014).
3.2. ETHICAL ANALYSIS AND LEGAL RESEARCH: Society and data
3.2.1. Introduction
Social computing systems, including social media systems and social networking systems, facilitate personal communication and voluntary collaboration. They are frequently used to transmit local and individual information, and during disasters they offer information that may not be available from other sources. Once transmitted, much of this information becomes accessible to third parties and may be collected, stored and used in ways that have not been anticipated by the individual. The widespread growth as well as the innovative nature of social computers has outpaced legal mechanisms: The General Data Protection Regulation (GDPR) and its parallel directive aimed at law enforcement (LEPD), initially published by the European Commission in 2012, will be adopted across the EU in May 2018.
Within a project consisting of operational End Users, academic researchers and SMEs, it was important to first address the need to address differences in experience and expertise relating to data use. End Users heighted a ‘duty of care’ whereby they are required to first save life and property with whatever information is available to them, and secondarily to consider any other legal requirements. The use of social media data forms part of an opaque landscape of legal and ethical questions, and this project answered these questions through dialogue between legal experts, ethical researchers and End Users. A Socratic Dialogue method was opened between legal experts, End Users and SMEs in the early stages of the project (November 2015). The successful results of this process showed that expertise in law was needed to adequately address the societal challenges inherent in use of data for emergency management, and this led to an adjustment of focus toward the legal landscape that exists in copyright, data protection, IP and human rights.
There is a basic and ongoing tension between the public openness of mass communication and the informal nature of personal communication. SLANDAIL research into the impacts of using social media in a security context explores these ethical and social tensions to create a framework for navigating the evolving informational space whilst ensuring factual and ethical provenance.
SLANDAIL approached the societal impact of social computing systems in a security context from an ethical perspective. We analysed relevant critical thinking to develop a coherent framework for balancing the competing rights and responsibilities of stakeholders in disaster events. This research showed that following existing human rights law is the most appropriate means by which to consistently protect the dignity of citizens and the integrity of data (De Stefani 2000, Berger, De Stefani et al. 2016).
Ethical analysis included ongoing dialogue between End Users, academic partners and legal experts and functioned as a scaffold for the formation of a licence agreement. This addressed the second of our key objectives: to create protocols to protect the rights of citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens.
The main output of this work is a model Licence Agreement – the first of its kind – and an accompanying Legal Checklist, representing the translation of SLANDAIL’s analysis into a practical and enforceable agreement.
• Licence Agreement: SLANDAIL research found that the system needed to deal with a complex but surmountable set of issues that could be dealt with using existing legal mechanisms. The Licence Agreement conforms to existing protocols in the field of data management and creates specific clauses for the field of disaster management. It specifies the scope of the system’s use, limiting its deployment to humanitarian purposes and specific territories. It also points the End User towards the steps to be taken to access live social media data and comply with local statutes.
• Checklist: The accompanying checklist empowers End Users with information and awareness of the issues surrounding the use of social media data in an emergency management context. It functions to encourage diligence on the part of the licensee and to highlight areas – such as data security, privacy, and copyright – in which particular safeguards are needed.
SLANDAIL has charted the legal and ethical landscape surrounding the use of social media data in disasters by synthesising relevant research and expertise. An audit trail of the decision-making processes behind the Licence Agreement is recorded in project documents, showing its collaborative development. In a dynamic social, technical, legal and regulatory environment, the Licence Agreement and accompanying Checklist represent a template for further legal agreements as well as a baseline for future innovation. They provide a foundation for future projects and represent a novel contribution to the research landscape of the ethics of disaster management, disaster response law, human rights law and data protection in exceptional circumstances.
3.2.2. Review of Societal Impact of Social Media in Security Contexts
SLANDAIL conducted a comprehensive review of the current critical thinking on the societal impact of using social media in security contexts. This comprised a review of academic, legislative and media sources, with a focus on the legal and regulatory framework and policy context within which the project operates. A four-part framework was developed encompassing desirability, acceptability, ethics and data management aspects of using social media data to enhance decision support for emergency managers. This provided an initial assessment of the relevant societal concerns for SLANDAIL.
Following this assessment, it became clear to the project that expertise in law, particularly law concerning social media in emergency response, would be required to fit End User and technical requirements for the project. A partner in Data Protection at major law firm Arthur Cox (Dublin) was invited to attend a seminar organised and opened a Socratic Dialogue between End Users, technical researchers, ethical researchers and SMEs. This dialogue resulted in further legal research and the drafting of guidelines for the use of social media in emergency contexts. The success of this research led to an adjustment of the project to focus on the complicated and opaque legal landscape surrounding use of social media in emergency response.
3.2.3. Ethical Framework and Guidelines
We applied the theoretical approaches of value pluralism and state of exception ethics to the specific domain of the use of social media in disaster response (D2.2). The framework set out measures to be taken in the technical design, End User practices at the various phases of emergency management, licensing, administration and governance of the SLANDAIL system to mitigate ethical risks. An ethical matrix method was applied to tensions between emergency management and ethical requirements. One of the key outcomes was highlighting the fact that restrictions to more sensitive data would be required to ensure that personal information is only accessible to emergency managers with a high level of access.
SLANDAIL explored the technological and legislative developments in the field of data harvesting from social and formal media, presenting a set of guidelines and a template of harvesting social and print media data. We identified implications, most notably the importance of adhering to principles of proportionality, anonymity and security. Results stated that data collected should be proportional to the need of emergency managers; personal data that is not anonymised should only be kept for as long as it is required; and data should be stored with a reasonable level of security (Jackson and Hayes 2016). The guidelines recommended specific procedures including a high standard of security, compliance with data security practice, a threshold of factual reliability, and an in-depth logging system to track and record all activities of the system.
We developed a framework comprising protocols by which to structure the analysis of disaster response practices with appropriate regard to the interests of all stakeholders involved. This comprises two phases: reflective analysis and critical reflection, incorporating respectively an analysis of stakeholder interactions and interests, and a reflection on the responsibilities arising from existing power dynamics (D2.10). Critical reflection on the power implications of extant practices enables context-sensitive ethical deliberation through re-examination of these practices and development of proposals for their refinement, alteration or replacement.
3.2.4. Guidelines and Templates
3.2.4.1. Data privacy in disaster management
Copyright and data protection legislation within the framework of German and EU law was assessed to deliver legal opinion for the project (D2.4). We determined that SLANDAIL is intended to be deployed in a lawful manner in pursuit of the End Users’ legal obligation to protect life and property, meeting national legal requirements applicable in the context of their deployments. However, given the multiplicity of contexts and the lack of harmonisation in legal systems, it was recommended that social media analysis be made available to End Users on the basis that they would be contractually obliged to ensure that they address any data protection, privacy and copyright issues that might arise.
We further analysed the legal framework of European copyright law (with a focus on German law, D2.7) to ascertain the requirements for the use of copyright protected material within a project like SLANDAIL. We found that copyright law is significant for our project, having the potential to be a hindrance to the use of social media content for disaster management. Social media content is often deemed subject to protection of the social media provider. The use of the system in a disaster scenario should be enabled and facilitated by specific statutory legislation. Such a provision could be developed in the framework of limitations and exception for copyright-protected works for public security. As of 2018, the General Data Protection Regulation will provide a legal basis for projects like SLANDAIL, permitting disaster management entities access to personal data in cases of disaster response. Our findings fed into the final software licence agreement (D2.6).
3.2.4.2. Legal protection of human rights in disaster management
SLANDAIL analysed the legal scenario of the proposed system with specific reference to the spectrum of internationally recognised human rights, with a distinctive focus on privacy and data protection rights. Meetings and workshops were held between the project beneficiaries – including experts in intellectual property, data protection, human rights and internet law, and the ethics of technology – to discuss and develop this research (Berger, De Stefani et al. 2016). The result was a set of guidelines on the transparent use of the SLANDAIL system for disaster management, mainly addressed to End Users (D2.5). These included recommendations that End Users familiarise themselves with legal procedures for using data in an emergency, determine the conditions under which the SLANDAIL system can be deployed, and maintain open channels of communication during an emergency. These guidelines provided for the consistent integration of human rights awareness in the disaster management system.
Our research emphasised the importance of fully integrating SLANDAIL into the institutional and legal architecture of civil protection structures at the local, national, European and international levels, and highlighted the dignity of the human person as a common consensual principle through these legal levels (Hayes 2017). We found that the SLANDAIL prototype fit the legal design outlined, and that the project’s original features and specific characteristics of flexibility, multilingual capability, transparency and people-centeredness make our system a tool that complements the human rights based approach to civil protection policies advocated (D2.8).
3.2.5. Creating Guidelines for Developers and End Users
We delivered a Template Software Licence Agreement to be used as a model to license the SLANDAIL system to End Users. Based on an analysis of the Legal Caucus and with the experience of four legal experts in four EU countries, we distilled the core issues identified into (a) a template software licence agreement and (b) a legal checklist which shows, from a practical legal perspective, how the system can be licensed.
The license includes an acknowledgement that the Licensor does not own any intellectual property rights of data from social computing systems, an acceptance of responsibility for ensuring that the use of social media data complies with applicable laws, and an agreement on the part of the End User to submit to EU law. The legal checklist is a synthesis of the research authored by ethicists and legal experts in Ireland, the United Kingdom, Germany and Italy. The checklist was designed to assist End Users in identifying legal requirements prior to deploying SLANDAIL in any live environment. The checklist covers five key legal areas relevant to the use of SLANDAIL: contract; data protection; privacy; copyright and human rights.
SLANDAIL also explored the parameters and limitations of laws that govern Internet contracts, specifically personal data and proprietary information on social media, in relevant jurisdictions. We produced a report (D2.9) on these laws that highlights the extent to which SLANDAIL software programmers, vendors, emergency responders or End Users can safely and lawfully derogate from prescribed laws without infringing third party rights in personal data and proprietary information on social media.
Figure 7: Rob Corbet, Partner in Data Protection at leading Irish law firm Arthur Cox, speaking at the launch of the SLANDAIL System Licence in February 2017.
3.2.6. Information Ethics and Fiduciary Theory
SLANDAIL research into the ethics of information and implications for the large-scale harvesting of social media data resulted in the development of the Intrusion Index (D4.2) which informs the End User when a large volume of personally identifying information is being collected. This feature recognises the importance of building software tools with issues of trust and security in mind (Delgado, Torres et al. 2005). It functions as a moral mediator that preserves the discretion and decision-making abilities of End Users while recognising the value of privacy and mitigating the risks of potential undesirable societal and ethical impacts of the system. It was subsequently incorporated into the SLANDAIL Social Media Monitor (cf. Section 3.4.2.1).
Further research was undertaken to bring research in ethics beyond the state-of-the-art. SLANDAIL has given rise to a doctoral research programme within the School of Ecumenics in TCD for dealing with the ethics of using technology in disaster management. The project completed transfer in December 2015 and is entitled “An Analysis of Emerging Human Rights and Ethical Issues in the Harvesting of Data from Social Media During Emergency Response to Natural Hazards.”
This research presents a joint ethical and legal analysis within the frameworks of Information Ethics and Fiduciary Theory. Through an analysis of emergency management systems developed by SLANDAIL partners, it assesses the potential human rights and ethical impacts of these systems. It considers social computing systems with respect to values of Privacy, Justice, Trust, and Accountability. The methodology comprised interviews with software developers and emergency responders involved in the project as well as observation and use of emergency management software systems. In interviewing key actors, it was possible to discern the functionality of systems such as SLANDAIL as well as the potential for additional functionality not present in the system itself. While some of this work has been completed during the project (Hayes 2017), the work is ongoing as part of a PhD thesis with a proposed submission date of September 2017.
3.3. ANALYTICS RESEARCH: Advances in Social Media Analytics
3.3.1. Introduction
Figure 8: SLANDAIL Analytics Research culminated in the development and deployment of a social media monitor that combines research strands.
Much of the focus in social computing analytics is on text analytics (Batrinca and Treleaven 2015, Wu, Cao et al. 2016), as the processing of speech and images poses difficulties of a different order of magnitude (Tzelepis, Ma et al. 2016). We have developed methods and prototype systems to analyse and render text available through social networks into items of actionable information. For image analytics, we have developed a new technique for faster image processing based on recent findings on human visual systems and developed the use of neural networks for image annotation, thereby rendering images and videos into complementary actionable information. We have combined analytics in a Social Media Monitor, where text, image, geolocation and other data are combined to integrate with emergency management systems.
We have made use of metadata from social media messages related to location. Geolocation of impacted areas is a critical issue in disaster management, and social media systems have been coupled to geographical information systems with some success (Chae, Thom et al. 2012). Location data, when used with professional care, will yield information that can be mapped onto modern geolocation information systems, comprising socio-economic and other reliance data (Stefanidis, Crooks et al. 2013). We have fused data to create actionable location-specific information incorporating the EU geolocation framework, INSPIRE .
Furthermore, SLANDAIL investigated the role of non-verbal cues in disaster communications – facial expressions, speech quality, and hand gestures – particularly in disaster mitigation speeches made by leaders. The emotion shown by authority figures can, in negative instances, have a devastating impact on disaster victims (Wang, Zhang et al. 2016). We have used state-of-the-art emotion recognition systems for each of the three modalities, and validated the results against an observational survey. This approach will be useful for disaster communications training and has extended recognition research by using it for real-world data.
The research contributed significantly to the development of a suite of programs that can collect, filter, and analyse (multi-modal) social computing outputs, link this output to geolocation data, and convert these into actionable information. Some integration was carried out in the Social Media Monitor, a standalone system that combines elements of text and image analytic research. Some research (multimodal aggregation and multi-criteria decision making) was not integrated but was carried out to show feasibility in future similar systems.
The software developed be made available to any GIS-enabled emergency management control room system. The state-of-the-art research was conducted in collaboration with End Users, legal and ethical partners, and the technology partners, who participated in specification and evaluation. Considerable advances have been made in the novel use of social media and social network outputs in the following areas, and these have been published in conference papers and journal papers.
End Users emphasised the need to cultivate a relationship of trust with citizens, and for analytical methods and outgoing communications to conform to local linguistic conventions. SLANDAIL created clear communication guidelines through linguistic and communication research. We enabled the processing of social media data in German and Italian, and developed model messages for organisations to communicate real-time information to the public. To combine and integrate our analytics with the Emergency Management System, we implemented a common information exchange format (XMI) for compatibility.
3.3.2. Text Analytics
Social media output and web feeds released by agencies, experts, and citizens, can augment existing data. A number of techniques are being developed to analyse this information, such as tweet pooling and topic modelling (Mehrotra, Sanner et al. 2013) as well as scraping location information related to an unusual event (Wakamiya, Jatowt et al. 2016). Studies have explored methods for harvesting community-oriented information related to disaster alerts (Shih, Han et al. 2014) and for aggregating tweets to profile and geo-reference an impacted area during a flood (Johnson, Ruess et al. 2016).
Social media has been used to gauge levels of public concern about a disaster caused by potential human error (Olteanu, Vieweg et al. 2015, Woo, Cho et al. 2015) by adapting a social media analysis system developed in Germany to process social media messages in English. See, for example, work in English (De Choudhury, Gamon et al. 2013), German (Thom, Kruger et al. 2015), Italian (Avvenuti, Cresci et al. 2014, Cresci, Tesconi et al. 2015, Avvenuti, Cimino et al. 2016), and Chinese (Fung, Fu et al. 2013)). Large-scale text mining systems have been offered for disaster management - see SAS’s text miner (Lazard, Scheinfeld et al. 2015), IBM’s Modeler (He, Zha et al. 2013) and the RapidMiner system (Ristoski, Bizer et al. 2015).
SLANDAIL developed a text analytic module for analysing static text data collected from both social and formal media; later in the project, a real-time multi-modal aggregation system (written in Java and using existing systems for dependency) that can capture, process, and display information extracted from live social media stream and web feeds was developed. We gathered and tested a body of text numbering over 500,000 domain documents with the assistance of automation algorithms (CiCui) and from these we created a lexicon for disaster management terms, particularly those related to incidents of flooding. We simultaneously developed standalone tools for research, the Summary of High Incidents (SHI) and the SLANDAIL Newsletter, to provide regularly updated information about emergency and disaster-related Twitter messages and news. These operate primarily through English, but can include some German and Italian texts.
Some of the text analytic methods developed above were later integrated into the SLANDAIL Social Media Monitor which combines image analytics when performing aggregated analysis over live social media stream.
3.3.2.1. Specifying and Creating Grammar for Social Media
CiCui is a background technology for SLANDAIL. It comprises text analytic module enhanced within the project that features functions such as calculating keyword metrics, extraction of collocation contexts, terminology identification, latent semantic analysis, and contains corpus management facilities (e.g. automatic downloading and parsing of domain documents). CiCui has been used to producing a tri-lingual data base of disaster terms that will complement EU environmental data bases and is being delivered to the EU terminology clearing house IATE. We have used methods for identifying terms automatically, developed in (Sparck Jones 1972, Ahmad and Rogers 2001). The processing requires a database of domain keywords, phrases, and, language patterns commonly used to discuss emergency events. The building of such knowledge bases are fulfilled through a system called CiCui (Collection of Words in Chinese), developed as part of a related doctoral project on the statistical learning of language patterns related to events (Zhang 2016). The main functionality is to convert unstructured textual content into structured indices from which one can extract information about the texts with standard querying languages. CiCui has also been used to support text analysis as part of the CITU image annotation system (see 3.3.5.1).
It has been observed that in specialist domains and during emergencies, variations of fixed phrases are used to indicate abnormal events. Compare the sentences ‘road between Carlow and Leighlinbridge is closed now’; ‘road Carlow to Leighlinbridge closed’; and ‘fallen trees on Carlow/Leighlinbridge’. For a human being these phrases mean roughly the same. However, these patterns defy lexical/syntactic/collocational conventions of the English language known also as frozen phrases or recurrent patterns that hallmark discourse in specialist domains and during abnormal events. The term used here is local grammar that governs the behaviour of tokens in these phrases and one can assign a probability of usage to each pattern.
Our investigation of the use of the so-called local grammar in dealing with the convention defying phrases that are a distinguish feature of social media systems, like Twitter, and other (micro) blogging services, is amongst the first in this field. We made a statistical analysis of two corpora comprising more than 14 million Twitter messages collected during abnormal whether events (Hurricane Matthew and a severe snowstorm in eastern US in 2016). We had taken advantage of algorithms developed previously for sequential pattern mining (VGEN and VMSP) that has been used traditionally in identifying behavioural patterns in transactional databases used typically in finance, consumer research, on-line trading. We have developed a specialised R package that interface with the indices produced by CiCui to determine the sequential patterns and generalise these to build a corpus-driven local grammar patterns like the examples provided in Figure 10.
Figure 10: Definition of an example local grammar pattern for capturing road/bridge/lane closures/reopens.
Once elicited and validated by experts, the local grammars are encoded using regular expressions (specifically TokensRegex) that allows complex word sequences to be matched based on not only the characters but also the linguistic meta-data associated with the words. These patterns are stored in the SLANDAIL terminology database for recognition of the frozen phrases. These phrases are used in processing and converting social computing output into actionable information. The system is adaptable, in that local grammars can be refined during each operational cycle. The workflow of this process is shown in Figure 11.
Figure 11: Workflow of the local grammar extraction process
3.3.2.2. Specify and Create Lexica for Social & Formal Media
SLANDAIL collected corpora and extracted terminology to develop a terminology wiki (the SLANDAIL Lexicon), which reflected state-of-the-art terminology management. Secondly, we automated term extraction and refinement methodology using the CiCui system. Thirdly, we developed automatic extraction of information (definitions, contexts and ontologies) to compile the Lexicon entries and develop a routine to make it available in an easily accessible format such as XML. This methodology enabled the greatest possible degree of automation of the main processes in terminology, and has allowed us to submit our terminology to IATE (InterActive Terminology for Europe), the EU terminology reference.
CiCui term extraction capabilities was evaluated in a comparison with four commercially available term extraction systems (Synchoterm, TaaS, Termostat, and Vocabgrabber). The test corpus comprises a 326319-word document corpus containing FEMA fact sheets, handbooks from different emergency management agencies and news items extracted from LexisNexis. Extracted terms from the corpus were evaluated by linguistic and translation experts and each of system produced terms on a three-point scale: ‘Not a term’- R(ed); ‘Needs further valuation’ -A(mber); and ‘System correctly identifies term’ G(reen). CiCui outperformed the four commercial systems in emergency terms, and had the lowest rate of producing red terms (Table 1).
Table 1: Comparing the performances of automatic term extraction systems
System # of TCs
extracted Precision RAG evaluation
Red (%) Amber (%) Green (%)
Synchroterm 14791 56% 30 14 56
TaaS 345 42% 48 10 42
TermoStat 4082 64% 20 16 64
Vocabgrabber 2501 14% 78 14 14
CiCui 602 77% 6 17 77
Average 36.4 14.2 50.6
Standard Deviation 27.8 2.7 24.1
We created a term bank to international standards and made this available to stakeholders, minimising the burden of labour-intensive, time-consuming terminology management. This term bank develops mission-critical terminology by considering different communicative situations, and displays terms and their equivalents accompanied by visualisation and information about usage. It was made available as a knowledge base for the project ontology. The Lexicon also establishes best international practices by extending the method used for English to the less widely spoken languages of the project, German and Italian.
3.3.2.3. The SHI System and the SLANDAIL Newsletter
SHI (势 SHI) - Summary of High Incidents - is a background system developed during the project that contributes to research outputs. It harvests news through RSS feeds and English-language tweets (via Twitter’s Streaming API) from selected sources. It is based on Rocksteady, an affect analysis system, and combines a data collection system with a data analysis system that filters harvested texts in relation to domain and sentiment dictionaries. It automatically detects unusual frequencies of lexical categories in emergency-related dictionaries and sends an automatic alert message published through a Twitter account, @SLANDAILalert.
SHI helps to produce the SLANDAIL Newsletter. The Newsletter, updated daily, provides information about emergency and disaster-related events (Figure 12). Keywords are identified using weather- and disaster-related thesauri (for example FEMA, Global Disaster Alert and Coordination System, WHO). In one sense, the Newsletter is a record of trends emerging within a specific period. The system uses graphs, Tweets and captions to display outputs. It was built to process English language text messages and has limited capability to analyse German and Italian texts.
3.3.2.4. Text Clustering
Text clustering, or document clustering, refers to the grouping of textual materials into “clusters” considered to share a common theme. SLANDAIL developed a clustering application through existing system Leipzig Corpus Miner (LCM) in which topic modelling techniques were used to automatically categorise texts into topic clusters. We adapted this to automatically identify disaster-related topics in formal and social media (Schlaf, Gründer-Fahrer et al. 2016), applied temporal clustering techniques to automatically identify characteristic phases of the disaster event (Gründer-Fahrer Submitted), and classified messages on social computing systems with respect to their relevance to flood damages (Musacchio, Panizzon et al. 2016). We explored the representativeness of linguistic features such as unigrams (single words) and bigrams (two consecutive words) as well as the language patterns defined in the SLANDAIL Lexicon. Human annotators labelled a dataset of social media messages to judge whether they contained flood damage information: vectors associated with the messages were then used to train statistic classifiers to automatically identify messages that report flood damage. This allowed for integration with the media monitoring system Topic Analyst to show phases of disaster management in recovery and analysis stages.
3.3.3. Speech and Non-Verbal Communication Analytics
The management of a catastrophic event requires authority figures to communicate with the population showing empathy and leadership. Social media and networks are also used to “telecast” disaster warnings and recovery information in which messages are conveyed not only through words but also through facial expressions, vocal quality, and body gestures. Leadership studies show that audience perception of facial, vocal and hand gestures plays a key role in the audience perception of the message quality (Burgoon, Guerrero et al. 2016).
An automatic analysis of non-verbal disaster communication complements spoken and written communication. This will serve to make the disaster managers aware of the possible perception of the public when they hear and see the managers make life- and business-critical announcements.
3.3.3.1. Towards an Integrated System for Non-verbal Communication
SLANDAIL assessed the feasibility of using speech and gesture technology for automatic extraction of data on speech and non-verbal language. We evaluated the relevance of existing systems and patents of potential future systems (D3.2). This involved the creation of a corpus of videos relating to disaster news during an Italian flood (2015) to identify patterns of voice modulation and body language and create a novel ontology of gestures (Busà and Cravotta 2016). This phase produced two clear findings: (a) that non-verbal communications are essential for gaining trust, especially during a disaster (Busa, Musacchio et al. 2015) and (b) that the extraction of speech data from emergency situations is currently not feasible due to the presence of noise and multiple voices in emergency speech excerpts, but that physical aspects of speech (such as pitch and intensity) could be of value to emergency management systems.
Following this, we looked at the integration of voice quality and hand gestures with research on automation of facial expression recognition (Bartlett, Littlewort et al. 2006, Bartlett, Littlewort-Ford et al. 2014). Emotient is a facial recognition system used for studying facial expressions in controlled conditions. It analyses the movement of facial muscles against a pre-trained database of facial expressions, using actors simulating positive and negative emotional states. This work, in turn, is based on the work of Ekman and Friesen (Ekman and Friesen 1969, Ekman and Friesen 1976), who contributed to the development of the Affectiva emotion recognition system (El Kaliouby, Mahmoud et al. 2016), based on earlier research in non-verbal communications in diverse fields (Picard, Vyzas et al. 2001, El Kaliouby and Robinson 2005, El Kaliouby, Picard et al. 2006). We have used both Emotient and Affectiva and have incorporated access to Emotient in the SLANDAIL system (See Section 3.4.3.2).
Simultaneously, work was carried out on gestures using ELAN (Brugman, Crasborn et al. 2004). Similar work was carried out on emotional voice quality using two major speech emotion recognition systems – PRAAT (Boersma 1999, Boersma and Weenink 2012) and OpenEar (Eyben, Wöllmer et al. 2009, Eyben, Scherer et al. 2016). The gestures in these videos were annotated using ELAN (Wittenburg, Brugman et al. 2006).
3.3.3.2. A Multi-Modal Communications Case Study in Disaster Communications
Much of the work in facial expression recognition (Bartlett, Littlewort-Ford et al. 2014, Du, Tao et al. 2014), emotional speech detection (Eyben, Wöllmer et al. 2009, Eyben, Scherer et al. 2016), and hand gesture identification (Pisharady and Saerbeck 2015) has been carried out on actors making faces, modulating their voices, and making pre-determined gestures. Our work is a significant departure and is part of a shift in research from laboratory to real-world communications (McDuff, El Kaliouby et al. 2015, McDuff and el Kaliouby 2016).
The rise of multi-modal information processing systems – where three different aspects of non-verbal communications (facial expressions, vocal quality and hand gestures) are analysed with simulated and real-world data – addresses the issue of socially interactive computer systems (Poria, Cambria et al. 2016, Poria, Cambria et al. 2017), (Castellano, Kessous et al. 2008, Kessous, Castellano et al. 2010)
To assess the interaction between three modalities, we collected and analysed how public speakers and broadcasters communicate with the public through their facial expressions, voice and body language in disaster situations. This involved the creation of a gender-balanced data set for analysing facial expressions and gestures of authoritative figures (e.g. mayors and governors) who announce impending hurricanes and snowstorms. The data set comprises the governors and mayors issuing disaster mitigation warnings and non-emergency announcements. The data was acquired from YouTube, and comprises 2.5 hours of running data in emergencies and 3 hours of normal conversations. We evaluated the systems through an observational study of how people perceive the governors (Ahmad, Busà et al. In Preparation).
Slice
(Emphasis) Slice extended finger
(Emphasis) Double Slice
(Empathic, openness) Ring
(Precise information)
instructions in our website. emergency disasters a very difficult time
state agencies can respond specifics about damage
Figure 14: SLANDAIL examined typical gestures used by leaders when communicating disaster information to the public to understand how non-verbal communication affects emotion recognition.
In a preliminary study of the three modes we found significant correlation between facial expression and voice quality; gestures were not correlated to the same extent as the other two modalities. This research can be used for training public speakers to deliver congruent messages during emergencies (Ahmad, Busà et al. In Preparation). The work on gesture and emotion analysis will increase understanding of how different stakeholders in a disaster event communicate with each other, often across cultural and ethnic divides. Streamlined access to the Emotient system has been incorporated within the SLANDAIL System to allow communications training as part of the overall system.
3.3.4. Image Analytics for still images and videos
3.3.4.1. Disaster Images: Photo-sharing and crowdsourcing
Images can be processed automatically and quickly to discard those that contain little useful information and to extract as much available actionable information as possible from those with a high relevance. This can be a challenging task: images that have useful actionable data are generally confounded by content that includes debris, objects in unusual positions, and noise arising from the circumstances in which the image may have been captured. An image shot during an abnormal event, like a flooding episode, will be noisy: the weather is likely to have been inclement, the camera may have been shaking, and lighting may have been poor.
The proliferation of images on social computing systems requires the development of systems that automatically categorise images and extract information from the images during an abnormal event. Social media interfaces make it easy to make images available to others. Images can be manually annotated and distributed using social computing systems through photo-sharing (Liu, Palen et al. 2008). Images can be used to enhance situation reporting (Yang, Ha et al. 2011, Yang and Chen 2015), and systems for crowdsourcing image data can help emergency managers (Xu, Liu et al. 2016).
Disaster victims can locate the disaster and the extent of the damage as efficiently as an expert in two out of three cases (Panteras, Lu et al. 2016). SLANDAIL has deployed automated analysis of images to extract useful information, as well as combining information from text and images. Images can play an important part not only in disaster reporting, but also in disaster management by helping to establish a two-way interaction between the public and authorities, as well as interactions between members of the public (Liu, Palen et al. 2008). This two-way ‘visual representation of disasters’ is an important component of disaster management, especially in remote places with poor infrastructure (Bica, Palen et al. 2017).
SLANDAIL has demonstrated how image analytics can aid disaster response by integrating image analytics into the SLANDAIL Social Media Monitor (see Section 3.4.2).
3.3.4.2. An Innovative Image Representation Framework
Given the continuous high-volume of data available through social computing systems, we have developed a Square Spiral Image Processing (SIP; D3.3) framework (Jing 2015) to speed up computation. The software implementations based on SIP facilitate the storage of image pixel indices in a one-dimensional vector based on a spiral architecture. This enables faster access and processing than the conventional representation of images that uses a two-dimensional Cartesian system. For image processing in the SIP framework, fast access to pixel locations in an image is achieved by mimicking aspects of the eye tremor phenomenon in the human visual system, combined with a non-overlapping convolution technique.
The extraction of local image features is based on key interest points using the Scale-invariant Feature Transform (SIFT) method (Lowe 1999, Lowe 2004), which are designed to persist over a range of scales and are robust to changes in conditions under which images are captured. Our approach uses a non-overlapping convolution method that can extract image features faster than conventional convolution techniques. SIP is a fundamental contribution, as it facilitates fast access to image data for processing at the pixel level and accelerates the execution of subsequent image processing algorithms. (Jing, Scotney et al. 2017).
The design of multi-scale SIP operators enabled us to capture features of objects at different levels of detail (Jing, Coleman et al. 2015). Application of the SIP framework has also been extended to video frame processing in which consecutive frames in the video are considered as the eye tremor images that are used for feature extraction based on SIP convolution. The improvement in computational performance has been evaluated for various tasks. For example, the results shown below for detecting edges in images of various complexities, using three benchmark images, Lena, peppers and coins, illustrate the way that SIP typically outperforms the standard algorithms (Table 2) (D3.3).
Table 2: Comparison of edge maps from standard 2D convolution and SIP-based convolution
Algorithm LENA RUN TIME (MS) PEPPER RUN TIME (MS) COINS RUN TIME (MS)
Standard 6.8 61.7 605.9
SIP 0.28 1.1 16.6
Image Size 81 X 81 243 X 243 729 X 729
3.3.4.3. Real-time Image and Video Analysis
In a disaster situation, video files uploaded by victims or observers can provide key information about ground conditions, such as the severity of a storm. Videos of storms uploaded to Twitter and YouTube often feature trees blown by strong winds, strong currents in floodwater, and rapidly rising water levels. SLANDAIL uses these to classify the severity of a storm or flood event. We deployed the concept of eye tremor to video data analysis by considering a sequence of video frames as a set of eye tremor images.
Our evaluation for video analytics focused on videos of blowing trees and floodwater extracted from social networking sites. In terms of storm severity, each sequence in our evaluation set was categorised visually into low, medium or high strength by four observers to create ground truth. A two-class grading scheme was developed to distinguish between low storm strength, and medium-to-high storm strength. An additional source for video data with potential use in flood management is provided by public network web-cams near rivers and bridges. Updated images are often posted at regular intervals, offering a visual indicator of change of water level or volume over a specified interval.
Our experiments demonstrate that moderately dynamic videos can be handled successfully using a video stabilisation technique (Matsushita, Ofek et al. 2006). (See Figure 15).
3.3.5. Multimodal Aggregation
SLANDAIL developed a novel framework that enables the integration of linguistic evidence with image features (D3.4). This approach built on methods developed (D3.1 D3.3) by combining contributions to image and text feature extraction.
Sharing and crowdsourcing of disaster images relies upon humans to categorise images. What is required is a system that not only can categorise and extract information from images with the help of collateral modalities, but also learns to automatically repeat and improve on this process. This necessitates the use of learning algorithms like neural networks that can be trained to recognise characteristic shapes, textures and boundaries and to recognise accompanying keywords and metadata (Hong, Wang et al. 2014). Once learnt successfully, the learning system can then recognise unseen images and annotate these images with keywords for subsequent retrieval (Ahmad, Tariq et al. 2003). We have used Ulster’s SIP framework to analyse images (see 3.3.4.2 above).
Text Representation: For training a neural network, we collect N-images, distributed over C categories, where C
The multimodal framework developed as part of SLANDAIL includes methods for integrating text and image features for image retrieval and image-text aggregation. The method incorporates linguistics evidence from the collateral text of the image, and has been trained using data from an Image Database built for SLANDAIL from social media (Flickr, Instagram, Facebook). The features for image and text data were extracted using content analysis methods developed in SLANDAIL and are aggregated in a multi-modal method to explore connections between modalities, and have been implemented in the CITU system (see below). The multi-modal method uses self-organising maps to learn relevant and key features in text and images for the domain of emergency management and learns the association between these modalities. Combining feature vectors in this way allows messages related to an emergency situation such as flooding to be retrieved more accurately as the image and/or text content can be used in classification and retrieval. This approach and the CITU system allows recognised images to be annotated with text or keywords and allow sample images to be given for certain keys words or text features. Combining modalities in this way and learning associations between images and the collateral text means the classification of messages in a social media stream can be improved.
The CITU system and underlying methods were developed to allow annotation of images and to aid classification of emergency related messages retrieved from a social media stream. CITU is a desktop application with a user-friendly GUI (Figure 16) and allows a user to: (a) automatically analyse images using image segmentation and features based on colour, texture, and shapes detected; (b) automatically process collateral text based on the frequency and collocations of keywords; and (c) automatically learn associations between image features and textual features. We looked at using the multi-net framework of CITU for emergency management by annotating unseen disaster related images with keywords to improve classification.
The advantages of using image and text features together to improve overall image classification has been highlighted in (Jing (b), Scotney et al. 2016). Using image and text modalities together the accuracy of classification can be improved for the application chosen in SLANDAIL, to classify and index emergency related messages (Table 3). Building on this work, the use of image annotation and self-organising maps to link text and image modalities to improve message classification is currently being prepared in (Kelly, Ahmad et al. In Preparation).
Table 3: Comparison of recognition accuracy and performance using self-organising maps for the image and text collections collected as part of SLANDAIL.
Image Only
Accuracy Image and Text
Accuracy
Minimum Maximum Minimum Maximum
General Flood Corpus 50% 51% 66% 68%
Flooded and Submerged Corpus 75% 80% 65% 95%
This approach of using all the available content of a social media message has meant overall recognition can be improved compared to just using an image or text alone. The annotation method can be trained off line and used in conjunction with the image classification and SIPF algorithm (Jing (b), Scotney et al. 2015) to perform real-time processing of social media messages and their image and text content.
We extended and tested the approach for recognition of flood-related events such as submerged cars and road-signs. This work was implemented in the SLANDAIL Social Media Monitor. This is a considerable achievement in terms of technology transfer in computer vision, enabling emergency managers to act on information extracted automatically from images that have been harvested from social media or other contributing sources. The software can alert emergency managers to a developing disaster and help them to focus in on and assess the most relevant information available. The prototype interface for emergency managers in shown in Figure 17.
Figure 17: SLANDAIL Image User Interface for emergency management: a) Images are categorized (flooding, fallen tree). b) Alert status of a geographical location - a rise in alert status (red line) merits attention; blue line is classification on an individual image basis. c) Images are then aggregated into various sub-categories and displayed as a pie chart - hover over or click various sub-categories to retrieve further information. d) A ranked image set displays the three most pertinent images relating to a selected category from the pie chart.
3.3.6. Multi-Criteria Decision Making
SLANDAIL developed a method of aggregation and decision-making that relies on fuzzy integrals and aggregation theory to bring together data typically used by emergency managers with outputs from social media. This includes a scoring mechanism that uses various pieces of relevant information, including vulnerability of an area, economic information and geological information. We have added social media analysis to this list of available data. This has resulted in the development and publication of a decision-making model that integrates social media messages with other information including geospatial information.
The SLANDAIL System uses geospatial information used in typical emergency management systems, with processed social media data (cf. section 3.4.2). It aggregates conventional data sources with advanced machine learning methods for image and text analysis. The output of this analysis is a signal that can be used to help decision makers make more informed choices based on the available information. Many open data sources exist that can be used as valuable sources of geospatial information: INSPIRE, for example, contains spatial data for all EU countries and is mandated to include this data, supplied by local authorities.
Using our model, specific regions or locations can be ranked to get a complete map of vulnerability by combining the available data and assessing the risk. An example of sample data variables has been presented in (Mesiarová-Zemánková, Kelly et al. Forthcoming), in which the output of the classification for social media data (which determines relevance and reliabililty) forms a social media index. Our method combines relevant variables in a way that reflects an expert’s decision during an emergency event. The weighted combination of the data results in a normalised output value representing the magnitude of risk, or severity, of the situation based on available data.
Figure 18: Our method determines the level of risk and vulnerability for a region by combining relevant variables: Geography (G), Accessibility (A), Infrastructure (I), Population (P), Current condition (CC), and Conditions in catchment area (CA).
Figure 18 shows the input variables for a region and their “score” as determined by the data passed into the model. The model aggregates data in three different ways, as illustrated with different models (AWBM, AW, A), giving an output and level of risk based on the input data. This analysis allows an emergency management system to offer information on key locations that might be high risk, based on a number of variables including social media.
3.3.7. Multilingual Aggregation
The SLANDAIL Prototype IV, originally created in Italian, was successfully localised into English and German. A key asset for completion of this task was the SLANDAIL Lexicon, which was used to translate approximately 20% of the messages of the interface. We identified challenges to localisation, mainly consisting in issues of interpretation arising from the fact that strings are presented as out-of-context chunks of text and the need to accommodate varying word lengths to the space available. The quality of this work was tested by both English- and German-speaking End Users, who provided feedback to ensure usability and communicativeness (cf. Section 3.4.4.3 for full description). Software localisation is an expensive and time-consuming task as it requires the collaboration of different experts at various stages of the work: by managing the entire process internally, we achieved cost-effective and high-quality results.
Natural disasters do not respect borders: the European Union has 24 official languages, and many regions that share disaster risk zones do not share a common language. A review of current emergency management systems showed that most are available in one language only (usually English) and no provision is made for porting them into other languages, while others are machine translated or localised through crowdsourcing.
Note: Use of background IPR: Software systems
3.4. CONTROL ROOM TECHNOLOGY: Building a Social Media-Enabled Emergency Response System
3.4.1. Introduction
In the final eighteen months of the project, SLANDAIL consolidated the methods and analysis developed earlier into an integrated system accessible via a single screen. The resulting prototype assimilates End User requirements as well as the findings from our ethical and social analysis, and enabled us to complete the third of our objectives: build and test a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management in an ethically and legally sound manner.
At a Project Development Meeting in Dublin in January 2016 attended by all partners, we assessed SLANDAIL’s progress to date and determined the necessary steps to be taken to deliver an integrated system (Figure 19).
This involved the integration of the Social Media Monitor with the Emergency Management System to allow them to share relevant information. End Users – who were closely involved in evaluating and testing the prototype – can now access aggregated data from social media sources within an emergency management environment.
We transferred the Image Analysis software, integrating the automated processing of social media images with the SLANDAIL System. End Users can now perform multiple actions within a simple, consistent and user-friendly environment. The integrated system enhances the situational awareness of emergency managers by leveraging geospatial informational resources together with information extracted ethically and legally from social computing systems.
Our innovation has been to interface a disaster management system to a social media monitor. Initially, we coupled the media monitor (Topic Analyst) to the EMS (SIGE). Subsequently, CID confirmed that they were unable to provide live social media data for testing and evaluation due to commercial and contractual reasons relating to German copyright law. In the latter stages of the project, we developed a new Social Media Monitor that incorporates a privacy intrusion detection system: this is used to supply text and image data to the system.
SLANDAIL synthesised research on trust into a set of communications guidelines for End Users. These guidelines are the outcome of linguistic and terminological investigations and provide emergency managers with essential information on how to generate and maintain trust with citizens before, during and after crisis events. Our research on non-verbal communication included an analysis of facial expressions and gestures – elements of communication that have a universal, language-independent valence – and will act as a platform for the creation of training tools for public figures required to speak in crisis situations.
The SLANDAIL System enables emergency managers to communicate information effectively during crises. Our crisis communication templates, based on End User evaluation, enable communication with citizens through the system. The templates are based on corpora collected in three languages and have been tested for clarity and readability. The system was localised in three languages and has been adapted to potentially localise in other languages using structured terminology databases.
The integrated emergency management system can be adapted to different geographical settings (it can incorporate different cartographic projections with no software changes) and enables End Users to perform a range of complex analytical actions within a user-friendly environment. It can successfully gather and process information from social media, aggregate and interpret the data according to relevant metrics, and circulate messages to citizens. As such, it presents a powerful new set of tools to reduce the existing burden of information processing upon emergency managers and enable them to respond more effectively to threats.
3.4.2. Integration of Emergency Management and Social Media Monitor
Figure 20: The work in other areas of the project, including user requirements, technical developments and legal licence, are all integrated to some degree in the final SLANDAIL system
SLANDAIL’s first prototype leveraged web-based information for use in an Emergency Management System (EMS). Existing software systems by DataPiano and CID were coupled to share relevant information regarding emergency analysis and disaster management.
3.4.2.1. Deployment of Emergency Management System for User Needs
DataPiano deployed its Emergency Management System, SIGE, in a dedicated server environment. SIGE provides crucial database information to emergency managers and includes a locative system and custom-developed GIS mapping system to assist End Users in finding key areas during a natural disaster. This emergency management system, used by Italian Civil Protection since the early 2000s, formed the basis for the SLANDAIL System.
3.4.2.2. Coupling of Emergency Management System and Social Media Monitor
In April 2015, the coupling of the Social Media Monitor (Topic Analyst) and Emergency Management System (adapted from SIGE) was demonstrated to End Users at a Usability Subcommittee Meeting. The first stage of coupling was the development of an API schema document to provide an input-output protocol (API) and define each partner’s role in the project, facilitating subsequent integration phases. This enabled the complete transfer of documents from Topic Analyst to SIGE (including metadata), and the filtering and searching of documents from SIGE via API in Topic Analyst.
We held a series of discussions on the needs of End User beneficiaries. During evaluation, End Users highlighted that social media is only one strand of information and cannot the core source. As such, our Social Media Monitor was developed to incorporate social media data into an existing emergency management system. These discussions led to the definition of requirements for the SLANDAIL system, and the meetings of the Technical Subcommittees led to the development of a SLANDAIL User Requirements Document, which shaped the design of the coupled prototype. The first step was the creation of a SLANDAIL-specific deployment of CID’s Topic Analyst with a CORPUS system, a Social Media Data Capture and Visualisation System (SMDCVS) in a dedicated server environment.
Topic Analyst provides an analytical instrument for filtering, search, investigating and data mining on huge amounts of documents collected and pre-processed by CORPUS back-end as well as academic partners. Its features are language processing, entity recognition as well as lemmatized keyword extraction. Together with its SLANDAIL-specific deployment, Topic Analyst was equipped with a GeoInformationService that offers the capability to reference geolocation coordinates on a given map as well as performing filtering functionality. Due to legal restrictions on the use of real-time text collection and analytics from Topic Analyst, the Social Media Monitor was used instead.
The first working, coupled social media monitor and emergency management system enables emergency managers to not only analyse social media during times of disaster, but also to regulate and manage their response to disaster notifications.
3.4.2.3. Deployment of Social Media Monitor in Line with End User Needs
The SLANDAIL Social Media Monitor amalgamates many of the technologies developed within the project. It combines spatio-temporal metadata of social media messages with an analysis of the complete content – namely, the text and images – of each message. The system enables the extraction of real-time information from social media stream by incorporating text and image analytics and geospatial mapping within a multimodal analysis framework (Zhang (a), Kelly et al. 2016). It is supported by text and image analysis technologies, described in detail in Section 3.3.
Utility: The system utilises sophisticated filtering mechanisms to ensure that the data extracted is relevant. For example, the Monitor collects data from a geo-constrained data stream, ensuring that disaster managers – who will be responding to emergencies within specific geographical regions – receive data relevant to their area (Kelly and Ahmad 2014, Kelly and Ahmad 2015). Additionally, the Monitor draws on several NLP (Natural Language Processing) techniques as well as language patterns identified by experts as being common to disaster events (such as those relating to road blockages and power outages) (Toutanova, Klein et al. 2003, Derczynski, Ritter et al. 2013, Chang and Manning 2014, Spyropoulou 2016) to filter out unnecessary information. The system also detects key words in messages containing images as well as using a pre-trained image classifier to detect flood images (Jing (b), Scotney et al. 2016). All this information is mapped onto GIS systems. The mapped areas highlight outlier information that may not be apparent from existing GIS information, such as flood preparedness levels or economic vulnerability. This allows it to take advantage of the full content of an image and maximise the effectiveness of its analysis (Figure 21).
Architecture: At its core, the SLANDAIL Social Media Monitor is powered by a concurrent data processing pipeline that applies the analytic algorithms to the social media stream (Section 3.3). We added a web application that implements the visualisation of geospatial mapping and provides End Users with an intuitive interface. The system contains a number of components, including: A Social Media Streaming Access Module (which connects to live social media streams); an NLP Pre-processing Module to filter incoming text; a knowledge base, the Semantic Disaster Lexicon, containing common language patterns relating to emergency events; and a Geospatial Mapping Layer that infers (and, when possible, detects) the geo-locational information of the message. The Intrusion Index is a text analytical tool that detects and visualises the extent to which sensitive information (such as individual names, institutions and places) is being collected (Kelly and Ahmad 2014, Kelly and Ahmad 2015).
User Interface: The interface of the Monitor is implemented as a web application and allows End Users to access the aggregated results of the analysis in a clear and intuitive way. Red markers indicate message with native geo-coordinates attached, while green markers represent message with locations inferred from text contents (Figure 22). Such capability makes it possible to reuse semantic dictionaries that contains language patterns that capture common themes across different types of emergencies. Users can view the details of individual messages by clicking these markers; a time-series plot allows an overview of events as they unfold. Evaluation of the Hurricane Matthew event (2016) showed the occurrence of power outage and road closure information in areas, often preceding other sources of information.
Figure 22: Front end of the social media monitor system showing filtered and relevant geolocated messages and images for the location under investigation, the frequency at which these messages have occurred, and a drill-down menu to examine the messages and accompanying metadata. The monitor can be configured to use different semantic dictionaries when tracking different types of events (e.g. flooding, fire, crowd control). Furthermore, the system allows the user to use multiple dictionaries at once; for example, the user may combine a semantic dictionary containing only language patterns related to flooding with another dictionary that tracks general sentiment words.
The SLANDAIL Social Media Monitor can be deployed online on a standalone web server, allowing it to be operated within an existing emergency management control room, and it can be viewed in mobile and standard web browsers for demonstration purposes. Much of the recent work of the Social Media Monitor has been summarised and published in a forthcoming paper accepted at the International Conference on Information Systems for Crisis Response and Management (Kelly, Zhang et al. 2017).
3.4.3. Building of Final Demonstrator
3.4.3.1. User Interface Design
SLANDAIL’s User Interface was designed with the primary requirements of being simple and communicative. We determined that End Users should not require specific technical guidance, and that navigation through the various functions of the program should be intuitive. Additionally, the nature of crisis operations demands that Users can perform every operation with the lowest possible number of clicks. The Dashboard contains large buttons, or tiles, that communicate their function through simple icons and is divided into three sections: Quick Start, Ongoing Incidents and Tools (Figure 23). These sections allow End Users an immediate overview of the current emergency situation.
The emergency management UI is structured as a “wall of monitors” featuring interchangeable panels that show essential information: the generic event data (type, location, etc.), operations carried out to date, resources or facilities involved in each emergency stage, messages sent, and documents relating to the event. These “monitors” are automatically updated and can be moved, closed or substituted with others (showing a list of messages sent or received, personnel available in a hospital) within the screen area.
Figure 23: SIGE’s dashboard is divided into three sections: Quick emergency, Ongoing Incidents, and Tools, each with user-friendly start tiles to allow End Users to quickly navigate the emergency situation.
The main interface is localised using a special text file containing terms, words and phrases used in interface command names and tiles: a new supported language can easily be added. This localisation of the software, a collaborative endeavour involving linguistic and communication expertise, ensures that SLANDAIL’s interface language is consistent with the language used by local emergency management operators. End Users tested and evaluated the interface and terminology to ensure its accuracy and communicative effectiveness.
End Users are required to accept the terms of the Licence Agreement when logging on to the system
(Figure 24). This mitigates the risk of infringement of the human right of the data subject or owners, the copyright of the data owners, and the relevant personal data protection and privacy provisions of the EU. The Licence highlights the fact that End User organisations have a responsibility to negotiate agreements to secure permission from the data owners (usually the social media providers) to collect and analyse the data, and a responsibility to provide training for users. It specifies when data should be archived off the live system and outlines the protocols that should be in place to provide for its appropriate handling by authorised persons prior to its timely destruction.
3.4.3.2. Incorporation of Text, Speech, Non-Verbal Communication and Image Analytic Systems
Building on the initial coupling of SIGE with Topic Analyst (D4.1) and the adaptation of the system GUI to End Users’ linguistic and cultural contexts (D3.5) we built methods for including advanced text and image data into our system (D4.2). Existing technologies were migrated to web-based technologies to enable multi-agency response, and integrated into a user-friendly interface. End Users can now perform multiple actions (including complex GIS queries) within a simple, consistent and user-friendly environment.
TEXT ANALYSIS: We integrated a topic modelling module into the coupled SLANDAIL system (D4.2). Social media data from Facebook and Twitter was processed to automatically retrieve the topics contained in the corpus and in individual messages. The data can be easily filtered, aggregated and combined with other results via the graphical user interface of the Topic Analyst module.
In addition, an API library, the SLANDAIL power outages, was developed; this library encompasses the incident mapping and intrusion detection methods based on the analysis of the textual content of social media messages. The methods detect the uses of certain language patterns from social media streams that often indicate the occurrences of certain key disaster incidents (e.g. road blockage, fallen trees, power and water outages, people getting trapped, etc.). The results of these analyses are organised into structured formats so that third-party applications can integrate with the API and gain access to social media analytics. This API library has been integrated into SIGE, enabling it to monitor social media streams within their system.
SPEECH AND NON-VERBAL COMMUNICATION: We successfully integrated the non-verbal communications module by connecting the Emergency Management platform with iMotions, a software system for video analysis, via its API. The integration was implemented as a web application, and is thus accessible to End Users through web browsers. The system is designed to provide users with a “one-button” experience when analysing videos through the iMotions software by automating setup and teardown processes as much as possible.
Users can upload videos for analysis to assess emotional communication before uploading disaster messages via Facebook or YouTube, for example (Figure 25). Two visualisation options then become available, displaying the dominant emotion class and the trending of different emotion categories. The efficiency of emergency and disaster-related message delivery is crucially based upon not only on verbal content, but also on non-verbal signals. This addition to the system, along with the communication guidelines introduced with D1.4 will aid emergency managers and public relations representatives in developing communication skills.
IMAGE ANALYSIS: SLANDAIL tested the transfer of Image Analysis software, completing integration in February 2017. We ported the image analytic algorithm into SLANDAIL’s Social Media Monitor, thus integrating the automated processing of social media images with SIGE’s emergency management system. Following this integration, SLANDAIL System is equipped to automatically recognise flood events, and to deliver an automated alert when the volume of these images reaches a certain point.
Each image receives a classification score, designating a flood or non-flood image. The amount of flood images determines the overall alert status of the system according to a three-level scale (Low, Medium and High). When the alert status reaches High, a notification is automatically sent to SIGE to alert the emergency manager to the presence of a large quantity of flood images. This allows the End User to create an active event tile, which aggregates all the live information relating to a geographical location.
A simple implementation of the social media monitor involves a comparison of the number of posts (including text and images) that contain emergency-related words or images. We have developed a more complex index based on individual scores for each post based on its level of relevance, credibility, and affect. A message’s relevance is determined by analysing its content to detect textual references to an emergency event or images recognised by the classifier. With the credibility score, heuristics drawn from the literature (AlMansour, Brankovic et al. 2014, Alrubaian, Al-Qurishi et al. 2016) can assess whether the post is from a credible source. This helps determine how reliable the message is likely to be. Sentiment analysis techniques can be used to compute affect, giving an indication of a message’s urgency. The interaction between these inputs and their aggregation into an individual post score is modelled using the Choquet integral (Choquet 1954). The overall aggregation of outputs from social media content filters out most messages, identifying only the most relevant and reliable.
3.4.3.3. Trust-Building and Communication
SLANDAIL research on trust-building synthesised findings from several research strands (D1.4 D2.3 D3.2). We produced guidelines for End Users on how to communicate in a trusted manner across a range of platforms before, during and after a disaster event. The value of messages being sent out by emergency managers is dependent on clear and relevant communication, necessitating a strong communications plan. Our research shows that trusted communications are established before, during and after a disaster, forming part of a continuous dialogue between emergency managers and the public (Busa, Musacchio et al. 2015).
Linguistic and terminological investigations of trust communication revealed that: (1) clear and simple messages contribute to successful bonding with the public; (2) trust is first established in times of peace, when the public can become acquainted with terms used during disasters; (3) informing, reassuring and appealing to the public ensures that people will feel safe and well-informed during an emergency; (4) “emotional contagion” (Quasthoff, Goldhahn et al. 2013) occurs when individuals are exposed to emotionally loaded content: hence, transmitting a balanced perception of risk can engender trust (Hodas, Ver Steeg et al. 2014, Kramer, Guillory et al. 2014, Musacchio 2014). Agencies need to devise effective and timely messages anchored on shared values (Musacchio and Panizzon 2017).
SLANDAIL gathers basic, language-independent gestures to complement affective information from facial expressions and speech analytics. In emergency situations, it is important that authority figures use non-verbal language that: 1) does not show emotional involvement that could spread alarm; 2) is not misunderstood by people of different cultures; 3) shows professionalism and conveys trust in the public. Non-professional behaviour or speech patterns may reduce the public’s incentive to act in a disaster situation.
Our work also showed that for public communication: (1) Speech cues (pitch, intensity and speech rate) must be modulated to convey credibility for successful trust building; (2) These should be coordinated with facial expressions, body posture and gestures, and the general purpose of non-verbal language should convey respect and trust.
3.4.4. System Testing
3.4.4.1. Test Data and System Testing
Continuous cooperation between SLANDAIL beneficiaries was achieved through Usability Sub-committee meetings, End User requirement reports, and presentations on existing data management systems in End User locations. To ensure the functionality and effectiveness of the SLANDAIL prototypes, End User partners provided test data and created case studies. As well as End Users influencing and subsequently testing the prototype system, two independent tests of the system were carried out by emergency planners representing the Business Continuity Institute and the Emergency Planning Society to validate the tests, with support from Media Foundation of Sparkasse Leipzig (Figure 26).
A final comprehensive evaluation event was hosted at Trinity College Dublin on 13th September 2016, allowing End Users a further opportunity to test and tweak the prototype. While localisation configuration will be required in advance of real-life application, the system’s functionalities were fully demonstrated and understood. End Users reported satisfaction with the abilities of the system and were impressed with its potential for use in emergency management. We determined that – subject to a separate examination of ethical and legal impacts – the system has high potential for use as an Information Management tool in other End User scenarios, such as manmade disasters and major event management. The results of these tests facilitated further development of the prototype. The findings regarding further potential uses for the system stimulated commercialisation discussions, as well as flagging other potential R&D opportunities.
3.4.4.2. Building Clarity for Communications
For SLANDAIL, the issues of Plain Language – a pan-EU and national aspiration – and effective communication are central. Surveys on existing communication guidelines and protocols and interviews with End Users were coupled with conclusions gained through corpora analysis and terminology work. The aim was to develop a methodology for automatic assessment of key metrics for measuring readability and comprehensibility. This allowed us to compare a variety of language registers – in major EU languages (EN, DE and IT) – and use representative corpora to understand usage.
This multilingual study, the first of its kind to our knowledge, has led us to develop templates for releasing information about major disasters in line with plain language requirements through a range of media including text message, Facebook, Twitter and press releases. End User beneficiaries all identified a need for model messages drafted to suit the requirements of different media and different stages of an emergency. By studying the structure of warnings in a range of disaster management documents, we generated models to integrate prototypical communications into the SLANDAIL system. These messages were designed and assessed to achieve a high level of readability and acceptable lexical density.
Based on this analysis we created prototype messages in English, German and Italian and adapted these for presentation in different formats (e-mail, Facebook post, text message and Tweet). SLANDAIL provides End Users with ready-made templates adaptable to specific incidents. These can be used as a base to draft communicative and effective texts when time is of the essence. The structure and language chosen for templates have been tested for communicativeness against the source texts. English messages have been successfully tested through the Plain English Campaign software, and results suggest that our messages can be read and understood by a wide audience across various education levels in each language.
3.4.4.3. Localizing SLANDAIL’s Outputs to Multiple Languages
Following professional standards published by localisation experts, the system GUI was successfully and cost-effectively translated into English and German by adapting its textual elements to End Users’ cultural backgrounds (D3.5). SLANDAIL terminology, the SLANDAIL corpora and the INSPIRE webpages were used to localise the SLANDAIL prototype. We achieved key results in expanding the multilingual communicativeness of the prototype system by collecting and investigating corpora in three languages covering both formal and social media. The knowledge gained from metrics along with the assessments of communications experts and End Users has led to the creation of actionable information – crisis communication templates – that can be integrated into the SLANDAIL system and sent out to warn and inform the population.
Figure 27: The SLANDAIL System localised from Italian (left) into German and English (right)
Potential Impact:
4. Potential impact and main dissemination activities and exploitation of results
4.1. Contribution to the State-of-the-Art
Social computing systems carry an enormous volume of relevant and timely information, and have proven to be a useful method for communicating when traditional lines of communication are unavailable or inoperable. There are strong indications that social media is playing an ever-increasing role in the management of emergencies, and can play a key role in bridging the gap between traditional media reporting and adequate resource deployment (Young-McLear, Mazzuchi et al. 2015).
When integrated with geo-location systems already used by emergency managers, information harvested from social media systems can be a vital resource for the coordination of emergency response. A combined geospatial information system and social computing system in emergency response will be the next step in the evolution of disaster management, allowing citizens to participate in the processes of gathering and sharing vital information (see, for instance, Maresh-Fuehrer and Smith 2016). Research to date has identified the potential for pilot programs and public-private partnerships to integrate social media usage into disaster management (Lindsay 2016).
SLANDAIL research has laid the groundwork for such a combined system, delivering advances in informational analytics tailored to the emergency management environment. Our work addresses crucial ethical, social and legal issues surrounding the use of social media in disaster management, and the findings from our cross-disciplinary analysis of these factors are embedded in the SLANDAIL System. Contributions have been made to text analytics in the areas of topic modelling, ontology, and terminology, time series analysis using social media as an input, image analysis, in to aggregation studies. We have also extended the state-of-the-art in the ethics of social media use.
The System and its accompanying platforms have introduced ethics into data collection approaches in novel ways, informed academic approaches to systems development with real-world End User experience, and helped emergency managers to incorporate technical and ethical expertise into their practices.
4.2. Key Achievements
4.2.1. Legal Licence Agreement
Ethical and social considerations were embedded in SLANDAIL from the beginning of the project, and our legal and ethical research had considerable impact. It identified measures necessary to the protection of data privacy for citizens, which then took the form of technical functionality implemented in the prototype. The research undertaken in this strand of the project culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. SLANDAIL legal caucus led work to develop a software license suited to systems that bridge the divide between privately-generated content and public (or commercial) re-use, and where ethics and provenance are both relevant and important. With EU Data Protection Legislation soon to be implemented (de Hert and Papakonstantinou 2016), issues of privacy and data protection are emerging as key ones in determining future policy in relation to the use of social media in emergency management, and SLANDAIL advances have made significant contributions to the ethical and legal landscape of emergency management.
4.2.2. Terminology Management
SLANDAIL has made terminology management as automated as is currently possible. We completed the development of a terminology wiki, evaluation and refinement of automated term extraction (Musacchio, Panizzon et al. In Preparation), and the development of a method for automatic extraction of definitions, contexts and ontologies. Key results lie in the use of terminology for localising user interfaces broadly (Schmitz 2015). The development of terminology and the creation of ontologies has contributed to avoiding a key knowledge acquisition bottleneck. The SLANDAIL lexicon is currently being integrated into the IATE (InterActive Terminology for Europe) database, the EU’s terminology database, in line with guidelines provided by TermCoord, the EU’s terminology coordination database. This will make SLANDAIL’s terminology in all three project languages available to European institutions.
4.2.3. Integration of Image and Text Analysis into Control Room environments
The project has developed methods to integrate image analysis with the results of text analysis and has built a solid foundation for multimodal information aggregation. SLANDAIL goes beyond the state-of-the-art by developing an image recognition framework that automatically recognises specific disaster-related images. The framework also maximises the use of rich information (such as metadata) by integrating text analysis with the image analysis system to enhance the performance for retrieval of flood-related images. The image processing framework implemented in SLANDAIL for image analysis uses a newly developed SIP (Square-Spiral Image Processing) architecture that enables fast computation of image convolution. The SIP framework can be incorporated into standard image processing systems, making the image analysis system extensible.
Additionally, the SLANDAIL System creates a multi-modal feature vector based on image features and results of text analysis. The fusion of text and image features also enables the development of a higher-level system involving more complex ontologies for multiclass image classification and annotation (Kelly, Zhang et al. 2017). The image analysis is extended to the analysis of video data (for video content retrieval). The final platform can incorporate the image analysis outcomes for geo-located and timestamped live social media data analysis.
The integration of text analysis from social media into existing disaster management control room systems is a major impact. The development of a live Social Media Monitor that performs advanced text analysis on large bodies of social media text provides analysed data in real-time. Furthermore, the flexibility of the analysed results to be mapped onto GIS or similar mapping systems allows for the visualisation of text analytic results in real-time during a disaster event. The Social Media Monitor developed on the project is designed to work with any standard control room operation using GIS models and will be further exploited beyond the end of the project to enhance control room technologies in emergency management.
Furthermore, in Natural Language Processing research and crisis informatics (Sorensen and Sorensen 2007), we found that topic modelling techniques were suited to social media analysis in disaster management. Well-trained models for flood-related Twitter and Facebook data provide a means to automatically filter, sort and analyse social media data, and represent powerful tools to cope with the problem of information overload in disaster management application and research (Kelly and Ahmad 2015). Using the SLANDAIL System, emergency managers can monitor prevalent data trends and patterns to see the different communicational phases of the disaster event life cycle and link these to the different phases of a disaster event: preparedness, response, mitigation and recovery.
We focused on the extraction of actionable information from live social media traffic during emergency events. Such information includes knowledge about incidents that may influence decisions made by responders when dealing with emergencies. The analysis potentially enables End Users to make better plans as to where to dispatch resources and rescue services in addition to their existing resources.
4.2.4. Improvements in Disaster Management Communication
Disaster management agencies are increasingly adopting social media as a channel through which to disseminate information and coordinate recovery efforts. Recent research has found that communication – in particular, social media communication – has a positive impact on disaster preparedness, mitigation, and recovery (Westerman, Spence et al. 2014, Brengarth and Mujkic 2016).
SLANDAIL developed a method to analyse, translate and produce messages that can be applied in languages other than the project ones and extended for use in other emergencies (Musacchio and Panizzon 2017). The SLANDAIL System GUI has been developed with the potential to port the system of analysis for use in other languages quickly and cost-effectively. The emergency management systems we reviewed exist in one language only (usually English) and no provision is made for porting them into other languages. This brings localisation beyond the state-of-the-art as it is an example of “glocalisation”: the original GUI language was a “local” language, Italian, which was then “globalised” into English and finally “localised” again into German.
The development of templates for communicating disaster messages across a variety of media, including social media, is a further impact that can be used in future emergency planning (Musacchio, Panizzon et al. 2018 (Under Review)). These templates, which can be sent automatically from the SLANDAIL System, are customisable and contain recommendations of good practice for communication between emergency management teams and the public.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our work on non-verbal communication analytics will provide the blueprint for a platform that can accomplish real-time analysis of videos retrieved from the web and detect emotive content by multimodal integration (facial expression, speech, gestures analysis). Research papers leading on from the project in non-verbal communications and disaster communications will soon be published (Musacchio and Panizzon 2017, Musacchio, Panizzon et al. 2018 (Under Review)).
4.3. Dissemination
4.3.1. Overview
SLANDAIL is addressed towards the universal real-world need of all citizens for effective emergency response in cases of natural disaster. The project also targeted improvements in areas of specific occupational expertise and delivered advances in a range of academic fields. As such, the results of our work are of interest to a range of audiences, both specific and general.
We undertook a comprehensive dissemination programme to communicate the mission, progress and results of the project to specific target audiences. At the beginning of the project, we developed a dissemination strategy comprising multiple communication plans: this was designed to communicate with six key audiences (first responders, industry, the public, advocacy and representative groups, the media, and other related projects) through carefully-crafted key messages.
Our dissemination and communication activities throughout the project ensured that both message and channel were optimised for the audience(s) to be reached. Unlike many Information and Communications Technologies (ICT) and security projects, our primary target audience was not solely technologists, scientists and researchers: of equal importance were first responders and security agencies (police, civil defence, emergency response teams), as well as ethicists and legal experts.
Our strategy included the creation of the SLANDAIL website and the SLANDAIL Disaster Newsletter to create public engagement as well as targeted press releases, as well as a digital magazine to communicate with interested stakeholders and create demand for the information and the analytical systems developed in and after the project. The project attracted attention in the mainstream news media as well as in publications dedicated to business and technology (details found in Appendix A2). Publications in peer-reviewed journals have been a key part of the SLANDAIL dissemination strategy. We have delivered several publications in high-ranking journals to date (listed in Appendix A1). SLANDAIL members have presented research at many international conferences, ensuring that the project achievements were communicated in a range of academic fields – scientific, humanities, and legal. Project coordinator Prof. Ahmad has been appointed to the International Editorial Board of the University of Padova’s journal Peace Human Rights Governance, and a book on the topic of Dignity, Privacy and Survival in relation to the use of social media in disaster management is being prepared for submission to a major publisher (Ahmad In Preparation).
To identify, manage and protect the intellectual property rights relating to the project technologies, SLANDAIL developed an IP Protection Plan. We produced a Commercialisation and Exploitation Plan to maximise impact and research value and achieve the second of our Exploitation Objectives (“map out commercial and research development of the project results in the future”). Advanced conversations are ongoing in the potential to spin out a company from SLANDAIL technologies, offering social media services in emergency management contexts.
The collaborative, multidisciplinary nature of SLANDAIL has been one of the projects’ key strengths. We created a Virtual Research Community to enable the continued cooperation of End Users, experts in academic and legal fields, and technology partners beyond the close of the project.
4.3.2. Website Design & Implementation
At the beginning of the project, SLANDAIL launched a website (www.SLANDAIL.eu) (D6.1). This was built upon the open source content management system WordPress and displayed a brief introduction to the project along with an attractive set of icons to illustrate the various emergency situations that the project will consider. The site was responsive and linked to SLANDAIL’s social media accounts. Throughout the project, the website was updated with new project content, news and information about project activities, and publications.
Through the website, users can also access the SLANDAIL Newsletter, which aggregates news on disasters as well as providing automatic news analyses. The Newsletter visualises the results of the Summary of High Incidents (SHI), which harvests and analyses data from selected sources. The result is a monitoring system which is updated daily according to SHI-provided results, and can be publicly surfed by a wide audience, contributing to both the dissemination of SLANDAIL outputs and the detection of potential early warnings for emergencies (See Section 3.3.2.1 for more information).
We have an active social media presence, both by presenting our own work and by selecting and highlighting relevant social media contributions by others that demonstrate, illustrate or complement key points from the project. As a result, our Twitter following has increased by over 100% during the second half of the project. Our Facebook presence also continues to grow, with periodic posts and updates.
4.3.3. Media Management
The multi-disciplinary nature of SLANDAIL, and the way we address topics of significant public interest, has meant that the project has enjoyed significant media interest and coverage. SLANDAIL created a logo, a strong web presence (website, Twitter, Facebook), the SLANDAIL Disaster Newsletter, a terminological database (wiki), released a series of press releases (modified for multiple partners), and built a mailing list which was then used to distribute our digital magazine (to serve as a brochure for interested stakeholders) (Figure 29).
Figure 29: The SLANDAIL Magazine communicates the mission, progress and results of our work, serving as a brochure for interested stakeholders.
The project had a dedicated media contact in the Press Office of the Faculty of Engineering, Mathematics and Science at TCD, and the team was available for media interviews and other interactions with the media on request. The project has received coverage in The Sunday Times, The Journal.ie (Ireland’s leading online newspaper) and TechCentral. In the business media, the project carried out knowledge transfer and gave presentations to the Business Continuity Ireland Forum (Dublin), and delivered the Emergency Management Research Symposium (Dublin). A large editorial feature on the SLANDAIL system featured in the 112 Emergencies magazine, 2015, which has an emergency management circulation of over 30,000. A final press release for project results is currently being prepared for circulation.
The project was active in academic media, with a range of publications and presentations that reflects the multi-disciplinary nature of the project and the timeliness and relevance of the subject matter. Examples included presentations at the DIMPLE workshop in Reykjavik (where the project also contributed to workshop organisation and editing of proceedings), UCAml in Belfast, SWDM15 in Florence, TKE2014 in Berlin, COSMIC FP7 final Conference in Brussels, the EU Social Media and Crisis Management concertation event in Brussels, ISCRAM in Kristiansand, Norway, at ICIP 2015, Quebec, the 20th European Symposium on Languages for Special Purposes in Vienna, IAPR MVA, NLP 4CMC/GSCL in Essen, Germany, I4CM DRIVER in Berlin, MUININ 2016 and LREC, Portorož, Slovenia.
SLANDAIL presented the project to various emergency management groups. The project’s research and results were presented to An Garda Síochána Major Emergency Planning Unit on December 8th 2016. A follow-on presentation was given at the Cross-border Working Group on Major Emergency Management meeting, Town Hall Enniskillen, on January 25th, 2017 (Figure 30). This meeting was attended by principal response agencies from Ireland and the UK, including all those responsible for dealing with major natural disasters: Police, ambulance and fire services, local authority, and members of Ordnance Survey Ireland.
A video documenting the project results was created in early 2016. It was developed by professional filmmakers to clearly disseminate the positive results of the project and circulated to End Users and other interested parties via email, as well as uploaded to social media channels. A second video is currently in post-production, documenting the successful development of the Social Media Monitor.
Figure 30: Shane Finan, Xiubo Zhang (TCD) and Rebecca Bury (Stillwater) presented the project to an international meeting of emergency planners at the Cross-Border Major Emergency Planning meeting, January 25th 2017.
4.3.4. Links with Researchers and Other Projects
The strong networks of the project consortium was exploited throughout the project. In addition to conference and journal publications, SLANDAIL organised conferences, symposia and workshops. Such events provided excellent opportunities to discuss topics of shared interest with our peers from other EU projects and from the wider scientific, End User and legal communities. Events for which SLANDAIL had full or shared responsibility, or made several presentations, include the following:
▪ The MUININ workshop on technology and emergency management. This was part of the ARES International Conference on Availability, Reliability and Security, held in Salzburg, Austria, in August 2016.
▪ The EMOT workshop (Emotions, Metaphors, Ontology and Terminology), explored aspects of trust building and effective communication in emergencies, particularly the use of social media. This workshop (in May, 2016) was part of the Language Resources and Evaluation Conference (LREC), at Portorož, Slovenia.
Figure 31: SLANDAIL researchers in terminology, ontology and disaster communication (UNIPD) at the LREC 2016 Conference at Portorož, Slovenia. L-R: Virginia Zorzi, Raffaella Panizzon, Alice Cravotta, Maria Teresa Musacchio, Maria Grazia Busá.
▪ Preparing for floods was the focus of an industry-facing event held at TCD in May 2016, in collaboration with the Business Continuity Institute and the Emergency Planning Society. Speakers included SLANDAIL research staff and Irish emergency planning agencies, and attendees represented public bodies, industry representatives, emergency response agencies, insurance companies, banks and technologists
▪ SLANDAIL’s outreach to other EU projects involved a contribution to the Innovation for Crisis Management (I4CM DRIVER) conference in Berlin, in December 2015. We shared experience, best practice and insights with peer projects DRIVER, SecurePART, TACTIC, SecinCoRe, EmerGent, EPISECC, SALUS and FORTRESS.
▪ SLANDAIL hosted the DISONTOLOGY workshop at the 20th European Symposium on Languages for Special Purposes (LSP, Vienna, July 2015). This highlighted the importance of clarity and effective communication in an emergency context, particularly in multi-ethnic and multilingual environments (such as many major cities), and explored the challenges inherent in multiple different specialist terminologies in emergencies.
The project has a high profile across the security and emergency management research domains. We will leverage this profile when building new research consortia to address emerging challenges and to encourage researchers from across Europe to engage with us, use our tools and resources, and contribute to shared scientific progress and social benefit.
4.4. Exploitation & Future Planning
SLANDAIL has developed an IP Protection Plan that shows how each beneficiary will own their own IP as well as how the overall project IP can be commercialised by beneficiaries. These technologies are primarily the work of the academic partners (TCD, INFAI, UNIPD, ULSTER), but have benefited from the inputs of the commercial partners (CID, DataPiano, Stillwater) and the End User representatives (GARDA, PSNI, BLS). The intellectual property rights are primarily vested in the relevant academic institution(s), but the wider team is committed to a policy of facilitating reasonable and fair use of IP based on fair commercial agreement.
• TCD holds the intellectual property rights to the social media monitor
• TCD holds the intellectual property rights to the Intrusion Index
• INFAI holds the intellectual property rights to the topic modelling system
• UNIPD holds the intellectual property rights to the multilingual terminology wiki
• ULSTER holds the intellectual property rights to the “Squiral” image processing technology and their image processing system
Ownership reflects the Consortium Agreement, as well as the reality that SLANDAIL builds on many years of partner R&D in these and related fields, so that the background IP contributed by each partner is tightly coupled to the new foreground IP.
Each of these potential assets runs autonomously, either online or on a local PC: they can be “mixed and matched” as required for any given application. The development of these tools helps existing systems go beyond the state-of-the-art. Each has been developed as a “standalone” system, but their outputs can feed into SIGE or Topic Analyst or into other third party systems as required. This increases their potential for exploitation, in both commercial and scientific settings. The “products” that the academic partners offer can thus be software libraries, or access to web-based services.
4.4.1. Commercialisation
SLANDAIL partners have explored the potential for commercialisation of research developed during the project. Strong commercial partnerships have been formed for future work beyond the scope of the three-year run-time, including a partnership between the coupled technologies of SMEs DataPiano and CID.
Further relationships between academic partners and SME partners have been formed. INFAI have a long-standing relationship with CID, and continue to provide methods and tools for research. ULSTER and TCD have integrated text an image analytic tools with DataPiano’s SIGE system: these integrated tools can work as standalone platforms to be offered as part of a sales package to emergency management teams.
Figure 32: The Fifth SLANDAIL Plenary Meeting (at which all eleven project beneficiaries were represented) was held in October 2016 in Berlin.
TCD have explored the opportunities available through Irish funding bodies to create a company from the research efforts created in text analytics and event management through social media. Meetings have been held with government agencies (European Union funding agencies, Enterprise Ireland, Department of Foreign Affairs), university funding opportunities (KIC funding, Atlantic Bridge) and venture capital (Frontline) to explore the potential for commercial outputs from the research developed. Negotiations with two of these bodies are in an advanced state, and the potential to start a company that brings some of the research outputs forward will be realised later in 2017.
4.4.2. Exploitation Workshop
A Commercialisation and Exploitation Workshop in May 2016 by TCD led to initial plans on exploitation of the SLANDAIL System. This one-day event involved the three commercial partners (CID, DataPiano and Stillwater), as well as commercialisation and spin-out experts from the Irish national agency responsible for the development and growth of Irish Enterprises in world markets (Enterprise Ireland), and the coordinator at TCD. At this workshop the SLANDAIL system and advanced components were demonstrated to Enterprise Ireland (EI), the Irish government organisation responsible for the development and growth of Irish enterprises in world markets. EI gave feedback on potential target markets for SLANDAIL software, and showed interest in the possibility of TCD software being engaged in spin-off opportunities through the university.
A detailed analysis of the conversations and conclusions from the workshop was presented in D6.3. In summary, the key points are as follows:
▪ The most important commercial output of SLANDAIL is the coupled emergency management and social media monitoring system (D4.1). This combines two commercial product lines – DataPiano’s SIGE EMS and CID’s Topic Analyst social media analysis platform. Both individual systems have also benefited significantly from the project, with a shift to cloud-based architectures, greater architectural flexibility and modularity, and (in the case of SIGE) a new user interface paradigm.
▪ New capabilities can be added to the integrated system, based on the technology modules developed by academic partners. These include image social and legacy media monitors (from TCD), topic models (from INFAI), Multilingual terminology support (from UNIPID), image processing (from ULSTER), and intrusion index calculation (from TCD). Such modules can be added on a case by case basis, to meet the needs of specific customers.
▪ License fees will be payable to the relevant academic partners by the commercial partners, where the academic modules are added.
▪ There are promising markets in both the private and public sectors for the integrated system, with and without additional modules.
▪ A spin-off company opportunity is being explored by TCD to add another avenue of exploitation to its media monitoring and intrusion-index technology.
▪ Professional services and training represent a further potential revenue stream, and one that might support further development effort.
Figure 34: SLANDAIL beneficiaries with members of Enterprise Ireland at the SLANDAIL Exploitation Workshop, 25th May 2016, Lloyd Institute Dublin.
4.4.3. SLANDAIL Virtual Research Community
At the close of the project, SLANDAIL presented a Virtual Research Community Roadmap and Joint Research Action Plan (D6.4] a document that explores the legacy of the SLANDAIL project and considers how the multi-disciplinary relationships and perspectives of the consortium are preserved and new collaborations facilitated.
This Roadmap was based on the concept of a “virtual research community” – a collection of organisations with a common vision of pursuing multi-disciplinary research in ethics, technology, law, media and public safety. The community includes all members of the SLANDAIL consortium, but can be expanded to add new members with new skills and resources as required. Members sign up to a “Memorandum of Understanding” that states their willingness to work together and to contribute key resources if a suitable opportunity arises.
We developed this collective vision by looking at the future research domains that each partner is particularly interested in developing. For all partners, SLANDAIL is a stepping-stone in longer-term research, product development and service agendas. By identifying commonalities and complementarities in these areas (some, of course, closely aligned with our work in SLANDAIL), we derived a set of common research areas in which we see opportunities for further collaboration – the Joint Research Action Plan. This ensures that the collaborative relationships established during the project endure and serve as a foundation for further research and development.
List of Websites:
The SLANDAIL website is at http://SLANDAIL.eu/
The Coordinator of the Project is Prof. Khurshid Ahmad (The Provost, Fellows, Foundation Scholars & the Other Members of Board of The College of The Holy & Undivided Trinity of Queen Elizabeth Near Dublin): kahmad@cs.tcd.ie
Citizens regularly communicate through social media computing systems, particularly in times of rapid upheaval. During major emergencies or events the sharing of information (for example, infrastructural warnings, details of threat levels, and transportation advice) becomes an urgent necessity. As a result, a large volume of information relevant to emergency response is being generated in real time across social computing networks in a variety of new forms.
These developments mean that hierarchical models of communication between institutions and citizens are being challenged, with information now flowing in multiple directions through what have become known as “backchannel communications” (Cardoso 2008, Lindsay 2016). In situations of natural disaster, the scale and speed of events can threaten to overwhelm the capacities of emergency managers. By exploring the potential of modern computing systems to incorporate real-time feedback from the public, and by viewing the citizenry as a powerful, self-organizing, and collectively intelligent force (as Palen, Anderson et al. 2010 have influentially argued we should), we can add new tools to those currently available in disaster response. However, technical constraints as well as the complex ethical and legal issues surrounding data collection mean that emergency managers are currently unable to aggregate and systematically utilise this information.
SLANDAIL navigated the complexity of computing, ethics, the law and communications to create an aggregated and integrated system. Operational emergency managers were introduced to legal and ethical principles, while academic researchers developed modules in tandem with SMEs working in data analytics, emergency management and communications. The complexity of roles of active partners was carefully managed throughout the project, and disparate areas of research were carefully combined to create a working Emergency Management System (Figure 2).
At the time of SLANDAIL’s inception, the integration of social media analysis into disaster response processes was rare. SLANDAIL addressed this problem by equipping emergency managers with analytical tools to enhance response and planning capabilities during natural disasters, as well as with communications guidelines and legal and ethical principles for harvesting data and protecting citizens. The project combined expertise and resources in social media, security, ethics, law, emergency management, communication, text analytics, and image analytics, to deliver a multi-disciplinary solution to a challenging, real-world problem.
The system uses location-based geospatial information combined with processed social media data. The overall objective, in the context of security under the FP7 Framework, is for the project to help increase the security of individuals and groups living in areas affected by natural disasters by adding social media to sources of information available to emergency managers.
The project has been designed in response to requirements drawn up by control room teams that manage disasters and emergency events. End Users of the system have been closely involved in the design, testing and evaluation of each strand of the project. SLANDAIL partners collaborated to build and test a prototype system with due regard to ethical and factual data provenance. The SLANDAIL System aims to remove the burden of search and interpretation of the latent information contained in social media data from End Users.
Finally, SLANDAIL followed a Commercialisation and Exploitation Plan to further develop research outputs.
2.2. Glossary
Title Definition
SLANDAIL System An integrated system using emergency management and social media
SLANDAIL Social Media Monitor A platform that monitors social media live using advanced text and image analysis tools developed through project research
SLANDAIL Licence Agreement A licence agreement for use of social media data in emergency management systems that takes into account research in four areas of law: Data protection, copyright, IP and Human Rights
SLANDAIL End Users Emergency management teams and control room operators, from police and army to civil protection agencies, with operational experience in managing major natural and man-made disasters
2.3. Objectives
# Objective Achievement
1 COLLATE and REVIEW the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property Emergency Managers were involved throughout system design and testing. Research completed in the first six months outlined the usability requirements for End User beneficiaries. Test data collated as part of specific emergency related case studies was used to perform system tests.
2 CREATE protocols to protect the rights of the citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens The project created an ethical framework for using data in an emergency and followed this with preliminary legal guidelines and recommendations. This culminated in the development of an innovative model licence agreement for a social media analysis system for disaster response.
3 BUILD AND TEST a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management Built and tested a working prototype that combines emergency management systems with social media. As the project developed, state-of-the-art methods (text, image, speech/non-verbal communication) were introduced, tested and evaluated, leading to the creation and testing of an integrated system.
2.4. Achievements
Approaches to exploiting social computing systems to enhance the management of public crises have been co-evolving organically alongside the media themselves (Hughes and Palen 2009, Shklovski, Burke et al. 2010, Acar and Muraki 2011). SLANDAIL integrates these approaches and methodological reflections and explores how they can best be directed towards the development of a system for using social media in a disaster management context, focusing on improving communication during disasters and addressing the ethical issues involved. The project navigated the complexities of analytics, emergency management, ethics and communication. We successfully deployed, tested and evaluated a Social Media Monitor that analyses social media data in real-time, using advanced analysis in text and image. This was integrated with an Emergency Management System that is supported by a Licence Agreement with communication functionality to enhance End User capabilities.
Text analytics: Existing methods of text mining have been designed for formally written texts. Social media communications tend to be characterised by different structural and linguistic properties. SLANDAIL worked to develop methods of text analysis that would be effective in identifying and extracting relevant information from spontaneous communications in social computing systems.
Terminology (disaster-specific terms) and ontology (a defined set of concepts in emergency management) were identified as key research areas. These contributed to the design and build of the SLANDAIL Social Media Monitor and localisation of the SLANDAIL system to include all three project languages (English, German and Italian). Our methods of language processing and of terminology management can be exploited during operations to facilitate communications when time is limited and stress may limit operators’ ability to communicate effectively.SLANDAIL combined models of and investigations on the language of affect (Martin and White 2005), sentiment analysis (Ahmad 2011), disaster communications (Haddow and Haddow 2014) and the language of social media (Zappavigna 2012) and adapted these to the study of the language of affect in formal and social media in English, German and Italian.
Image analytics: Images have to be processed in a timely manner and as close to real time as possible. SLANDAIL research involved studying images and videos, often of inferior quality, made and posted by citizens affected by disasters. We deployed an image recognition system for detection of images related to flooding and events related to flooding. The results of our controlled experiments using degraded images indicate that the SLANDAIL system can still produce acceptable results when all images are degraded, giving it a broad application in practice. We have produced a system in which the outputs of text analysis are integrated with image analyses at the feature level to enhance the informational content. Furthermore, we tested a video analysis system for shaky camera footage in disaster zone areas, evaluated by End Users.
Integration of an emergency management system (EMS) and social media monitor: At the outset of the project, two separate systems existed that each performed complex tasks – CID’s Topic Analyst and DataPiano’s Sistema Informativo Gestione Emergenze (SIGE). Topic Analyst, a tool for the monitoring and analysis of textual data, was capable of processing documents in volume from the internet and providing in-depth analysis. SIGE, an Emergency Management System (EMS) and the base program used to build the SLANDAIL System, was capable of retrieving information on ongoing disasters and assisting emergency management communication by accessing databases of key personnel. However, a system that integrated both capabilities, as well as incorporating image analytical capabilities, had the potential to deliver numerous benefits, including real-time information and “eyes on the ground” information from citizens.
Early on in the project, we loosely coupled these existing systems to prove feasibility. Due to limitations in German law and contractual agreements concerning CID, we were unable to use live data from their system. So, we employed and integrated a Social Media Monitor in line with End User needs to enhance emergency management control room capabilities. Using multi-modal aggregation we tested a method for increasing decision-making capabilities using different signals of information, including disaster-zone information aggregated with social media data.
Ethical and societal awareness: Data gleaned from social computing systems is characterised by an inherent tension between mass and personal communication, since the informal and local nature of much of the content can sit uneasily with the global, open-access nature of the platforms on which it is shared. SLANDAIL sought to balance the needs of emergency managers and first responders for an effective and flexible system with the rights of citizens to privacy and data protection.
SLANDAIL examined the societal impact of social media in a security context and outlined an ethical framework for use in disaster management. This framework sets out specific measures to be taken in the technical design, licencing, administration, governance and use of the system. Given that there are various controversies on government surveillance relating to the use of social media (Bertot, Jaeger et al. 2012, Bekkers, Edwards et al. 2013), existing disaster management literature needs clarity as to the ethical validity of collating large amounts of personal data when people are at their most vulnerable.
Legal research culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. This agreement enabled the project to pursue the objective of ensuring the ethical harvesting and analysis of social media data to improve disaster management outcomes. It will provide a foundation for future projects and represents a novel contribution to the research landscape of the ethics of disaster management practice, disaster response law, human rights law and data protection in exceptional circumstances.
Communication: Some emergency management experts refer to social media as a kind of psychological first aid and as a system for ensuring community cohesion (Taylor, Wells et al. 2012). Social networks can be key tools in building real networks and fostering community resilience (National Academies of Sciences and Medicine 2017). Our case studies demonstrated the importance in emergency management of delivering clear, authoritative information to citizens in a tonally appropriate manner – namely, a manner that generates trust. End User beneficiaries identified the need to have model messages available that can be swiftly adapted to incident-specific situations. SLANDAIL developed a method to analyse, translate and produce messages for emergency management, and implemented these through an integrated communication system.
Emergency management systems reviewed by SLANDAIL exist in one language only (usually English) and no provision is made for porting them into other languages. The SLANDAIL prototype’s Graphical User Interface (GUI) has been localised in three languages, with the potential to port the system of analysis for use in other languages. Graphical tiles are used so that communication via text is reduced to a minimum and translation into another language can be carried out quickly and in a cost-effective way.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our research includes exploring the use of facial expression analysis software to obtain real-time data on speakers’ facial expressions. SLANDAIL has developed systems to be integrated in the system prototype as APIs (Application Programming Interfaces), which can be used as communication training tools for emergency operators and first responders. iMotions, a system with this capability, was integrated in to the SLANDAIL System.
Project Context and Objectives:
Citizens regularly communicate through social media computing systems, particularly in times of rapid upheaval. During major emergencies or events the sharing of information (for example, infrastructural warnings, details of threat levels, and transportation advice) becomes an urgent necessity. As a result, a large volume of information relevant to emergency response is being generated in real time across social computing networks in a variety of new forms.
These developments mean that hierarchical models of communication between institutions and citizens are being challenged, with information now flowing in multiple directions through what have become known as “backchannel communications” (Cardoso 2008, Lindsay 2016). In situations of natural disaster, the scale and speed of events can threaten to overwhelm the capacities of emergency managers. By exploring the potential of modern computing systems to incorporate real-time feedback from the public, and by viewing the citizenry as a powerful, self-organizing, and collectively intelligent force (as Palen, Anderson et al. 2010 have influentially argued we should), we can add new tools to those currently available in disaster response. However, technical constraints as well as the complex ethical and legal issues surrounding data collection mean that emergency managers are currently unable to aggregate and systematically utilise this information.
SLANDAIL navigated the complexity of computing, ethics, the law and communications to create an aggregated and integrated system. Operational emergency managers were introduced to legal and ethical principles, while academic researchers developed modules in tandem with SMEs working in data analytics, emergency management and communications. The complexity of roles of active partners was carefully managed throughout the project, and disparate areas of research were carefully combined to create a working Emergency Management System (Figure 2).
At the time of SLANDAIL’s inception, the integration of social media analysis into disaster response processes was rare. SLANDAIL addressed this problem by equipping emergency managers with analytical tools to enhance response and planning capabilities during natural disasters, as well as with communications guidelines and legal and ethical principles for harvesting data and protecting citizens. The project combined expertise and resources in social media, security, ethics, law, emergency management, communication, text analytics, and image analytics, to deliver a multi-disciplinary solution to a challenging, real-world problem.
The system uses location-based geospatial information combined with processed social media data. The overall objective, in the context of security under the FP7 Framework, is for the project to help increase the security of individuals and groups living in areas affected by natural disasters by adding social media to sources of information available to emergency managers.
The project has been designed in response to requirements drawn up by control room teams that manage disasters and emergency events. End Users of the system have been closely involved in the design, testing and evaluation of each strand of the project. SLANDAIL partners collaborated to build and test a prototype system with due regard to ethical and factual data provenance. The SLANDAIL System aims to remove the burden of search and interpretation of the latent information contained in social media data from End Users.
Finally, SLANDAIL followed a Commercialisation and Exploitation Plan to further develop research outputs.
2.2. Glossary
Title Definition
SLANDAIL System An integrated system using emergency management and social media
SLANDAIL Social Media Monitor A platform that monitors social media live using advanced text and image analysis tools developed through project research
SLANDAIL Licence Agreement A licence agreement for use of social media data in emergency management systems that takes into account research in four areas of law: Data protection, copyright, IP and Human Rights
SLANDAIL End Users Emergency management teams and control room operators, from police and army to civil protection agencies, with operational experience in managing major natural and man-made disasters
2.3. Objectives
# Objective Achievement
1 COLLATE and REVIEW the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property Emergency Managers were involved throughout system design and testing. Research completed in the first six months outlined the usability requirements for End User beneficiaries. Test data collated as part of specific emergency related case studies was used to perform system tests.
2 CREATE protocols to protect the rights of the citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens The project created an ethical framework for using data in an emergency and followed this with preliminary legal guidelines and recommendations. This culminated in the development of an innovative model licence agreement for a social media analysis system for disaster response.
3 BUILD AND TEST a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management Built and tested a working prototype that combines emergency management systems with social media. As the project developed, state-of-the-art methods (text, image, speech/non-verbal communication) were introduced, tested and evaluated, leading to the creation and testing of an integrated system.
2.4. Achievements
Approaches to exploiting social computing systems to enhance the management of public crises have been co-evolving organically alongside the media themselves (Hughes and Palen 2009, Shklovski, Burke et al. 2010, Acar and Muraki 2011). SLANDAIL integrates these approaches and methodological reflections and explores how they can best be directed towards the development of a system for using social media in a disaster management context, focusing on improving communication during disasters and addressing the ethical issues involved. The project navigated the complexities of analytics, emergency management, ethics and communication. We successfully deployed, tested and evaluated a Social Media Monitor that analyses social media data in real-time, using advanced analysis in text and image. This was integrated with an Emergency Management System that is supported by a Licence Agreement with communication functionality to enhance End User capabilities.
Text analytics: Existing methods of text mining have been designed for formally written texts. Social media communications tend to be characterised by different structural and linguistic properties. SLANDAIL worked to develop methods of text analysis that would be effective in identifying and extracting relevant information from spontaneous communications in social computing systems.
Terminology (disaster-specific terms) and ontology (a defined set of concepts in emergency management) were identified as key research areas. These contributed to the design and build of the SLANDAIL Social Media Monitor and localisation of the SLANDAIL system to include all three project languages (English, German and Italian). Our methods of language processing and of terminology management can be exploited during operations to facilitate communications when time is limited and stress may limit operators’ ability to communicate effectively.SLANDAIL combined models of and investigations on the language of affect (Martin and White 2005), sentiment analysis (Ahmad 2011), disaster communications (Haddow and Haddow 2014) and the language of social media (Zappavigna 2012) and adapted these to the study of the language of affect in formal and social media in English, German and Italian.
Image analytics: Images have to be processed in a timely manner and as close to real time as possible. SLANDAIL research involved studying images and videos, often of inferior quality, made and posted by citizens affected by disasters. We deployed an image recognition system for detection of images related to flooding and events related to flooding. The results of our controlled experiments using degraded images indicate that the SLANDAIL system can still produce acceptable results when all images are degraded, giving it a broad application in practice. We have produced a system in which the outputs of text analysis are integrated with image analyses at the feature level to enhance the informational content. Furthermore, we tested a video analysis system for shaky camera footage in disaster zone areas, evaluated by End Users.
Integration of an emergency management system (EMS) and social media monitor: At the outset of the project, two separate systems existed that each performed complex tasks – CID’s Topic Analyst and DataPiano’s Sistema Informativo Gestione Emergenze (SIGE). Topic Analyst, a tool for the monitoring and analysis of textual data, was capable of processing documents in volume from the internet and providing in-depth analysis. SIGE, an Emergency Management System (EMS) and the base program used to build the SLANDAIL System, was capable of retrieving information on ongoing disasters and assisting emergency management communication by accessing databases of key personnel. However, a system that integrated both capabilities, as well as incorporating image analytical capabilities, had the potential to deliver numerous benefits, including real-time information and “eyes on the ground” information from citizens.
Early on in the project, we loosely coupled these existing systems to prove feasibility. Due to limitations in German law and contractual agreements concerning CID, we were unable to use live data from their system. So, we employed and integrated a Social Media Monitor in line with End User needs to enhance emergency management control room capabilities. Using multi-modal aggregation we tested a method for increasing decision-making capabilities using different signals of information, including disaster-zone information aggregated with social media data.
Ethical and societal awareness: Data gleaned from social computing systems is characterised by an inherent tension between mass and personal communication, since the informal and local nature of much of the content can sit uneasily with the global, open-access nature of the platforms on which it is shared. SLANDAIL sought to balance the needs of emergency managers and first responders for an effective and flexible system with the rights of citizens to privacy and data protection.
SLANDAIL examined the societal impact of social media in a security context and outlined an ethical framework for use in disaster management. This framework sets out specific measures to be taken in the technical design, licencing, administration, governance and use of the system. Given that there are various controversies on government surveillance relating to the use of social media (Bertot, Jaeger et al. 2012, Bekkers, Edwards et al. 2013), existing disaster management literature needs clarity as to the ethical validity of collating large amounts of personal data when people are at their most vulnerable.
Legal research culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. This agreement enabled the project to pursue the objective of ensuring the ethical harvesting and analysis of social media data to improve disaster management outcomes. It will provide a foundation for future projects and represents a novel contribution to the research landscape of the ethics of disaster management practice, disaster response law, human rights law and data protection in exceptional circumstances.
Communication: Some emergency management experts refer to social media as a kind of psychological first aid and as a system for ensuring community cohesion (Taylor, Wells et al. 2012). Social networks can be key tools in building real networks and fostering community resilience (National Academies of Sciences and Medicine 2017). Our case studies demonstrated the importance in emergency management of delivering clear, authoritative information to citizens in a tonally appropriate manner – namely, a manner that generates trust. End User beneficiaries identified the need to have model messages available that can be swiftly adapted to incident-specific situations. SLANDAIL developed a method to analyse, translate and produce messages for emergency management, and implemented these through an integrated communication system.
Emergency management systems reviewed by SLANDAIL exist in one language only (usually English) and no provision is made for porting them into other languages. The SLANDAIL prototype’s Graphical User Interface (GUI) has been localised in three languages, with the potential to port the system of analysis for use in other languages. Graphical tiles are used so that communication via text is reduced to a minimum and translation into another language can be carried out quickly and in a cost-effective way.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our research includes exploring the use of facial expression analysis software to obtain real-time data on speakers’ facial expressions. SLANDAIL has developed systems to be integrated in the system prototype as APIs (Application Programming Interfaces), which can be used as communication training tools for emergency operators and first responders. iMotions, a system with this capability, was integrated in to the SLANDAIL System.
Project Results:
3. Description of the Main S&T Results/Foregrounds
In SLANDAIL, academics and SMEs worked together with emergency operatives to demonstrate a cost-effective and ethically robust way to integrate information from social media into an emergency management system. Our goal was to make maximum ethical use of the information obtainable from social computing systems to enhance the performance of emergency management systems and reduce the burden of interpretation on End Users. SLANDAIL partners collaborated to model a system that delivers consistent and relevant messages to disaster management personnel working in three languages (English, German and Italian) with accompanying guidelines for preserving data confidentiality and ensuring factual provenance of information.
The core of the research and development on this project is summarised in four main strands:
1) Emergency Management: What do emergency managers do in a disaster? What technologies do they use? How can social media data improve their operations?
2) Ethical Analysis and Legal Research: Is it legal to use social media? Are there any ethical boundaries for using personal data in an emergency?
3) Analytics Research: How can text and image be automatically aggregated and analysed to help emergency management teams?
4) Control Room Technology: How can social media be integrated into technical systems to improve emergency response? Can communications between emergency managers and the public be improved?
Figure 4: The four areas of SLANDAIL research developed in tandem with one another during the project
The following sections present an account of the work undertaken and results achieved under four headings. The first section, EMERGENCY MANAGEMENT: Requirements and Technologies (3.1) describes our analysis of the operational environment in which End User beneficiaries respond to disasters. The following section, ETHICAL ANALYSIS AND LEGAL RESEARCH: Society and data (3.2) examines the tension between the demands upon the different stakeholders involved in disaster response – the requirements upon emergency managers to protect life and property, alongside the need of citizens for privacy and confidentiality – and describes the measures taken by SLANDAIL to mitigate this tension and ensure the system would be ethically robust.
ANALYTICS RESEARCH: Advances in Social Media Analytics (3.3) outlines the work undertaken to develop the technical components of the system. It describes our work on advancing and adapting text and image processing techniques to the analysis of social media data, our research on speech and non-verbal communications, and our development of methods to aggregate these techniques to deliver relevant information. Finally, CONTROL ROOM TECHNOLOGY: Building a Social Media-Enabled Emergency Response System (3.4) describes the final stages of SLANDAIL’s work to deliver an integrated system to analyse data extracted from social computing systems in an ethical manner. This section outlines the methods used to consolidate and integrate our research and present a simple, consistent and user-friendly system that allows emergency managers to use social media to increase their situational awareness of an emergency. This reduces the existing burden of information processing upon emergency managers and enables them to respond more effectively to threats.
3.1. EMERGENCY MANAGEMENT: Requirements and Technologies
3.1.1. Introduction
SLANDAIL research was directed at increasing the security of citizens by enhancing the capabilities of emergency managers to respond in an informed and effective way to natural disasters. It was essential for the project to involve emergency managers from the outset, and our research during the first six months of the project focused on analysis of existing disaster response practices. Our work established the basis of the subsequent research by ascertaining the requirements and real-world operations of our End User beneficiaries – PSNI, GARDA, and BLS, with input from Protezione Civile, via DataPiano.
The acquisition of End User requirements, case studies and technical specifications allowed us to fulfil the first of our objectives: to collate and review the sharing and dissemination of disaster zone information amongst experts tasked to improve security of citizens and property. It placed the project on a practical and empirical footing, informing our academic and technical research with an understanding of the situational contexts in which the system would function. It also clarified the tension (a tension which would later culminate in the creation of an innovative Licence Agreement) between, on the one hand, the ethical questions surrounding mass data collection and the privacy of citizens, and on the other, the need of first responders for a system that would deliver real-time data in an effective and relevant manner.
End Users led an analysis of disaster response studies to discover common patterns in the management of crises and to establish the extent to which social media analysis is integrated into emergency response practices. These case studies focused on flooding, a problem common to a range of natural disasters, to enable an effective comparison of disaster management protocols and operational environments. The case study outcomes demonstrated the need for more effective analysis of social media data as well as providing an outline of the phases and communicative demands of a crisis.
Our analysis included a technological assessment of End User systems. We found that current systems would be compatible with a social media monitor and identified the importance of a flexible, web-based system that would be adaptable to disaster response practices. This analysis also revealed the value to End Users of having communication templates and guidelines to communicate information to citizens during times of crisis, and highlighted the importance of building trust in advance of emergency situations.
End User involvement continued throughout the project in the form of meetings, software demonstrations and software evaluations. Feedback was assimilated into each successive stage of software development. End User feedback was also incorporated into research on the ethical and social impact of social media analysis. End Users highlighted the duty of care to protect life and property first during an emergency. Emergency managers’ primary focus is the safety of citizens, and their key concern in using data is in the recovery period when subsequent investigations are carried out. At this stage, they wish to ensure there is a hierarchy of responsibility for decisions made.
Meetings between emergency managers, legal experts and academics were key to identifying the tension between this obligation on the part of End Users and the need to protect the privacy of citizens. This tension informed our research into the relevant ethical, practical and statutory issues and development of the legal framework within which the system could operate.
Figure 5: SLANDAIL’s first 6-month plenary meeting was held at the Titanic Centre in Belfast in October 2014. The meeting was organised by members of the PSNI and was attended by representatives from all partners.
Attendance from all partners included (l-r, back row-front row): ROW 1: Jose Maria Zavala Perez (CIES), Sebastian Wustmann (CID), Eamon O’Loughlin (GARDA), Henrik Selsøe Sørensen (Strategic Advisory Board); ROW 2: Bryan Scotney (Ulster), Stephen Kelly, Shane Finan, Damian Jackson (TCD); ROW 3: Sonya Coleman, Min Jing (Ulster), Rebecca Bury (Stillwater), Gemma Galdon Clavell, Sadhbh McCarthy, Gary Doyle (CIES), Alexander Stumpfegger (CID), Maria Grazia Busà (UNIPD), Jon Henry (PSNI); ROW 4: Raffaella Panizzon (UNIPD), Victoria Macarthur (Pintail), Brian Halligan (GARDA), Khurshid Ahmad (TCD), Una Williamson (PSNI), Gerhard Heyer (INFAI), Daniel Zschuckelt (BLS), Enrico Musacchio (DataPiano), Sabine Gründer-Fahrer (INFAI), Francesco Russo (DataPiano), Carlo Aldrovandi (TCD)
3.1.2. Disaster Management Case Studies
At the outset of SLANDAIL, each End User developed a case study in their own catchment area to demonstrate real life scenarios in which social media was or was not used in disaster or emergency management. GARDA detailed an incident of widespread flooding on the M11, a major arterial route between Dublin city and the East coast of Ireland. BLS covered flooding in June 2013, when a string of rivers in the area of Saxony broke their banks. PSNI detailed the serious threat of coastal flooding during the winter of 2013/14 which had the potential to seriously impact a built-up area of Belfast city. This archive of disaster management case studies (D1.1) facilitated the sharing of information between SLANDAIL partners. Overall, these studies revealed that there was no consistent use of social media in disaster management by the agencies. End Users recognised that social computing systems could play an important role in terms of resource deployment, communication and decision making.
Subsequently, SLANDAIL reviewed social media usage in crisis management, focusing on specific instances of flood disaster management in the USA, Italy, and Germany, in order to understand the operational environment and identify potential development. Each of these instances showed different levels and sophistication of usage, and the review served as an analytical tool to illustrate the ways in which social computing systems have helped emergency services in the past, and identify where social media use could be improved upon.
It emerged that emergency services are hampered by the lack of a clear and efficient method of aggregating Twitter or Facebook streams to form concise and direct messages. It also became clear that in times of crisis the demand for information rises dramatically, and users’ ability to source credible, reliable information decreases. The issue of trust again came to the fore as an essential factor in communication between emergency services and the public.
3.1.3. Review of Technological systems
We reviewed the technological systems and capabilities of each of the End Users, using case studies and questionnaires to ascertain the technologies used by End Users. Two police forces and one army reserve (PSNI, BLS, and GARDA) as well as the Protezione Civile of Veneto Region (responsible for coordinating the work of civil protection agencies in the Veneto region), provided information on the technologies they use during the management of emergency situations. This review encompassed all the components of the End User beneficiaries’ IT equipment as well as extending the analysis to IT use during emergency management by other significant US and International organisations (like FEMA and the Red Cross) to ascertain common capabilities and practices.
We determined that all existing End User resources were compatible with design requirements, but that some integration of existing structures would be necessary. SLANDAIL was developed to answer End Users’ needs by using multi-platform software to ensure adaptability and enable the re-use of existing hardware. We designed the Emergency Management System to be highly configurable to minimise customisation needs. We found that social media tended to be used to manage the earliest phases of an emergency on an ad-hoc, manual basis, and we identified potential for it to be further integrated into existing operations. We subsequently integrated the Emergency Management and Social Media Monitor (3.4.2). The review highlighted the importance of clear and fast communication during emergencies, as well as the need to establish a trusting relationship with citizens. This gave the project team a broader picture of civil technologies and communication media used by End Users, and facilitated advances in research and technology development.
3.1.4. Information Sharing and Public Communications
SLANDAIL determined how communication occurs during emergencies and how information is exchanged and interpreted by the public and other stakeholders. The approach included a workshop hosted at the LREC Conference in Reykjavik in May 2014, analyses of video footage of official communications during crises, and the development of guidelines for effective disaster communications reflecting best practice in professional communications. The resulting analysis (documented in D1.4) identified gaps in information exchange processes and enabled an overview of how communication occurs during emergencies (Busà and Brugnerotto 2014, Musacchio 2014). This aided the technical development and assessment of the SLANDAIL prototype and helped End Users to improve their communications by building trust with stakeholders before disasters. Trust emerged as a key issue in this research (Fennell 2014).
3.2. ETHICAL ANALYSIS AND LEGAL RESEARCH: Society and data
3.2.1. Introduction
Social computing systems, including social media systems and social networking systems, facilitate personal communication and voluntary collaboration. They are frequently used to transmit local and individual information, and during disasters they offer information that may not be available from other sources. Once transmitted, much of this information becomes accessible to third parties and may be collected, stored and used in ways that have not been anticipated by the individual. The widespread growth as well as the innovative nature of social computers has outpaced legal mechanisms: The General Data Protection Regulation (GDPR) and its parallel directive aimed at law enforcement (LEPD), initially published by the European Commission in 2012, will be adopted across the EU in May 2018.
Within a project consisting of operational End Users, academic researchers and SMEs, it was important to first address the need to address differences in experience and expertise relating to data use. End Users heighted a ‘duty of care’ whereby they are required to first save life and property with whatever information is available to them, and secondarily to consider any other legal requirements. The use of social media data forms part of an opaque landscape of legal and ethical questions, and this project answered these questions through dialogue between legal experts, ethical researchers and End Users. A Socratic Dialogue method was opened between legal experts, End Users and SMEs in the early stages of the project (November 2015). The successful results of this process showed that expertise in law was needed to adequately address the societal challenges inherent in use of data for emergency management, and this led to an adjustment of focus toward the legal landscape that exists in copyright, data protection, IP and human rights.
There is a basic and ongoing tension between the public openness of mass communication and the informal nature of personal communication. SLANDAIL research into the impacts of using social media in a security context explores these ethical and social tensions to create a framework for navigating the evolving informational space whilst ensuring factual and ethical provenance.
SLANDAIL approached the societal impact of social computing systems in a security context from an ethical perspective. We analysed relevant critical thinking to develop a coherent framework for balancing the competing rights and responsibilities of stakeholders in disaster events. This research showed that following existing human rights law is the most appropriate means by which to consistently protect the dignity of citizens and the integrity of data (De Stefani 2000, Berger, De Stefani et al. 2016).
Ethical analysis included ongoing dialogue between End Users, academic partners and legal experts and functioned as a scaffold for the formation of a licence agreement. This addressed the second of our key objectives: to create protocols to protect the rights of citizens and to manage the confidentiality of the collected data and processed information relating to individual citizens.
The main output of this work is a model Licence Agreement – the first of its kind – and an accompanying Legal Checklist, representing the translation of SLANDAIL’s analysis into a practical and enforceable agreement.
• Licence Agreement: SLANDAIL research found that the system needed to deal with a complex but surmountable set of issues that could be dealt with using existing legal mechanisms. The Licence Agreement conforms to existing protocols in the field of data management and creates specific clauses for the field of disaster management. It specifies the scope of the system’s use, limiting its deployment to humanitarian purposes and specific territories. It also points the End User towards the steps to be taken to access live social media data and comply with local statutes.
• Checklist: The accompanying checklist empowers End Users with information and awareness of the issues surrounding the use of social media data in an emergency management context. It functions to encourage diligence on the part of the licensee and to highlight areas – such as data security, privacy, and copyright – in which particular safeguards are needed.
SLANDAIL has charted the legal and ethical landscape surrounding the use of social media data in disasters by synthesising relevant research and expertise. An audit trail of the decision-making processes behind the Licence Agreement is recorded in project documents, showing its collaborative development. In a dynamic social, technical, legal and regulatory environment, the Licence Agreement and accompanying Checklist represent a template for further legal agreements as well as a baseline for future innovation. They provide a foundation for future projects and represent a novel contribution to the research landscape of the ethics of disaster management, disaster response law, human rights law and data protection in exceptional circumstances.
3.2.2. Review of Societal Impact of Social Media in Security Contexts
SLANDAIL conducted a comprehensive review of the current critical thinking on the societal impact of using social media in security contexts. This comprised a review of academic, legislative and media sources, with a focus on the legal and regulatory framework and policy context within which the project operates. A four-part framework was developed encompassing desirability, acceptability, ethics and data management aspects of using social media data to enhance decision support for emergency managers. This provided an initial assessment of the relevant societal concerns for SLANDAIL.
Following this assessment, it became clear to the project that expertise in law, particularly law concerning social media in emergency response, would be required to fit End User and technical requirements for the project. A partner in Data Protection at major law firm Arthur Cox (Dublin) was invited to attend a seminar organised and opened a Socratic Dialogue between End Users, technical researchers, ethical researchers and SMEs. This dialogue resulted in further legal research and the drafting of guidelines for the use of social media in emergency contexts. The success of this research led to an adjustment of the project to focus on the complicated and opaque legal landscape surrounding use of social media in emergency response.
3.2.3. Ethical Framework and Guidelines
We applied the theoretical approaches of value pluralism and state of exception ethics to the specific domain of the use of social media in disaster response (D2.2). The framework set out measures to be taken in the technical design, End User practices at the various phases of emergency management, licensing, administration and governance of the SLANDAIL system to mitigate ethical risks. An ethical matrix method was applied to tensions between emergency management and ethical requirements. One of the key outcomes was highlighting the fact that restrictions to more sensitive data would be required to ensure that personal information is only accessible to emergency managers with a high level of access.
SLANDAIL explored the technological and legislative developments in the field of data harvesting from social and formal media, presenting a set of guidelines and a template of harvesting social and print media data. We identified implications, most notably the importance of adhering to principles of proportionality, anonymity and security. Results stated that data collected should be proportional to the need of emergency managers; personal data that is not anonymised should only be kept for as long as it is required; and data should be stored with a reasonable level of security (Jackson and Hayes 2016). The guidelines recommended specific procedures including a high standard of security, compliance with data security practice, a threshold of factual reliability, and an in-depth logging system to track and record all activities of the system.
We developed a framework comprising protocols by which to structure the analysis of disaster response practices with appropriate regard to the interests of all stakeholders involved. This comprises two phases: reflective analysis and critical reflection, incorporating respectively an analysis of stakeholder interactions and interests, and a reflection on the responsibilities arising from existing power dynamics (D2.10). Critical reflection on the power implications of extant practices enables context-sensitive ethical deliberation through re-examination of these practices and development of proposals for their refinement, alteration or replacement.
3.2.4. Guidelines and Templates
3.2.4.1. Data privacy in disaster management
Copyright and data protection legislation within the framework of German and EU law was assessed to deliver legal opinion for the project (D2.4). We determined that SLANDAIL is intended to be deployed in a lawful manner in pursuit of the End Users’ legal obligation to protect life and property, meeting national legal requirements applicable in the context of their deployments. However, given the multiplicity of contexts and the lack of harmonisation in legal systems, it was recommended that social media analysis be made available to End Users on the basis that they would be contractually obliged to ensure that they address any data protection, privacy and copyright issues that might arise.
We further analysed the legal framework of European copyright law (with a focus on German law, D2.7) to ascertain the requirements for the use of copyright protected material within a project like SLANDAIL. We found that copyright law is significant for our project, having the potential to be a hindrance to the use of social media content for disaster management. Social media content is often deemed subject to protection of the social media provider. The use of the system in a disaster scenario should be enabled and facilitated by specific statutory legislation. Such a provision could be developed in the framework of limitations and exception for copyright-protected works for public security. As of 2018, the General Data Protection Regulation will provide a legal basis for projects like SLANDAIL, permitting disaster management entities access to personal data in cases of disaster response. Our findings fed into the final software licence agreement (D2.6).
3.2.4.2. Legal protection of human rights in disaster management
SLANDAIL analysed the legal scenario of the proposed system with specific reference to the spectrum of internationally recognised human rights, with a distinctive focus on privacy and data protection rights. Meetings and workshops were held between the project beneficiaries – including experts in intellectual property, data protection, human rights and internet law, and the ethics of technology – to discuss and develop this research (Berger, De Stefani et al. 2016). The result was a set of guidelines on the transparent use of the SLANDAIL system for disaster management, mainly addressed to End Users (D2.5). These included recommendations that End Users familiarise themselves with legal procedures for using data in an emergency, determine the conditions under which the SLANDAIL system can be deployed, and maintain open channels of communication during an emergency. These guidelines provided for the consistent integration of human rights awareness in the disaster management system.
Our research emphasised the importance of fully integrating SLANDAIL into the institutional and legal architecture of civil protection structures at the local, national, European and international levels, and highlighted the dignity of the human person as a common consensual principle through these legal levels (Hayes 2017). We found that the SLANDAIL prototype fit the legal design outlined, and that the project’s original features and specific characteristics of flexibility, multilingual capability, transparency and people-centeredness make our system a tool that complements the human rights based approach to civil protection policies advocated (D2.8).
3.2.5. Creating Guidelines for Developers and End Users
We delivered a Template Software Licence Agreement to be used as a model to license the SLANDAIL system to End Users. Based on an analysis of the Legal Caucus and with the experience of four legal experts in four EU countries, we distilled the core issues identified into (a) a template software licence agreement and (b) a legal checklist which shows, from a practical legal perspective, how the system can be licensed.
The license includes an acknowledgement that the Licensor does not own any intellectual property rights of data from social computing systems, an acceptance of responsibility for ensuring that the use of social media data complies with applicable laws, and an agreement on the part of the End User to submit to EU law. The legal checklist is a synthesis of the research authored by ethicists and legal experts in Ireland, the United Kingdom, Germany and Italy. The checklist was designed to assist End Users in identifying legal requirements prior to deploying SLANDAIL in any live environment. The checklist covers five key legal areas relevant to the use of SLANDAIL: contract; data protection; privacy; copyright and human rights.
SLANDAIL also explored the parameters and limitations of laws that govern Internet contracts, specifically personal data and proprietary information on social media, in relevant jurisdictions. We produced a report (D2.9) on these laws that highlights the extent to which SLANDAIL software programmers, vendors, emergency responders or End Users can safely and lawfully derogate from prescribed laws without infringing third party rights in personal data and proprietary information on social media.
Figure 7: Rob Corbet, Partner in Data Protection at leading Irish law firm Arthur Cox, speaking at the launch of the SLANDAIL System Licence in February 2017.
3.2.6. Information Ethics and Fiduciary Theory
SLANDAIL research into the ethics of information and implications for the large-scale harvesting of social media data resulted in the development of the Intrusion Index (D4.2) which informs the End User when a large volume of personally identifying information is being collected. This feature recognises the importance of building software tools with issues of trust and security in mind (Delgado, Torres et al. 2005). It functions as a moral mediator that preserves the discretion and decision-making abilities of End Users while recognising the value of privacy and mitigating the risks of potential undesirable societal and ethical impacts of the system. It was subsequently incorporated into the SLANDAIL Social Media Monitor (cf. Section 3.4.2.1).
Further research was undertaken to bring research in ethics beyond the state-of-the-art. SLANDAIL has given rise to a doctoral research programme within the School of Ecumenics in TCD for dealing with the ethics of using technology in disaster management. The project completed transfer in December 2015 and is entitled “An Analysis of Emerging Human Rights and Ethical Issues in the Harvesting of Data from Social Media During Emergency Response to Natural Hazards.”
This research presents a joint ethical and legal analysis within the frameworks of Information Ethics and Fiduciary Theory. Through an analysis of emergency management systems developed by SLANDAIL partners, it assesses the potential human rights and ethical impacts of these systems. It considers social computing systems with respect to values of Privacy, Justice, Trust, and Accountability. The methodology comprised interviews with software developers and emergency responders involved in the project as well as observation and use of emergency management software systems. In interviewing key actors, it was possible to discern the functionality of systems such as SLANDAIL as well as the potential for additional functionality not present in the system itself. While some of this work has been completed during the project (Hayes 2017), the work is ongoing as part of a PhD thesis with a proposed submission date of September 2017.
3.3. ANALYTICS RESEARCH: Advances in Social Media Analytics
3.3.1. Introduction
Figure 8: SLANDAIL Analytics Research culminated in the development and deployment of a social media monitor that combines research strands.
Much of the focus in social computing analytics is on text analytics (Batrinca and Treleaven 2015, Wu, Cao et al. 2016), as the processing of speech and images poses difficulties of a different order of magnitude (Tzelepis, Ma et al. 2016). We have developed methods and prototype systems to analyse and render text available through social networks into items of actionable information. For image analytics, we have developed a new technique for faster image processing based on recent findings on human visual systems and developed the use of neural networks for image annotation, thereby rendering images and videos into complementary actionable information. We have combined analytics in a Social Media Monitor, where text, image, geolocation and other data are combined to integrate with emergency management systems.
We have made use of metadata from social media messages related to location. Geolocation of impacted areas is a critical issue in disaster management, and social media systems have been coupled to geographical information systems with some success (Chae, Thom et al. 2012). Location data, when used with professional care, will yield information that can be mapped onto modern geolocation information systems, comprising socio-economic and other reliance data (Stefanidis, Crooks et al. 2013). We have fused data to create actionable location-specific information incorporating the EU geolocation framework, INSPIRE .
Furthermore, SLANDAIL investigated the role of non-verbal cues in disaster communications – facial expressions, speech quality, and hand gestures – particularly in disaster mitigation speeches made by leaders. The emotion shown by authority figures can, in negative instances, have a devastating impact on disaster victims (Wang, Zhang et al. 2016). We have used state-of-the-art emotion recognition systems for each of the three modalities, and validated the results against an observational survey. This approach will be useful for disaster communications training and has extended recognition research by using it for real-world data.
The research contributed significantly to the development of a suite of programs that can collect, filter, and analyse (multi-modal) social computing outputs, link this output to geolocation data, and convert these into actionable information. Some integration was carried out in the Social Media Monitor, a standalone system that combines elements of text and image analytic research. Some research (multimodal aggregation and multi-criteria decision making) was not integrated but was carried out to show feasibility in future similar systems.
The software developed be made available to any GIS-enabled emergency management control room system. The state-of-the-art research was conducted in collaboration with End Users, legal and ethical partners, and the technology partners, who participated in specification and evaluation. Considerable advances have been made in the novel use of social media and social network outputs in the following areas, and these have been published in conference papers and journal papers.
End Users emphasised the need to cultivate a relationship of trust with citizens, and for analytical methods and outgoing communications to conform to local linguistic conventions. SLANDAIL created clear communication guidelines through linguistic and communication research. We enabled the processing of social media data in German and Italian, and developed model messages for organisations to communicate real-time information to the public. To combine and integrate our analytics with the Emergency Management System, we implemented a common information exchange format (XMI) for compatibility.
3.3.2. Text Analytics
Social media output and web feeds released by agencies, experts, and citizens, can augment existing data. A number of techniques are being developed to analyse this information, such as tweet pooling and topic modelling (Mehrotra, Sanner et al. 2013) as well as scraping location information related to an unusual event (Wakamiya, Jatowt et al. 2016). Studies have explored methods for harvesting community-oriented information related to disaster alerts (Shih, Han et al. 2014) and for aggregating tweets to profile and geo-reference an impacted area during a flood (Johnson, Ruess et al. 2016).
Social media has been used to gauge levels of public concern about a disaster caused by potential human error (Olteanu, Vieweg et al. 2015, Woo, Cho et al. 2015) by adapting a social media analysis system developed in Germany to process social media messages in English. See, for example, work in English (De Choudhury, Gamon et al. 2013), German (Thom, Kruger et al. 2015), Italian (Avvenuti, Cresci et al. 2014, Cresci, Tesconi et al. 2015, Avvenuti, Cimino et al. 2016), and Chinese (Fung, Fu et al. 2013)). Large-scale text mining systems have been offered for disaster management - see SAS’s text miner (Lazard, Scheinfeld et al. 2015), IBM’s Modeler (He, Zha et al. 2013) and the RapidMiner system (Ristoski, Bizer et al. 2015).
SLANDAIL developed a text analytic module for analysing static text data collected from both social and formal media; later in the project, a real-time multi-modal aggregation system (written in Java and using existing systems for dependency) that can capture, process, and display information extracted from live social media stream and web feeds was developed. We gathered and tested a body of text numbering over 500,000 domain documents with the assistance of automation algorithms (CiCui) and from these we created a lexicon for disaster management terms, particularly those related to incidents of flooding. We simultaneously developed standalone tools for research, the Summary of High Incidents (SHI) and the SLANDAIL Newsletter, to provide regularly updated information about emergency and disaster-related Twitter messages and news. These operate primarily through English, but can include some German and Italian texts.
Some of the text analytic methods developed above were later integrated into the SLANDAIL Social Media Monitor which combines image analytics when performing aggregated analysis over live social media stream.
3.3.2.1. Specifying and Creating Grammar for Social Media
CiCui is a background technology for SLANDAIL. It comprises text analytic module enhanced within the project that features functions such as calculating keyword metrics, extraction of collocation contexts, terminology identification, latent semantic analysis, and contains corpus management facilities (e.g. automatic downloading and parsing of domain documents). CiCui has been used to producing a tri-lingual data base of disaster terms that will complement EU environmental data bases and is being delivered to the EU terminology clearing house IATE. We have used methods for identifying terms automatically, developed in (Sparck Jones 1972, Ahmad and Rogers 2001). The processing requires a database of domain keywords, phrases, and, language patterns commonly used to discuss emergency events. The building of such knowledge bases are fulfilled through a system called CiCui (Collection of Words in Chinese), developed as part of a related doctoral project on the statistical learning of language patterns related to events (Zhang 2016). The main functionality is to convert unstructured textual content into structured indices from which one can extract information about the texts with standard querying languages. CiCui has also been used to support text analysis as part of the CITU image annotation system (see 3.3.5.1).
It has been observed that in specialist domains and during emergencies, variations of fixed phrases are used to indicate abnormal events. Compare the sentences ‘road between Carlow and Leighlinbridge is closed now’; ‘road Carlow to Leighlinbridge closed’; and ‘fallen trees on Carlow/Leighlinbridge’. For a human being these phrases mean roughly the same. However, these patterns defy lexical/syntactic/collocational conventions of the English language known also as frozen phrases or recurrent patterns that hallmark discourse in specialist domains and during abnormal events. The term used here is local grammar that governs the behaviour of tokens in these phrases and one can assign a probability of usage to each pattern.
Our investigation of the use of the so-called local grammar in dealing with the convention defying phrases that are a distinguish feature of social media systems, like Twitter, and other (micro) blogging services, is amongst the first in this field. We made a statistical analysis of two corpora comprising more than 14 million Twitter messages collected during abnormal whether events (Hurricane Matthew and a severe snowstorm in eastern US in 2016). We had taken advantage of algorithms developed previously for sequential pattern mining (VGEN and VMSP) that has been used traditionally in identifying behavioural patterns in transactional databases used typically in finance, consumer research, on-line trading. We have developed a specialised R package that interface with the indices produced by CiCui to determine the sequential patterns and generalise these to build a corpus-driven local grammar patterns like the examples provided in Figure 10.
Figure 10: Definition of an example local grammar pattern for capturing road/bridge/lane closures/reopens.
Once elicited and validated by experts, the local grammars are encoded using regular expressions (specifically TokensRegex) that allows complex word sequences to be matched based on not only the characters but also the linguistic meta-data associated with the words. These patterns are stored in the SLANDAIL terminology database for recognition of the frozen phrases. These phrases are used in processing and converting social computing output into actionable information. The system is adaptable, in that local grammars can be refined during each operational cycle. The workflow of this process is shown in Figure 11.
Figure 11: Workflow of the local grammar extraction process
3.3.2.2. Specify and Create Lexica for Social & Formal Media
SLANDAIL collected corpora and extracted terminology to develop a terminology wiki (the SLANDAIL Lexicon), which reflected state-of-the-art terminology management. Secondly, we automated term extraction and refinement methodology using the CiCui system. Thirdly, we developed automatic extraction of information (definitions, contexts and ontologies) to compile the Lexicon entries and develop a routine to make it available in an easily accessible format such as XML. This methodology enabled the greatest possible degree of automation of the main processes in terminology, and has allowed us to submit our terminology to IATE (InterActive Terminology for Europe), the EU terminology reference.
CiCui term extraction capabilities was evaluated in a comparison with four commercially available term extraction systems (Synchoterm, TaaS, Termostat, and Vocabgrabber). The test corpus comprises a 326319-word document corpus containing FEMA fact sheets, handbooks from different emergency management agencies and news items extracted from LexisNexis. Extracted terms from the corpus were evaluated by linguistic and translation experts and each of system produced terms on a three-point scale: ‘Not a term’- R(ed); ‘Needs further valuation’ -A(mber); and ‘System correctly identifies term’ G(reen). CiCui outperformed the four commercial systems in emergency terms, and had the lowest rate of producing red terms (Table 1).
Table 1: Comparing the performances of automatic term extraction systems
System # of TCs
extracted Precision RAG evaluation
Red (%) Amber (%) Green (%)
Synchroterm 14791 56% 30 14 56
TaaS 345 42% 48 10 42
TermoStat 4082 64% 20 16 64
Vocabgrabber 2501 14% 78 14 14
CiCui 602 77% 6 17 77
Average 36.4 14.2 50.6
Standard Deviation 27.8 2.7 24.1
We created a term bank to international standards and made this available to stakeholders, minimising the burden of labour-intensive, time-consuming terminology management. This term bank develops mission-critical terminology by considering different communicative situations, and displays terms and their equivalents accompanied by visualisation and information about usage. It was made available as a knowledge base for the project ontology. The Lexicon also establishes best international practices by extending the method used for English to the less widely spoken languages of the project, German and Italian.
3.3.2.3. The SHI System and the SLANDAIL Newsletter
SHI (势 SHI) - Summary of High Incidents - is a background system developed during the project that contributes to research outputs. It harvests news through RSS feeds and English-language tweets (via Twitter’s Streaming API) from selected sources. It is based on Rocksteady, an affect analysis system, and combines a data collection system with a data analysis system that filters harvested texts in relation to domain and sentiment dictionaries. It automatically detects unusual frequencies of lexical categories in emergency-related dictionaries and sends an automatic alert message published through a Twitter account, @SLANDAILalert.
SHI helps to produce the SLANDAIL Newsletter. The Newsletter, updated daily, provides information about emergency and disaster-related events (Figure 12). Keywords are identified using weather- and disaster-related thesauri (for example FEMA, Global Disaster Alert and Coordination System, WHO). In one sense, the Newsletter is a record of trends emerging within a specific period. The system uses graphs, Tweets and captions to display outputs. It was built to process English language text messages and has limited capability to analyse German and Italian texts.
3.3.2.4. Text Clustering
Text clustering, or document clustering, refers to the grouping of textual materials into “clusters” considered to share a common theme. SLANDAIL developed a clustering application through existing system Leipzig Corpus Miner (LCM) in which topic modelling techniques were used to automatically categorise texts into topic clusters. We adapted this to automatically identify disaster-related topics in formal and social media (Schlaf, Gründer-Fahrer et al. 2016), applied temporal clustering techniques to automatically identify characteristic phases of the disaster event (Gründer-Fahrer Submitted), and classified messages on social computing systems with respect to their relevance to flood damages (Musacchio, Panizzon et al. 2016). We explored the representativeness of linguistic features such as unigrams (single words) and bigrams (two consecutive words) as well as the language patterns defined in the SLANDAIL Lexicon. Human annotators labelled a dataset of social media messages to judge whether they contained flood damage information: vectors associated with the messages were then used to train statistic classifiers to automatically identify messages that report flood damage. This allowed for integration with the media monitoring system Topic Analyst to show phases of disaster management in recovery and analysis stages.
3.3.3. Speech and Non-Verbal Communication Analytics
The management of a catastrophic event requires authority figures to communicate with the population showing empathy and leadership. Social media and networks are also used to “telecast” disaster warnings and recovery information in which messages are conveyed not only through words but also through facial expressions, vocal quality, and body gestures. Leadership studies show that audience perception of facial, vocal and hand gestures plays a key role in the audience perception of the message quality (Burgoon, Guerrero et al. 2016).
An automatic analysis of non-verbal disaster communication complements spoken and written communication. This will serve to make the disaster managers aware of the possible perception of the public when they hear and see the managers make life- and business-critical announcements.
3.3.3.1. Towards an Integrated System for Non-verbal Communication
SLANDAIL assessed the feasibility of using speech and gesture technology for automatic extraction of data on speech and non-verbal language. We evaluated the relevance of existing systems and patents of potential future systems (D3.2). This involved the creation of a corpus of videos relating to disaster news during an Italian flood (2015) to identify patterns of voice modulation and body language and create a novel ontology of gestures (Busà and Cravotta 2016). This phase produced two clear findings: (a) that non-verbal communications are essential for gaining trust, especially during a disaster (Busa, Musacchio et al. 2015) and (b) that the extraction of speech data from emergency situations is currently not feasible due to the presence of noise and multiple voices in emergency speech excerpts, but that physical aspects of speech (such as pitch and intensity) could be of value to emergency management systems.
Following this, we looked at the integration of voice quality and hand gestures with research on automation of facial expression recognition (Bartlett, Littlewort et al. 2006, Bartlett, Littlewort-Ford et al. 2014). Emotient is a facial recognition system used for studying facial expressions in controlled conditions. It analyses the movement of facial muscles against a pre-trained database of facial expressions, using actors simulating positive and negative emotional states. This work, in turn, is based on the work of Ekman and Friesen (Ekman and Friesen 1969, Ekman and Friesen 1976), who contributed to the development of the Affectiva emotion recognition system (El Kaliouby, Mahmoud et al. 2016), based on earlier research in non-verbal communications in diverse fields (Picard, Vyzas et al. 2001, El Kaliouby and Robinson 2005, El Kaliouby, Picard et al. 2006). We have used both Emotient and Affectiva and have incorporated access to Emotient in the SLANDAIL system (See Section 3.4.3.2).
Simultaneously, work was carried out on gestures using ELAN (Brugman, Crasborn et al. 2004). Similar work was carried out on emotional voice quality using two major speech emotion recognition systems – PRAAT (Boersma 1999, Boersma and Weenink 2012) and OpenEar (Eyben, Wöllmer et al. 2009, Eyben, Scherer et al. 2016). The gestures in these videos were annotated using ELAN (Wittenburg, Brugman et al. 2006).
3.3.3.2. A Multi-Modal Communications Case Study in Disaster Communications
Much of the work in facial expression recognition (Bartlett, Littlewort-Ford et al. 2014, Du, Tao et al. 2014), emotional speech detection (Eyben, Wöllmer et al. 2009, Eyben, Scherer et al. 2016), and hand gesture identification (Pisharady and Saerbeck 2015) has been carried out on actors making faces, modulating their voices, and making pre-determined gestures. Our work is a significant departure and is part of a shift in research from laboratory to real-world communications (McDuff, El Kaliouby et al. 2015, McDuff and el Kaliouby 2016).
The rise of multi-modal information processing systems – where three different aspects of non-verbal communications (facial expressions, vocal quality and hand gestures) are analysed with simulated and real-world data – addresses the issue of socially interactive computer systems (Poria, Cambria et al. 2016, Poria, Cambria et al. 2017), (Castellano, Kessous et al. 2008, Kessous, Castellano et al. 2010)
To assess the interaction between three modalities, we collected and analysed how public speakers and broadcasters communicate with the public through their facial expressions, voice and body language in disaster situations. This involved the creation of a gender-balanced data set for analysing facial expressions and gestures of authoritative figures (e.g. mayors and governors) who announce impending hurricanes and snowstorms. The data set comprises the governors and mayors issuing disaster mitigation warnings and non-emergency announcements. The data was acquired from YouTube, and comprises 2.5 hours of running data in emergencies and 3 hours of normal conversations. We evaluated the systems through an observational study of how people perceive the governors (Ahmad, Busà et al. In Preparation).
Slice
(Emphasis) Slice extended finger
(Emphasis) Double Slice
(Empathic, openness) Ring
(Precise information)
instructions in our website. emergency disasters a very difficult time
state agencies can respond specifics about damage
Figure 14: SLANDAIL examined typical gestures used by leaders when communicating disaster information to the public to understand how non-verbal communication affects emotion recognition.
In a preliminary study of the three modes we found significant correlation between facial expression and voice quality; gestures were not correlated to the same extent as the other two modalities. This research can be used for training public speakers to deliver congruent messages during emergencies (Ahmad, Busà et al. In Preparation). The work on gesture and emotion analysis will increase understanding of how different stakeholders in a disaster event communicate with each other, often across cultural and ethnic divides. Streamlined access to the Emotient system has been incorporated within the SLANDAIL System to allow communications training as part of the overall system.
3.3.4. Image Analytics for still images and videos
3.3.4.1. Disaster Images: Photo-sharing and crowdsourcing
Images can be processed automatically and quickly to discard those that contain little useful information and to extract as much available actionable information as possible from those with a high relevance. This can be a challenging task: images that have useful actionable data are generally confounded by content that includes debris, objects in unusual positions, and noise arising from the circumstances in which the image may have been captured. An image shot during an abnormal event, like a flooding episode, will be noisy: the weather is likely to have been inclement, the camera may have been shaking, and lighting may have been poor.
The proliferation of images on social computing systems requires the development of systems that automatically categorise images and extract information from the images during an abnormal event. Social media interfaces make it easy to make images available to others. Images can be manually annotated and distributed using social computing systems through photo-sharing (Liu, Palen et al. 2008). Images can be used to enhance situation reporting (Yang, Ha et al. 2011, Yang and Chen 2015), and systems for crowdsourcing image data can help emergency managers (Xu, Liu et al. 2016).
Disaster victims can locate the disaster and the extent of the damage as efficiently as an expert in two out of three cases (Panteras, Lu et al. 2016). SLANDAIL has deployed automated analysis of images to extract useful information, as well as combining information from text and images. Images can play an important part not only in disaster reporting, but also in disaster management by helping to establish a two-way interaction between the public and authorities, as well as interactions between members of the public (Liu, Palen et al. 2008). This two-way ‘visual representation of disasters’ is an important component of disaster management, especially in remote places with poor infrastructure (Bica, Palen et al. 2017).
SLANDAIL has demonstrated how image analytics can aid disaster response by integrating image analytics into the SLANDAIL Social Media Monitor (see Section 3.4.2).
3.3.4.2. An Innovative Image Representation Framework
Given the continuous high-volume of data available through social computing systems, we have developed a Square Spiral Image Processing (SIP; D3.3) framework (Jing 2015) to speed up computation. The software implementations based on SIP facilitate the storage of image pixel indices in a one-dimensional vector based on a spiral architecture. This enables faster access and processing than the conventional representation of images that uses a two-dimensional Cartesian system. For image processing in the SIP framework, fast access to pixel locations in an image is achieved by mimicking aspects of the eye tremor phenomenon in the human visual system, combined with a non-overlapping convolution technique.
The extraction of local image features is based on key interest points using the Scale-invariant Feature Transform (SIFT) method (Lowe 1999, Lowe 2004), which are designed to persist over a range of scales and are robust to changes in conditions under which images are captured. Our approach uses a non-overlapping convolution method that can extract image features faster than conventional convolution techniques. SIP is a fundamental contribution, as it facilitates fast access to image data for processing at the pixel level and accelerates the execution of subsequent image processing algorithms. (Jing, Scotney et al. 2017).
The design of multi-scale SIP operators enabled us to capture features of objects at different levels of detail (Jing, Coleman et al. 2015). Application of the SIP framework has also been extended to video frame processing in which consecutive frames in the video are considered as the eye tremor images that are used for feature extraction based on SIP convolution. The improvement in computational performance has been evaluated for various tasks. For example, the results shown below for detecting edges in images of various complexities, using three benchmark images, Lena, peppers and coins, illustrate the way that SIP typically outperforms the standard algorithms (Table 2) (D3.3).
Table 2: Comparison of edge maps from standard 2D convolution and SIP-based convolution
Algorithm LENA RUN TIME (MS) PEPPER RUN TIME (MS) COINS RUN TIME (MS)
Standard 6.8 61.7 605.9
SIP 0.28 1.1 16.6
Image Size 81 X 81 243 X 243 729 X 729
3.3.4.3. Real-time Image and Video Analysis
In a disaster situation, video files uploaded by victims or observers can provide key information about ground conditions, such as the severity of a storm. Videos of storms uploaded to Twitter and YouTube often feature trees blown by strong winds, strong currents in floodwater, and rapidly rising water levels. SLANDAIL uses these to classify the severity of a storm or flood event. We deployed the concept of eye tremor to video data analysis by considering a sequence of video frames as a set of eye tremor images.
Our evaluation for video analytics focused on videos of blowing trees and floodwater extracted from social networking sites. In terms of storm severity, each sequence in our evaluation set was categorised visually into low, medium or high strength by four observers to create ground truth. A two-class grading scheme was developed to distinguish between low storm strength, and medium-to-high storm strength. An additional source for video data with potential use in flood management is provided by public network web-cams near rivers and bridges. Updated images are often posted at regular intervals, offering a visual indicator of change of water level or volume over a specified interval.
Our experiments demonstrate that moderately dynamic videos can be handled successfully using a video stabilisation technique (Matsushita, Ofek et al. 2006). (See Figure 15).
3.3.5. Multimodal Aggregation
SLANDAIL developed a novel framework that enables the integration of linguistic evidence with image features (D3.4). This approach built on methods developed (D3.1 D3.3) by combining contributions to image and text feature extraction.
Sharing and crowdsourcing of disaster images relies upon humans to categorise images. What is required is a system that not only can categorise and extract information from images with the help of collateral modalities, but also learns to automatically repeat and improve on this process. This necessitates the use of learning algorithms like neural networks that can be trained to recognise characteristic shapes, textures and boundaries and to recognise accompanying keywords and metadata (Hong, Wang et al. 2014). Once learnt successfully, the learning system can then recognise unseen images and annotate these images with keywords for subsequent retrieval (Ahmad, Tariq et al. 2003). We have used Ulster’s SIP framework to analyse images (see 3.3.4.2 above).
Text Representation: For training a neural network, we collect N-images, distributed over C categories, where C
The multimodal framework developed as part of SLANDAIL includes methods for integrating text and image features for image retrieval and image-text aggregation. The method incorporates linguistics evidence from the collateral text of the image, and has been trained using data from an Image Database built for SLANDAIL from social media (Flickr, Instagram, Facebook). The features for image and text data were extracted using content analysis methods developed in SLANDAIL and are aggregated in a multi-modal method to explore connections between modalities, and have been implemented in the CITU system (see below). The multi-modal method uses self-organising maps to learn relevant and key features in text and images for the domain of emergency management and learns the association between these modalities. Combining feature vectors in this way allows messages related to an emergency situation such as flooding to be retrieved more accurately as the image and/or text content can be used in classification and retrieval. This approach and the CITU system allows recognised images to be annotated with text or keywords and allow sample images to be given for certain keys words or text features. Combining modalities in this way and learning associations between images and the collateral text means the classification of messages in a social media stream can be improved.
The CITU system and underlying methods were developed to allow annotation of images and to aid classification of emergency related messages retrieved from a social media stream. CITU is a desktop application with a user-friendly GUI (Figure 16) and allows a user to: (a) automatically analyse images using image segmentation and features based on colour, texture, and shapes detected; (b) automatically process collateral text based on the frequency and collocations of keywords; and (c) automatically learn associations between image features and textual features. We looked at using the multi-net framework of CITU for emergency management by annotating unseen disaster related images with keywords to improve classification.
The advantages of using image and text features together to improve overall image classification has been highlighted in (Jing (b), Scotney et al. 2016). Using image and text modalities together the accuracy of classification can be improved for the application chosen in SLANDAIL, to classify and index emergency related messages (Table 3). Building on this work, the use of image annotation and self-organising maps to link text and image modalities to improve message classification is currently being prepared in (Kelly, Ahmad et al. In Preparation).
Table 3: Comparison of recognition accuracy and performance using self-organising maps for the image and text collections collected as part of SLANDAIL.
Image Only
Accuracy Image and Text
Accuracy
Minimum Maximum Minimum Maximum
General Flood Corpus 50% 51% 66% 68%
Flooded and Submerged Corpus 75% 80% 65% 95%
This approach of using all the available content of a social media message has meant overall recognition can be improved compared to just using an image or text alone. The annotation method can be trained off line and used in conjunction with the image classification and SIPF algorithm (Jing (b), Scotney et al. 2015) to perform real-time processing of social media messages and their image and text content.
We extended and tested the approach for recognition of flood-related events such as submerged cars and road-signs. This work was implemented in the SLANDAIL Social Media Monitor. This is a considerable achievement in terms of technology transfer in computer vision, enabling emergency managers to act on information extracted automatically from images that have been harvested from social media or other contributing sources. The software can alert emergency managers to a developing disaster and help them to focus in on and assess the most relevant information available. The prototype interface for emergency managers in shown in Figure 17.
Figure 17: SLANDAIL Image User Interface for emergency management: a) Images are categorized (flooding, fallen tree). b) Alert status of a geographical location - a rise in alert status (red line) merits attention; blue line is classification on an individual image basis. c) Images are then aggregated into various sub-categories and displayed as a pie chart - hover over or click various sub-categories to retrieve further information. d) A ranked image set displays the three most pertinent images relating to a selected category from the pie chart.
3.3.6. Multi-Criteria Decision Making
SLANDAIL developed a method of aggregation and decision-making that relies on fuzzy integrals and aggregation theory to bring together data typically used by emergency managers with outputs from social media. This includes a scoring mechanism that uses various pieces of relevant information, including vulnerability of an area, economic information and geological information. We have added social media analysis to this list of available data. This has resulted in the development and publication of a decision-making model that integrates social media messages with other information including geospatial information.
The SLANDAIL System uses geospatial information used in typical emergency management systems, with processed social media data (cf. section 3.4.2). It aggregates conventional data sources with advanced machine learning methods for image and text analysis. The output of this analysis is a signal that can be used to help decision makers make more informed choices based on the available information. Many open data sources exist that can be used as valuable sources of geospatial information: INSPIRE, for example, contains spatial data for all EU countries and is mandated to include this data, supplied by local authorities.
Using our model, specific regions or locations can be ranked to get a complete map of vulnerability by combining the available data and assessing the risk. An example of sample data variables has been presented in (Mesiarová-Zemánková, Kelly et al. Forthcoming), in which the output of the classification for social media data (which determines relevance and reliabililty) forms a social media index. Our method combines relevant variables in a way that reflects an expert’s decision during an emergency event. The weighted combination of the data results in a normalised output value representing the magnitude of risk, or severity, of the situation based on available data.
Figure 18: Our method determines the level of risk and vulnerability for a region by combining relevant variables: Geography (G), Accessibility (A), Infrastructure (I), Population (P), Current condition (CC), and Conditions in catchment area (CA).
Figure 18 shows the input variables for a region and their “score” as determined by the data passed into the model. The model aggregates data in three different ways, as illustrated with different models (AWBM, AW, A), giving an output and level of risk based on the input data. This analysis allows an emergency management system to offer information on key locations that might be high risk, based on a number of variables including social media.
3.3.7. Multilingual Aggregation
The SLANDAIL Prototype IV, originally created in Italian, was successfully localised into English and German. A key asset for completion of this task was the SLANDAIL Lexicon, which was used to translate approximately 20% of the messages of the interface. We identified challenges to localisation, mainly consisting in issues of interpretation arising from the fact that strings are presented as out-of-context chunks of text and the need to accommodate varying word lengths to the space available. The quality of this work was tested by both English- and German-speaking End Users, who provided feedback to ensure usability and communicativeness (cf. Section 3.4.4.3 for full description). Software localisation is an expensive and time-consuming task as it requires the collaboration of different experts at various stages of the work: by managing the entire process internally, we achieved cost-effective and high-quality results.
Natural disasters do not respect borders: the European Union has 24 official languages, and many regions that share disaster risk zones do not share a common language. A review of current emergency management systems showed that most are available in one language only (usually English) and no provision is made for porting them into other languages, while others are machine translated or localised through crowdsourcing.
Note: Use of background IPR: Software systems
3.4. CONTROL ROOM TECHNOLOGY: Building a Social Media-Enabled Emergency Response System
3.4.1. Introduction
In the final eighteen months of the project, SLANDAIL consolidated the methods and analysis developed earlier into an integrated system accessible via a single screen. The resulting prototype assimilates End User requirements as well as the findings from our ethical and social analysis, and enabled us to complete the third of our objectives: build and test a prototype system for collecting, processing, aggregating and disseminating information for disaster emergency management in an ethically and legally sound manner.
At a Project Development Meeting in Dublin in January 2016 attended by all partners, we assessed SLANDAIL’s progress to date and determined the necessary steps to be taken to deliver an integrated system (Figure 19).
This involved the integration of the Social Media Monitor with the Emergency Management System to allow them to share relevant information. End Users – who were closely involved in evaluating and testing the prototype – can now access aggregated data from social media sources within an emergency management environment.
We transferred the Image Analysis software, integrating the automated processing of social media images with the SLANDAIL System. End Users can now perform multiple actions within a simple, consistent and user-friendly environment. The integrated system enhances the situational awareness of emergency managers by leveraging geospatial informational resources together with information extracted ethically and legally from social computing systems.
Our innovation has been to interface a disaster management system to a social media monitor. Initially, we coupled the media monitor (Topic Analyst) to the EMS (SIGE). Subsequently, CID confirmed that they were unable to provide live social media data for testing and evaluation due to commercial and contractual reasons relating to German copyright law. In the latter stages of the project, we developed a new Social Media Monitor that incorporates a privacy intrusion detection system: this is used to supply text and image data to the system.
SLANDAIL synthesised research on trust into a set of communications guidelines for End Users. These guidelines are the outcome of linguistic and terminological investigations and provide emergency managers with essential information on how to generate and maintain trust with citizens before, during and after crisis events. Our research on non-verbal communication included an analysis of facial expressions and gestures – elements of communication that have a universal, language-independent valence – and will act as a platform for the creation of training tools for public figures required to speak in crisis situations.
The SLANDAIL System enables emergency managers to communicate information effectively during crises. Our crisis communication templates, based on End User evaluation, enable communication with citizens through the system. The templates are based on corpora collected in three languages and have been tested for clarity and readability. The system was localised in three languages and has been adapted to potentially localise in other languages using structured terminology databases.
The integrated emergency management system can be adapted to different geographical settings (it can incorporate different cartographic projections with no software changes) and enables End Users to perform a range of complex analytical actions within a user-friendly environment. It can successfully gather and process information from social media, aggregate and interpret the data according to relevant metrics, and circulate messages to citizens. As such, it presents a powerful new set of tools to reduce the existing burden of information processing upon emergency managers and enable them to respond more effectively to threats.
3.4.2. Integration of Emergency Management and Social Media Monitor
Figure 20: The work in other areas of the project, including user requirements, technical developments and legal licence, are all integrated to some degree in the final SLANDAIL system
SLANDAIL’s first prototype leveraged web-based information for use in an Emergency Management System (EMS). Existing software systems by DataPiano and CID were coupled to share relevant information regarding emergency analysis and disaster management.
3.4.2.1. Deployment of Emergency Management System for User Needs
DataPiano deployed its Emergency Management System, SIGE, in a dedicated server environment. SIGE provides crucial database information to emergency managers and includes a locative system and custom-developed GIS mapping system to assist End Users in finding key areas during a natural disaster. This emergency management system, used by Italian Civil Protection since the early 2000s, formed the basis for the SLANDAIL System.
3.4.2.2. Coupling of Emergency Management System and Social Media Monitor
In April 2015, the coupling of the Social Media Monitor (Topic Analyst) and Emergency Management System (adapted from SIGE) was demonstrated to End Users at a Usability Subcommittee Meeting. The first stage of coupling was the development of an API schema document to provide an input-output protocol (API) and define each partner’s role in the project, facilitating subsequent integration phases. This enabled the complete transfer of documents from Topic Analyst to SIGE (including metadata), and the filtering and searching of documents from SIGE via API in Topic Analyst.
We held a series of discussions on the needs of End User beneficiaries. During evaluation, End Users highlighted that social media is only one strand of information and cannot the core source. As such, our Social Media Monitor was developed to incorporate social media data into an existing emergency management system. These discussions led to the definition of requirements for the SLANDAIL system, and the meetings of the Technical Subcommittees led to the development of a SLANDAIL User Requirements Document, which shaped the design of the coupled prototype. The first step was the creation of a SLANDAIL-specific deployment of CID’s Topic Analyst with a CORPUS system, a Social Media Data Capture and Visualisation System (SMDCVS) in a dedicated server environment.
Topic Analyst provides an analytical instrument for filtering, search, investigating and data mining on huge amounts of documents collected and pre-processed by CORPUS back-end as well as academic partners. Its features are language processing, entity recognition as well as lemmatized keyword extraction. Together with its SLANDAIL-specific deployment, Topic Analyst was equipped with a GeoInformationService that offers the capability to reference geolocation coordinates on a given map as well as performing filtering functionality. Due to legal restrictions on the use of real-time text collection and analytics from Topic Analyst, the Social Media Monitor was used instead.
The first working, coupled social media monitor and emergency management system enables emergency managers to not only analyse social media during times of disaster, but also to regulate and manage their response to disaster notifications.
3.4.2.3. Deployment of Social Media Monitor in Line with End User Needs
The SLANDAIL Social Media Monitor amalgamates many of the technologies developed within the project. It combines spatio-temporal metadata of social media messages with an analysis of the complete content – namely, the text and images – of each message. The system enables the extraction of real-time information from social media stream by incorporating text and image analytics and geospatial mapping within a multimodal analysis framework (Zhang (a), Kelly et al. 2016). It is supported by text and image analysis technologies, described in detail in Section 3.3.
Utility: The system utilises sophisticated filtering mechanisms to ensure that the data extracted is relevant. For example, the Monitor collects data from a geo-constrained data stream, ensuring that disaster managers – who will be responding to emergencies within specific geographical regions – receive data relevant to their area (Kelly and Ahmad 2014, Kelly and Ahmad 2015). Additionally, the Monitor draws on several NLP (Natural Language Processing) techniques as well as language patterns identified by experts as being common to disaster events (such as those relating to road blockages and power outages) (Toutanova, Klein et al. 2003, Derczynski, Ritter et al. 2013, Chang and Manning 2014, Spyropoulou 2016) to filter out unnecessary information. The system also detects key words in messages containing images as well as using a pre-trained image classifier to detect flood images (Jing (b), Scotney et al. 2016). All this information is mapped onto GIS systems. The mapped areas highlight outlier information that may not be apparent from existing GIS information, such as flood preparedness levels or economic vulnerability. This allows it to take advantage of the full content of an image and maximise the effectiveness of its analysis (Figure 21).
Architecture: At its core, the SLANDAIL Social Media Monitor is powered by a concurrent data processing pipeline that applies the analytic algorithms to the social media stream (Section 3.3). We added a web application that implements the visualisation of geospatial mapping and provides End Users with an intuitive interface. The system contains a number of components, including: A Social Media Streaming Access Module (which connects to live social media streams); an NLP Pre-processing Module to filter incoming text; a knowledge base, the Semantic Disaster Lexicon, containing common language patterns relating to emergency events; and a Geospatial Mapping Layer that infers (and, when possible, detects) the geo-locational information of the message. The Intrusion Index is a text analytical tool that detects and visualises the extent to which sensitive information (such as individual names, institutions and places) is being collected (Kelly and Ahmad 2014, Kelly and Ahmad 2015).
User Interface: The interface of the Monitor is implemented as a web application and allows End Users to access the aggregated results of the analysis in a clear and intuitive way. Red markers indicate message with native geo-coordinates attached, while green markers represent message with locations inferred from text contents (Figure 22). Such capability makes it possible to reuse semantic dictionaries that contains language patterns that capture common themes across different types of emergencies. Users can view the details of individual messages by clicking these markers; a time-series plot allows an overview of events as they unfold. Evaluation of the Hurricane Matthew event (2016) showed the occurrence of power outage and road closure information in areas, often preceding other sources of information.
Figure 22: Front end of the social media monitor system showing filtered and relevant geolocated messages and images for the location under investigation, the frequency at which these messages have occurred, and a drill-down menu to examine the messages and accompanying metadata. The monitor can be configured to use different semantic dictionaries when tracking different types of events (e.g. flooding, fire, crowd control). Furthermore, the system allows the user to use multiple dictionaries at once; for example, the user may combine a semantic dictionary containing only language patterns related to flooding with another dictionary that tracks general sentiment words.
The SLANDAIL Social Media Monitor can be deployed online on a standalone web server, allowing it to be operated within an existing emergency management control room, and it can be viewed in mobile and standard web browsers for demonstration purposes. Much of the recent work of the Social Media Monitor has been summarised and published in a forthcoming paper accepted at the International Conference on Information Systems for Crisis Response and Management (Kelly, Zhang et al. 2017).
3.4.3. Building of Final Demonstrator
3.4.3.1. User Interface Design
SLANDAIL’s User Interface was designed with the primary requirements of being simple and communicative. We determined that End Users should not require specific technical guidance, and that navigation through the various functions of the program should be intuitive. Additionally, the nature of crisis operations demands that Users can perform every operation with the lowest possible number of clicks. The Dashboard contains large buttons, or tiles, that communicate their function through simple icons and is divided into three sections: Quick Start, Ongoing Incidents and Tools (Figure 23). These sections allow End Users an immediate overview of the current emergency situation.
The emergency management UI is structured as a “wall of monitors” featuring interchangeable panels that show essential information: the generic event data (type, location, etc.), operations carried out to date, resources or facilities involved in each emergency stage, messages sent, and documents relating to the event. These “monitors” are automatically updated and can be moved, closed or substituted with others (showing a list of messages sent or received, personnel available in a hospital) within the screen area.
Figure 23: SIGE’s dashboard is divided into three sections: Quick emergency, Ongoing Incidents, and Tools, each with user-friendly start tiles to allow End Users to quickly navigate the emergency situation.
The main interface is localised using a special text file containing terms, words and phrases used in interface command names and tiles: a new supported language can easily be added. This localisation of the software, a collaborative endeavour involving linguistic and communication expertise, ensures that SLANDAIL’s interface language is consistent with the language used by local emergency management operators. End Users tested and evaluated the interface and terminology to ensure its accuracy and communicative effectiveness.
End Users are required to accept the terms of the Licence Agreement when logging on to the system
(Figure 24). This mitigates the risk of infringement of the human right of the data subject or owners, the copyright of the data owners, and the relevant personal data protection and privacy provisions of the EU. The Licence highlights the fact that End User organisations have a responsibility to negotiate agreements to secure permission from the data owners (usually the social media providers) to collect and analyse the data, and a responsibility to provide training for users. It specifies when data should be archived off the live system and outlines the protocols that should be in place to provide for its appropriate handling by authorised persons prior to its timely destruction.
3.4.3.2. Incorporation of Text, Speech, Non-Verbal Communication and Image Analytic Systems
Building on the initial coupling of SIGE with Topic Analyst (D4.1) and the adaptation of the system GUI to End Users’ linguistic and cultural contexts (D3.5) we built methods for including advanced text and image data into our system (D4.2). Existing technologies were migrated to web-based technologies to enable multi-agency response, and integrated into a user-friendly interface. End Users can now perform multiple actions (including complex GIS queries) within a simple, consistent and user-friendly environment.
TEXT ANALYSIS: We integrated a topic modelling module into the coupled SLANDAIL system (D4.2). Social media data from Facebook and Twitter was processed to automatically retrieve the topics contained in the corpus and in individual messages. The data can be easily filtered, aggregated and combined with other results via the graphical user interface of the Topic Analyst module.
In addition, an API library, the SLANDAIL power outages, was developed; this library encompasses the incident mapping and intrusion detection methods based on the analysis of the textual content of social media messages. The methods detect the uses of certain language patterns from social media streams that often indicate the occurrences of certain key disaster incidents (e.g. road blockage, fallen trees, power and water outages, people getting trapped, etc.). The results of these analyses are organised into structured formats so that third-party applications can integrate with the API and gain access to social media analytics. This API library has been integrated into SIGE, enabling it to monitor social media streams within their system.
SPEECH AND NON-VERBAL COMMUNICATION: We successfully integrated the non-verbal communications module by connecting the Emergency Management platform with iMotions, a software system for video analysis, via its API. The integration was implemented as a web application, and is thus accessible to End Users through web browsers. The system is designed to provide users with a “one-button” experience when analysing videos through the iMotions software by automating setup and teardown processes as much as possible.
Users can upload videos for analysis to assess emotional communication before uploading disaster messages via Facebook or YouTube, for example (Figure 25). Two visualisation options then become available, displaying the dominant emotion class and the trending of different emotion categories. The efficiency of emergency and disaster-related message delivery is crucially based upon not only on verbal content, but also on non-verbal signals. This addition to the system, along with the communication guidelines introduced with D1.4 will aid emergency managers and public relations representatives in developing communication skills.
IMAGE ANALYSIS: SLANDAIL tested the transfer of Image Analysis software, completing integration in February 2017. We ported the image analytic algorithm into SLANDAIL’s Social Media Monitor, thus integrating the automated processing of social media images with SIGE’s emergency management system. Following this integration, SLANDAIL System is equipped to automatically recognise flood events, and to deliver an automated alert when the volume of these images reaches a certain point.
Each image receives a classification score, designating a flood or non-flood image. The amount of flood images determines the overall alert status of the system according to a three-level scale (Low, Medium and High). When the alert status reaches High, a notification is automatically sent to SIGE to alert the emergency manager to the presence of a large quantity of flood images. This allows the End User to create an active event tile, which aggregates all the live information relating to a geographical location.
A simple implementation of the social media monitor involves a comparison of the number of posts (including text and images) that contain emergency-related words or images. We have developed a more complex index based on individual scores for each post based on its level of relevance, credibility, and affect. A message’s relevance is determined by analysing its content to detect textual references to an emergency event or images recognised by the classifier. With the credibility score, heuristics drawn from the literature (AlMansour, Brankovic et al. 2014, Alrubaian, Al-Qurishi et al. 2016) can assess whether the post is from a credible source. This helps determine how reliable the message is likely to be. Sentiment analysis techniques can be used to compute affect, giving an indication of a message’s urgency. The interaction between these inputs and their aggregation into an individual post score is modelled using the Choquet integral (Choquet 1954). The overall aggregation of outputs from social media content filters out most messages, identifying only the most relevant and reliable.
3.4.3.3. Trust-Building and Communication
SLANDAIL research on trust-building synthesised findings from several research strands (D1.4 D2.3 D3.2). We produced guidelines for End Users on how to communicate in a trusted manner across a range of platforms before, during and after a disaster event. The value of messages being sent out by emergency managers is dependent on clear and relevant communication, necessitating a strong communications plan. Our research shows that trusted communications are established before, during and after a disaster, forming part of a continuous dialogue between emergency managers and the public (Busa, Musacchio et al. 2015).
Linguistic and terminological investigations of trust communication revealed that: (1) clear and simple messages contribute to successful bonding with the public; (2) trust is first established in times of peace, when the public can become acquainted with terms used during disasters; (3) informing, reassuring and appealing to the public ensures that people will feel safe and well-informed during an emergency; (4) “emotional contagion” (Quasthoff, Goldhahn et al. 2013) occurs when individuals are exposed to emotionally loaded content: hence, transmitting a balanced perception of risk can engender trust (Hodas, Ver Steeg et al. 2014, Kramer, Guillory et al. 2014, Musacchio 2014). Agencies need to devise effective and timely messages anchored on shared values (Musacchio and Panizzon 2017).
SLANDAIL gathers basic, language-independent gestures to complement affective information from facial expressions and speech analytics. In emergency situations, it is important that authority figures use non-verbal language that: 1) does not show emotional involvement that could spread alarm; 2) is not misunderstood by people of different cultures; 3) shows professionalism and conveys trust in the public. Non-professional behaviour or speech patterns may reduce the public’s incentive to act in a disaster situation.
Our work also showed that for public communication: (1) Speech cues (pitch, intensity and speech rate) must be modulated to convey credibility for successful trust building; (2) These should be coordinated with facial expressions, body posture and gestures, and the general purpose of non-verbal language should convey respect and trust.
3.4.4. System Testing
3.4.4.1. Test Data and System Testing
Continuous cooperation between SLANDAIL beneficiaries was achieved through Usability Sub-committee meetings, End User requirement reports, and presentations on existing data management systems in End User locations. To ensure the functionality and effectiveness of the SLANDAIL prototypes, End User partners provided test data and created case studies. As well as End Users influencing and subsequently testing the prototype system, two independent tests of the system were carried out by emergency planners representing the Business Continuity Institute and the Emergency Planning Society to validate the tests, with support from Media Foundation of Sparkasse Leipzig (Figure 26).
A final comprehensive evaluation event was hosted at Trinity College Dublin on 13th September 2016, allowing End Users a further opportunity to test and tweak the prototype. While localisation configuration will be required in advance of real-life application, the system’s functionalities were fully demonstrated and understood. End Users reported satisfaction with the abilities of the system and were impressed with its potential for use in emergency management. We determined that – subject to a separate examination of ethical and legal impacts – the system has high potential for use as an Information Management tool in other End User scenarios, such as manmade disasters and major event management. The results of these tests facilitated further development of the prototype. The findings regarding further potential uses for the system stimulated commercialisation discussions, as well as flagging other potential R&D opportunities.
3.4.4.2. Building Clarity for Communications
For SLANDAIL, the issues of Plain Language – a pan-EU and national aspiration – and effective communication are central. Surveys on existing communication guidelines and protocols and interviews with End Users were coupled with conclusions gained through corpora analysis and terminology work. The aim was to develop a methodology for automatic assessment of key metrics for measuring readability and comprehensibility. This allowed us to compare a variety of language registers – in major EU languages (EN, DE and IT) – and use representative corpora to understand usage.
This multilingual study, the first of its kind to our knowledge, has led us to develop templates for releasing information about major disasters in line with plain language requirements through a range of media including text message, Facebook, Twitter and press releases. End User beneficiaries all identified a need for model messages drafted to suit the requirements of different media and different stages of an emergency. By studying the structure of warnings in a range of disaster management documents, we generated models to integrate prototypical communications into the SLANDAIL system. These messages were designed and assessed to achieve a high level of readability and acceptable lexical density.
Based on this analysis we created prototype messages in English, German and Italian and adapted these for presentation in different formats (e-mail, Facebook post, text message and Tweet). SLANDAIL provides End Users with ready-made templates adaptable to specific incidents. These can be used as a base to draft communicative and effective texts when time is of the essence. The structure and language chosen for templates have been tested for communicativeness against the source texts. English messages have been successfully tested through the Plain English Campaign software, and results suggest that our messages can be read and understood by a wide audience across various education levels in each language.
3.4.4.3. Localizing SLANDAIL’s Outputs to Multiple Languages
Following professional standards published by localisation experts, the system GUI was successfully and cost-effectively translated into English and German by adapting its textual elements to End Users’ cultural backgrounds (D3.5). SLANDAIL terminology, the SLANDAIL corpora and the INSPIRE webpages were used to localise the SLANDAIL prototype. We achieved key results in expanding the multilingual communicativeness of the prototype system by collecting and investigating corpora in three languages covering both formal and social media. The knowledge gained from metrics along with the assessments of communications experts and End Users has led to the creation of actionable information – crisis communication templates – that can be integrated into the SLANDAIL system and sent out to warn and inform the population.
Figure 27: The SLANDAIL System localised from Italian (left) into German and English (right)
Potential Impact:
4. Potential impact and main dissemination activities and exploitation of results
4.1. Contribution to the State-of-the-Art
Social computing systems carry an enormous volume of relevant and timely information, and have proven to be a useful method for communicating when traditional lines of communication are unavailable or inoperable. There are strong indications that social media is playing an ever-increasing role in the management of emergencies, and can play a key role in bridging the gap between traditional media reporting and adequate resource deployment (Young-McLear, Mazzuchi et al. 2015).
When integrated with geo-location systems already used by emergency managers, information harvested from social media systems can be a vital resource for the coordination of emergency response. A combined geospatial information system and social computing system in emergency response will be the next step in the evolution of disaster management, allowing citizens to participate in the processes of gathering and sharing vital information (see, for instance, Maresh-Fuehrer and Smith 2016). Research to date has identified the potential for pilot programs and public-private partnerships to integrate social media usage into disaster management (Lindsay 2016).
SLANDAIL research has laid the groundwork for such a combined system, delivering advances in informational analytics tailored to the emergency management environment. Our work addresses crucial ethical, social and legal issues surrounding the use of social media in disaster management, and the findings from our cross-disciplinary analysis of these factors are embedded in the SLANDAIL System. Contributions have been made to text analytics in the areas of topic modelling, ontology, and terminology, time series analysis using social media as an input, image analysis, in to aggregation studies. We have also extended the state-of-the-art in the ethics of social media use.
The System and its accompanying platforms have introduced ethics into data collection approaches in novel ways, informed academic approaches to systems development with real-world End User experience, and helped emergency managers to incorporate technical and ethical expertise into their practices.
4.2. Key Achievements
4.2.1. Legal Licence Agreement
Ethical and social considerations were embedded in SLANDAIL from the beginning of the project, and our legal and ethical research had considerable impact. It identified measures necessary to the protection of data privacy for citizens, which then took the form of technical functionality implemented in the prototype. The research undertaken in this strand of the project culminated in the development of a model Licence Agreement – the first of its kind – for a social media analysis system for disaster response. SLANDAIL legal caucus led work to develop a software license suited to systems that bridge the divide between privately-generated content and public (or commercial) re-use, and where ethics and provenance are both relevant and important. With EU Data Protection Legislation soon to be implemented (de Hert and Papakonstantinou 2016), issues of privacy and data protection are emerging as key ones in determining future policy in relation to the use of social media in emergency management, and SLANDAIL advances have made significant contributions to the ethical and legal landscape of emergency management.
4.2.2. Terminology Management
SLANDAIL has made terminology management as automated as is currently possible. We completed the development of a terminology wiki, evaluation and refinement of automated term extraction (Musacchio, Panizzon et al. In Preparation), and the development of a method for automatic extraction of definitions, contexts and ontologies. Key results lie in the use of terminology for localising user interfaces broadly (Schmitz 2015). The development of terminology and the creation of ontologies has contributed to avoiding a key knowledge acquisition bottleneck. The SLANDAIL lexicon is currently being integrated into the IATE (InterActive Terminology for Europe) database, the EU’s terminology database, in line with guidelines provided by TermCoord, the EU’s terminology coordination database. This will make SLANDAIL’s terminology in all three project languages available to European institutions.
4.2.3. Integration of Image and Text Analysis into Control Room environments
The project has developed methods to integrate image analysis with the results of text analysis and has built a solid foundation for multimodal information aggregation. SLANDAIL goes beyond the state-of-the-art by developing an image recognition framework that automatically recognises specific disaster-related images. The framework also maximises the use of rich information (such as metadata) by integrating text analysis with the image analysis system to enhance the performance for retrieval of flood-related images. The image processing framework implemented in SLANDAIL for image analysis uses a newly developed SIP (Square-Spiral Image Processing) architecture that enables fast computation of image convolution. The SIP framework can be incorporated into standard image processing systems, making the image analysis system extensible.
Additionally, the SLANDAIL System creates a multi-modal feature vector based on image features and results of text analysis. The fusion of text and image features also enables the development of a higher-level system involving more complex ontologies for multiclass image classification and annotation (Kelly, Zhang et al. 2017). The image analysis is extended to the analysis of video data (for video content retrieval). The final platform can incorporate the image analysis outcomes for geo-located and timestamped live social media data analysis.
The integration of text analysis from social media into existing disaster management control room systems is a major impact. The development of a live Social Media Monitor that performs advanced text analysis on large bodies of social media text provides analysed data in real-time. Furthermore, the flexibility of the analysed results to be mapped onto GIS or similar mapping systems allows for the visualisation of text analytic results in real-time during a disaster event. The Social Media Monitor developed on the project is designed to work with any standard control room operation using GIS models and will be further exploited beyond the end of the project to enhance control room technologies in emergency management.
Furthermore, in Natural Language Processing research and crisis informatics (Sorensen and Sorensen 2007), we found that topic modelling techniques were suited to social media analysis in disaster management. Well-trained models for flood-related Twitter and Facebook data provide a means to automatically filter, sort and analyse social media data, and represent powerful tools to cope with the problem of information overload in disaster management application and research (Kelly and Ahmad 2015). Using the SLANDAIL System, emergency managers can monitor prevalent data trends and patterns to see the different communicational phases of the disaster event life cycle and link these to the different phases of a disaster event: preparedness, response, mitigation and recovery.
We focused on the extraction of actionable information from live social media traffic during emergency events. Such information includes knowledge about incidents that may influence decisions made by responders when dealing with emergencies. The analysis potentially enables End Users to make better plans as to where to dispatch resources and rescue services in addition to their existing resources.
4.2.4. Improvements in Disaster Management Communication
Disaster management agencies are increasingly adopting social media as a channel through which to disseminate information and coordinate recovery efforts. Recent research has found that communication – in particular, social media communication – has a positive impact on disaster preparedness, mitigation, and recovery (Westerman, Spence et al. 2014, Brengarth and Mujkic 2016).
SLANDAIL developed a method to analyse, translate and produce messages that can be applied in languages other than the project ones and extended for use in other emergencies (Musacchio and Panizzon 2017). The SLANDAIL System GUI has been developed with the potential to port the system of analysis for use in other languages quickly and cost-effectively. The emergency management systems we reviewed exist in one language only (usually English) and no provision is made for porting them into other languages. This brings localisation beyond the state-of-the-art as it is an example of “glocalisation”: the original GUI language was a “local” language, Italian, which was then “globalised” into English and finally “localised” again into German.
The development of templates for communicating disaster messages across a variety of media, including social media, is a further impact that can be used in future emergency planning (Musacchio, Panizzon et al. 2018 (Under Review)). These templates, which can be sent automatically from the SLANDAIL System, are customisable and contain recommendations of good practice for communication between emergency management teams and the public.
SLANDAIL explored methods of extracting data on non-verbal language (speech, gestures and facial expressions) to complement the meanings expressed by texts in disaster communications. Our work on non-verbal communication analytics will provide the blueprint for a platform that can accomplish real-time analysis of videos retrieved from the web and detect emotive content by multimodal integration (facial expression, speech, gestures analysis). Research papers leading on from the project in non-verbal communications and disaster communications will soon be published (Musacchio and Panizzon 2017, Musacchio, Panizzon et al. 2018 (Under Review)).
4.3. Dissemination
4.3.1. Overview
SLANDAIL is addressed towards the universal real-world need of all citizens for effective emergency response in cases of natural disaster. The project also targeted improvements in areas of specific occupational expertise and delivered advances in a range of academic fields. As such, the results of our work are of interest to a range of audiences, both specific and general.
We undertook a comprehensive dissemination programme to communicate the mission, progress and results of the project to specific target audiences. At the beginning of the project, we developed a dissemination strategy comprising multiple communication plans: this was designed to communicate with six key audiences (first responders, industry, the public, advocacy and representative groups, the media, and other related projects) through carefully-crafted key messages.
Our dissemination and communication activities throughout the project ensured that both message and channel were optimised for the audience(s) to be reached. Unlike many Information and Communications Technologies (ICT) and security projects, our primary target audience was not solely technologists, scientists and researchers: of equal importance were first responders and security agencies (police, civil defence, emergency response teams), as well as ethicists and legal experts.
Our strategy included the creation of the SLANDAIL website and the SLANDAIL Disaster Newsletter to create public engagement as well as targeted press releases, as well as a digital magazine to communicate with interested stakeholders and create demand for the information and the analytical systems developed in and after the project. The project attracted attention in the mainstream news media as well as in publications dedicated to business and technology (details found in Appendix A2). Publications in peer-reviewed journals have been a key part of the SLANDAIL dissemination strategy. We have delivered several publications in high-ranking journals to date (listed in Appendix A1). SLANDAIL members have presented research at many international conferences, ensuring that the project achievements were communicated in a range of academic fields – scientific, humanities, and legal. Project coordinator Prof. Ahmad has been appointed to the International Editorial Board of the University of Padova’s journal Peace Human Rights Governance, and a book on the topic of Dignity, Privacy and Survival in relation to the use of social media in disaster management is being prepared for submission to a major publisher (Ahmad In Preparation).
To identify, manage and protect the intellectual property rights relating to the project technologies, SLANDAIL developed an IP Protection Plan. We produced a Commercialisation and Exploitation Plan to maximise impact and research value and achieve the second of our Exploitation Objectives (“map out commercial and research development of the project results in the future”). Advanced conversations are ongoing in the potential to spin out a company from SLANDAIL technologies, offering social media services in emergency management contexts.
The collaborative, multidisciplinary nature of SLANDAIL has been one of the projects’ key strengths. We created a Virtual Research Community to enable the continued cooperation of End Users, experts in academic and legal fields, and technology partners beyond the close of the project.
4.3.2. Website Design & Implementation
At the beginning of the project, SLANDAIL launched a website (www.SLANDAIL.eu) (D6.1). This was built upon the open source content management system WordPress and displayed a brief introduction to the project along with an attractive set of icons to illustrate the various emergency situations that the project will consider. The site was responsive and linked to SLANDAIL’s social media accounts. Throughout the project, the website was updated with new project content, news and information about project activities, and publications.
Through the website, users can also access the SLANDAIL Newsletter, which aggregates news on disasters as well as providing automatic news analyses. The Newsletter visualises the results of the Summary of High Incidents (SHI), which harvests and analyses data from selected sources. The result is a monitoring system which is updated daily according to SHI-provided results, and can be publicly surfed by a wide audience, contributing to both the dissemination of SLANDAIL outputs and the detection of potential early warnings for emergencies (See Section 3.3.2.1 for more information).
We have an active social media presence, both by presenting our own work and by selecting and highlighting relevant social media contributions by others that demonstrate, illustrate or complement key points from the project. As a result, our Twitter following has increased by over 100% during the second half of the project. Our Facebook presence also continues to grow, with periodic posts and updates.
4.3.3. Media Management
The multi-disciplinary nature of SLANDAIL, and the way we address topics of significant public interest, has meant that the project has enjoyed significant media interest and coverage. SLANDAIL created a logo, a strong web presence (website, Twitter, Facebook), the SLANDAIL Disaster Newsletter, a terminological database (wiki), released a series of press releases (modified for multiple partners), and built a mailing list which was then used to distribute our digital magazine (to serve as a brochure for interested stakeholders) (Figure 29).
Figure 29: The SLANDAIL Magazine communicates the mission, progress and results of our work, serving as a brochure for interested stakeholders.
The project had a dedicated media contact in the Press Office of the Faculty of Engineering, Mathematics and Science at TCD, and the team was available for media interviews and other interactions with the media on request. The project has received coverage in The Sunday Times, The Journal.ie (Ireland’s leading online newspaper) and TechCentral. In the business media, the project carried out knowledge transfer and gave presentations to the Business Continuity Ireland Forum (Dublin), and delivered the Emergency Management Research Symposium (Dublin). A large editorial feature on the SLANDAIL system featured in the 112 Emergencies magazine, 2015, which has an emergency management circulation of over 30,000. A final press release for project results is currently being prepared for circulation.
The project was active in academic media, with a range of publications and presentations that reflects the multi-disciplinary nature of the project and the timeliness and relevance of the subject matter. Examples included presentations at the DIMPLE workshop in Reykjavik (where the project also contributed to workshop organisation and editing of proceedings), UCAml in Belfast, SWDM15 in Florence, TKE2014 in Berlin, COSMIC FP7 final Conference in Brussels, the EU Social Media and Crisis Management concertation event in Brussels, ISCRAM in Kristiansand, Norway, at ICIP 2015, Quebec, the 20th European Symposium on Languages for Special Purposes in Vienna, IAPR MVA, NLP 4CMC/GSCL in Essen, Germany, I4CM DRIVER in Berlin, MUININ 2016 and LREC, Portorož, Slovenia.
SLANDAIL presented the project to various emergency management groups. The project’s research and results were presented to An Garda Síochána Major Emergency Planning Unit on December 8th 2016. A follow-on presentation was given at the Cross-border Working Group on Major Emergency Management meeting, Town Hall Enniskillen, on January 25th, 2017 (Figure 30). This meeting was attended by principal response agencies from Ireland and the UK, including all those responsible for dealing with major natural disasters: Police, ambulance and fire services, local authority, and members of Ordnance Survey Ireland.
A video documenting the project results was created in early 2016. It was developed by professional filmmakers to clearly disseminate the positive results of the project and circulated to End Users and other interested parties via email, as well as uploaded to social media channels. A second video is currently in post-production, documenting the successful development of the Social Media Monitor.
Figure 30: Shane Finan, Xiubo Zhang (TCD) and Rebecca Bury (Stillwater) presented the project to an international meeting of emergency planners at the Cross-Border Major Emergency Planning meeting, January 25th 2017.
4.3.4. Links with Researchers and Other Projects
The strong networks of the project consortium was exploited throughout the project. In addition to conference and journal publications, SLANDAIL organised conferences, symposia and workshops. Such events provided excellent opportunities to discuss topics of shared interest with our peers from other EU projects and from the wider scientific, End User and legal communities. Events for which SLANDAIL had full or shared responsibility, or made several presentations, include the following:
▪ The MUININ workshop on technology and emergency management. This was part of the ARES International Conference on Availability, Reliability and Security, held in Salzburg, Austria, in August 2016.
▪ The EMOT workshop (Emotions, Metaphors, Ontology and Terminology), explored aspects of trust building and effective communication in emergencies, particularly the use of social media. This workshop (in May, 2016) was part of the Language Resources and Evaluation Conference (LREC), at Portorož, Slovenia.
Figure 31: SLANDAIL researchers in terminology, ontology and disaster communication (UNIPD) at the LREC 2016 Conference at Portorož, Slovenia. L-R: Virginia Zorzi, Raffaella Panizzon, Alice Cravotta, Maria Teresa Musacchio, Maria Grazia Busá.
▪ Preparing for floods was the focus of an industry-facing event held at TCD in May 2016, in collaboration with the Business Continuity Institute and the Emergency Planning Society. Speakers included SLANDAIL research staff and Irish emergency planning agencies, and attendees represented public bodies, industry representatives, emergency response agencies, insurance companies, banks and technologists
▪ SLANDAIL’s outreach to other EU projects involved a contribution to the Innovation for Crisis Management (I4CM DRIVER) conference in Berlin, in December 2015. We shared experience, best practice and insights with peer projects DRIVER, SecurePART, TACTIC, SecinCoRe, EmerGent, EPISECC, SALUS and FORTRESS.
▪ SLANDAIL hosted the DISONTOLOGY workshop at the 20th European Symposium on Languages for Special Purposes (LSP, Vienna, July 2015). This highlighted the importance of clarity and effective communication in an emergency context, particularly in multi-ethnic and multilingual environments (such as many major cities), and explored the challenges inherent in multiple different specialist terminologies in emergencies.
The project has a high profile across the security and emergency management research domains. We will leverage this profile when building new research consortia to address emerging challenges and to encourage researchers from across Europe to engage with us, use our tools and resources, and contribute to shared scientific progress and social benefit.
4.4. Exploitation & Future Planning
SLANDAIL has developed an IP Protection Plan that shows how each beneficiary will own their own IP as well as how the overall project IP can be commercialised by beneficiaries. These technologies are primarily the work of the academic partners (TCD, INFAI, UNIPD, ULSTER), but have benefited from the inputs of the commercial partners (CID, DataPiano, Stillwater) and the End User representatives (GARDA, PSNI, BLS). The intellectual property rights are primarily vested in the relevant academic institution(s), but the wider team is committed to a policy of facilitating reasonable and fair use of IP based on fair commercial agreement.
• TCD holds the intellectual property rights to the social media monitor
• TCD holds the intellectual property rights to the Intrusion Index
• INFAI holds the intellectual property rights to the topic modelling system
• UNIPD holds the intellectual property rights to the multilingual terminology wiki
• ULSTER holds the intellectual property rights to the “Squiral” image processing technology and their image processing system
Ownership reflects the Consortium Agreement, as well as the reality that SLANDAIL builds on many years of partner R&D in these and related fields, so that the background IP contributed by each partner is tightly coupled to the new foreground IP.
Each of these potential assets runs autonomously, either online or on a local PC: they can be “mixed and matched” as required for any given application. The development of these tools helps existing systems go beyond the state-of-the-art. Each has been developed as a “standalone” system, but their outputs can feed into SIGE or Topic Analyst or into other third party systems as required. This increases their potential for exploitation, in both commercial and scientific settings. The “products” that the academic partners offer can thus be software libraries, or access to web-based services.
4.4.1. Commercialisation
SLANDAIL partners have explored the potential for commercialisation of research developed during the project. Strong commercial partnerships have been formed for future work beyond the scope of the three-year run-time, including a partnership between the coupled technologies of SMEs DataPiano and CID.
Further relationships between academic partners and SME partners have been formed. INFAI have a long-standing relationship with CID, and continue to provide methods and tools for research. ULSTER and TCD have integrated text an image analytic tools with DataPiano’s SIGE system: these integrated tools can work as standalone platforms to be offered as part of a sales package to emergency management teams.
Figure 32: The Fifth SLANDAIL Plenary Meeting (at which all eleven project beneficiaries were represented) was held in October 2016 in Berlin.
TCD have explored the opportunities available through Irish funding bodies to create a company from the research efforts created in text analytics and event management through social media. Meetings have been held with government agencies (European Union funding agencies, Enterprise Ireland, Department of Foreign Affairs), university funding opportunities (KIC funding, Atlantic Bridge) and venture capital (Frontline) to explore the potential for commercial outputs from the research developed. Negotiations with two of these bodies are in an advanced state, and the potential to start a company that brings some of the research outputs forward will be realised later in 2017.
4.4.2. Exploitation Workshop
A Commercialisation and Exploitation Workshop in May 2016 by TCD led to initial plans on exploitation of the SLANDAIL System. This one-day event involved the three commercial partners (CID, DataPiano and Stillwater), as well as commercialisation and spin-out experts from the Irish national agency responsible for the development and growth of Irish Enterprises in world markets (Enterprise Ireland), and the coordinator at TCD. At this workshop the SLANDAIL system and advanced components were demonstrated to Enterprise Ireland (EI), the Irish government organisation responsible for the development and growth of Irish enterprises in world markets. EI gave feedback on potential target markets for SLANDAIL software, and showed interest in the possibility of TCD software being engaged in spin-off opportunities through the university.
A detailed analysis of the conversations and conclusions from the workshop was presented in D6.3. In summary, the key points are as follows:
▪ The most important commercial output of SLANDAIL is the coupled emergency management and social media monitoring system (D4.1). This combines two commercial product lines – DataPiano’s SIGE EMS and CID’s Topic Analyst social media analysis platform. Both individual systems have also benefited significantly from the project, with a shift to cloud-based architectures, greater architectural flexibility and modularity, and (in the case of SIGE) a new user interface paradigm.
▪ New capabilities can be added to the integrated system, based on the technology modules developed by academic partners. These include image social and legacy media monitors (from TCD), topic models (from INFAI), Multilingual terminology support (from UNIPID), image processing (from ULSTER), and intrusion index calculation (from TCD). Such modules can be added on a case by case basis, to meet the needs of specific customers.
▪ License fees will be payable to the relevant academic partners by the commercial partners, where the academic modules are added.
▪ There are promising markets in both the private and public sectors for the integrated system, with and without additional modules.
▪ A spin-off company opportunity is being explored by TCD to add another avenue of exploitation to its media monitoring and intrusion-index technology.
▪ Professional services and training represent a further potential revenue stream, and one that might support further development effort.
Figure 34: SLANDAIL beneficiaries with members of Enterprise Ireland at the SLANDAIL Exploitation Workshop, 25th May 2016, Lloyd Institute Dublin.
4.4.3. SLANDAIL Virtual Research Community
At the close of the project, SLANDAIL presented a Virtual Research Community Roadmap and Joint Research Action Plan (D6.4] a document that explores the legacy of the SLANDAIL project and considers how the multi-disciplinary relationships and perspectives of the consortium are preserved and new collaborations facilitated.
This Roadmap was based on the concept of a “virtual research community” – a collection of organisations with a common vision of pursuing multi-disciplinary research in ethics, technology, law, media and public safety. The community includes all members of the SLANDAIL consortium, but can be expanded to add new members with new skills and resources as required. Members sign up to a “Memorandum of Understanding” that states their willingness to work together and to contribute key resources if a suitable opportunity arises.
We developed this collective vision by looking at the future research domains that each partner is particularly interested in developing. For all partners, SLANDAIL is a stepping-stone in longer-term research, product development and service agendas. By identifying commonalities and complementarities in these areas (some, of course, closely aligned with our work in SLANDAIL), we derived a set of common research areas in which we see opportunities for further collaboration – the Joint Research Action Plan. This ensures that the collaborative relationships established during the project endure and serve as a foundation for further research and development.
List of Websites:
The SLANDAIL website is at http://SLANDAIL.eu/
The Coordinator of the Project is Prof. Khurshid Ahmad (The Provost, Fellows, Foundation Scholars & the Other Members of Board of The College of The Holy & Undivided Trinity of Queen Elizabeth Near Dublin): kahmad@cs.tcd.ie
