Final Report Summary - NGHCS (NGHCS: Creating the Next-Generation Mobile Human-Centered Systems)
The project addresses fundamental distributed systems research problems and encompasses both theoretical and experimental tasks to enable the development of reliable, predictable and adaptable distributed real-time “human-centered” computational systems. The ambition of the project is to develop a comprehensive framework to simplify the development of such systems and improve their efficiency and automation, as well as make them predictable and reliable.
The realization of the vision presents numerous unsolved and exciting problems to the software research community, including operating under uncertainty, developing the foundations for meeting end-to-end timeliness and reliability demands of the applications, be adaptable, and adopt a human-centered design, where participatory sensing and crowdsourcing techniques are integrated into the design of the applications.
The outcome of the proposed work is expected to have significant impact on a wide variety of distributed systems applications in domains as diverse as transportation systems and emergency response.
We summarize the outcomes of our project below:
(i) Up to now we have investigated the use of mathematical models of varying complexity to address the impact of various factors that these systems face, including dynamic workloads, unpredictable occurrence of events, mobility, real-time demands of applications such as em-ergency situations, changes in the population growth and urban dynamics.
(ii) We have developed a set of practical and decentralized solutions, where we have integrated our theoretical models based on uncertainty, to effectively deal with the volume, scale and complexity of the problems involved.
(iii) We have developed a library of techniques that have been integrated into state-of-the-art systems, in particular the Apache Kafka pub/sub system, the Apache Hadoop MapReduce system, the Apache Storm streaming system and the Esper Complex Event Processing Engine. These provide with functionality to achieve scalable and low latency data processing by efficiently distributing the computation between multiple concurrently running components. Most of the distributed stream processing systems lack the expressiveness and ease of use of CEP systems like Esper, which provides an SQL-like language, EPL (Event Processing Language) for expressing the rules. Our library of mechanisms will be soon be publicly available from our webpage http://www.cs.aueb.gr/~vana/NGHCS
(iv) We have validated our current techniques in the real-world, using data from the transportation domain. The case of study is the real-time detection of traffic issues using streaming data that originate from public buses and are available at http://dublinked.com/datastore/datasets/dataset-304.php
(v) We have demonstrated our innovation potential by developing the CrowdAlert app (http://crowdalert.aueb.gr also available at Google Play) designed to enable users to receive traffic information and unusual events of interest, by exploiting real-time data generated from open data (in particular, transportation data generated from road sensors and bus sensors) and the human crowd. The CrowdAlert system demonstrates the potential of our techniques and will be the vehicle for our human-centered techniques. What distinguishes our approach is that we adopt a truly human-centered design, by setting up a constant, real-time feedback loop for evaluating the effectiveness of our techniques, their accuracy and performance. In the design and development of our app have adopted a privacy-by-design approach where privacy is taken into consideration throughout the entire process. CrowdAlert was promoted by the Dublin City Council (DCC), a local authority which is responsible for managing the traffic in Dublin. They advertised our app through their website and Live Drive, a radio program which is managed by DCC and provides live traffic updates.
We have also developed SmartStudent/SmartTeacher app that has also been regularly used inside the classroom at Athens University of Economics and Business as a valuable tool to get instant, real-time student feedback on student learning. It allows teachers to issue questions on current material to students and see the results in real-time, and students to receive instant feedback about their understanding of the material. This can provide valuable feedback to the teachers as they can monitor the students’ learning progress by creating lecture questions or quizzes on the fly, pushing the questions to the students, and monitoring and analyzing their replies in order to adjust their teaching based on student learning to maximize student benefit. It has been used by over 3000 undergraduate/graduate students in 8 courses at AUEB.
The realization of the vision presents numerous unsolved and exciting problems to the software research community, including operating under uncertainty, developing the foundations for meeting end-to-end timeliness and reliability demands of the applications, be adaptable, and adopt a human-centered design, where participatory sensing and crowdsourcing techniques are integrated into the design of the applications.
The outcome of the proposed work is expected to have significant impact on a wide variety of distributed systems applications in domains as diverse as transportation systems and emergency response.
We summarize the outcomes of our project below:
(i) Up to now we have investigated the use of mathematical models of varying complexity to address the impact of various factors that these systems face, including dynamic workloads, unpredictable occurrence of events, mobility, real-time demands of applications such as em-ergency situations, changes in the population growth and urban dynamics.
(ii) We have developed a set of practical and decentralized solutions, where we have integrated our theoretical models based on uncertainty, to effectively deal with the volume, scale and complexity of the problems involved.
(iii) We have developed a library of techniques that have been integrated into state-of-the-art systems, in particular the Apache Kafka pub/sub system, the Apache Hadoop MapReduce system, the Apache Storm streaming system and the Esper Complex Event Processing Engine. These provide with functionality to achieve scalable and low latency data processing by efficiently distributing the computation between multiple concurrently running components. Most of the distributed stream processing systems lack the expressiveness and ease of use of CEP systems like Esper, which provides an SQL-like language, EPL (Event Processing Language) for expressing the rules. Our library of mechanisms will be soon be publicly available from our webpage http://www.cs.aueb.gr/~vana/NGHCS
(iv) We have validated our current techniques in the real-world, using data from the transportation domain. The case of study is the real-time detection of traffic issues using streaming data that originate from public buses and are available at http://dublinked.com/datastore/datasets/dataset-304.php
(v) We have demonstrated our innovation potential by developing the CrowdAlert app (http://crowdalert.aueb.gr also available at Google Play) designed to enable users to receive traffic information and unusual events of interest, by exploiting real-time data generated from open data (in particular, transportation data generated from road sensors and bus sensors) and the human crowd. The CrowdAlert system demonstrates the potential of our techniques and will be the vehicle for our human-centered techniques. What distinguishes our approach is that we adopt a truly human-centered design, by setting up a constant, real-time feedback loop for evaluating the effectiveness of our techniques, their accuracy and performance. In the design and development of our app have adopted a privacy-by-design approach where privacy is taken into consideration throughout the entire process. CrowdAlert was promoted by the Dublin City Council (DCC), a local authority which is responsible for managing the traffic in Dublin. They advertised our app through their website and Live Drive, a radio program which is managed by DCC and provides live traffic updates.
We have also developed SmartStudent/SmartTeacher app that has also been regularly used inside the classroom at Athens University of Economics and Business as a valuable tool to get instant, real-time student feedback on student learning. It allows teachers to issue questions on current material to students and see the results in real-time, and students to receive instant feedback about their understanding of the material. This can provide valuable feedback to the teachers as they can monitor the students’ learning progress by creating lecture questions or quizzes on the fly, pushing the questions to the students, and monitoring and analyzing their replies in order to adjust their teaching based on student learning to maximize student benefit. It has been used by over 3000 undergraduate/graduate students in 8 courses at AUEB.