Final Report Summary - RTD (Real-Time Delivery for Sensor Networks in Unpredictable Environments)
Over the past few years, advances in wireless networking, miniaturisation and embedded microprocessors, have revolutionised the way we live, work and interact with the physical environment. Today, tiny, inexpensive, battery-operated sensor devices with integrated sensing, processing and communication capabilities make it feasible to offer novel services, such as monitor vehicular traffic or report habitat conditions, using inexpensive ubiquitous networks of embedded devices.
The ability to interact with the physical world in a non-intrusive manner opens up a huge range of possible economical and societal benefits to several disciplines. For example, sensor networks can be employed for environmental and habitat monitoring for scientific research, healthcare monitoring for improving the quality of life (e. g., at-home monitoring of elder's physical and social activity), active monitoring of building systems for reducing environmental impact, surveillance for emergency response for critical infrastructure protection, and transportation monitoring for control and safety.
The focus of this project was the development of a framework and the appropriate mechanisms to provide end-to-end real-time, reliable delivery of data and events in distributed event-based sensor systems. This is a fundamental problem due to the explosion in the availability and use of such sensor network infrastructures and the need to operate efficiently in highly unpredictable conditions and be able to report the occurrence of events in real-time and reliably. Previous approaches to deal with this problem are inadequate because they result in under-utilisation of the system and significant waste of resources, or are reactive in nature. Furthermore, mobility brings a new set of challenges in that, real-time data delivery must be achieved in more demanding environments and to nodes (human observers) whose locations change in time. It is therefore crucial to build systems that are able to deliver time-critical events in real-time, despite possible failures, given the computational, communication and energy limitations of the sensor network, and in dynamic environments due to node mobility.
In particular, our work contributed to the following research problems:
1) Congestion control
Developed congestion control techniques to address the problem of burstiness in data streams in distributed stream processing systems with applications to sensor networks (this work has received a Best Paper Award at IPDPS'09). We have developed the methodology, studied and understood critical thresholds, developed theoretical modeling to understand how the traffic intensity changes with various parameters including application data rates, network topology and density and interference patterns among routing paths. Our approach uses simple, decentralised and low-overhead techniques.
2) Synergistic Distributed Real-Time Scheduling and Traffic Regulation
Designed and developed synergistic distributed real-time scheduling and traffic regulation techniques. Our main idea is to dynamically control the execution of the applications and delay the processing of bursty data streams when resource capacities are exceeded. We have implemented the distributed scheduling and traffic regulation mechanisms into our prototype.
3) Real-Time Querying of historical data
We addressed the problem of real-time querying of historical data on the sensor devices. We developed a suite of index structures and algorithms on flash-equipped sensor devices, which permit us to efficiently query historical data in real-time. Our techniques are designed to consider the unique read, write and wear constraints of the flash memories and the very limited data memory on the sensor devices; this allows us to efficiently retrieve historical data in real-time, which has been considered in the past to be prohibitively expensive.
4) Failures
Studied the problem of enabling uninterrupted delivery of messages between endpoints, subject to spatially correlated failures in addition to independent failures. We developed a failure model-independent algorithm for computing routing paths based on failure correlations using both a-priory failure statistics together with available real-time monitoring information. We also designed and implemented an overlay routing service that allows applications to construct and maintain highly resilient end-to-end paths.
4) Real-time Scheduling under Mobility
We studied the problem of distributed real-time scheduling under mobility. More specifically, we presented a two level scheduling scheme that effectively predicts application execution times and dynamically schedules application tasks to meet end-to-end real-time response requirements. We also studied the problem of congestion avoidance via traffic management in sensor networks with a mobile sink. More specifically, we have developed a low cost, low complexity routing scheme based on on-line measurements of the Received Signal Strength Indicator (RSSI), which effectively considers the paths'dynamic reliability variations during sink mobility and rate regulation approaches that effectively compensate lower quality flows for lost packets by receiving higher rate.
5) Event Sampling Mechanisms
We also developed event sampling techniques to choose a representative subset from the produced data streams that preserves the data characteristics, while respecting resource costs and application constraints. These allow us to improve the quality of the systems with user feedback and achieve rapid detection of events.
6) System Development and Software Components
Another important challenge we addressed is the development of the distributed sensor system and the appropriate software modules that work hand-in-hand with lower layer functionalities (i. e., scheduling at the OS level and communication protocols), and provides the appropriate programming primitives at the higher layer for the development of distributed applications in the system. Part of this work was performed with the colleagues from Nokia Research (this work has been featured by the MIT Technology Review, ReadWriteWeb and other forums as a prominent technology). We have developed and demoed Misco, a mobile MapReduce framework targeted for any cell phone, mobile device with Python capabilities, to support applications with real-time response requirements while hiding many of the complexities of the underlying system, namely, intermediate network connectivity, mobility, limited resource availability. The goal of this work, is to provide a usable and practical platform for developers of distributed mobile sensor-based applications, to support real-time demands, while hiding the low-level concerns of scheduling and fault-tolerance from the application developer.
The project has resulted in several scientific journal and conference publications. All the materials of the project are available through the website of the project. The project website is available at: http://www.cs. aueb. gr/~vana/rtd
The ability to interact with the physical world in a non-intrusive manner opens up a huge range of possible economical and societal benefits to several disciplines. For example, sensor networks can be employed for environmental and habitat monitoring for scientific research, healthcare monitoring for improving the quality of life (e. g., at-home monitoring of elder's physical and social activity), active monitoring of building systems for reducing environmental impact, surveillance for emergency response for critical infrastructure protection, and transportation monitoring for control and safety.
The focus of this project was the development of a framework and the appropriate mechanisms to provide end-to-end real-time, reliable delivery of data and events in distributed event-based sensor systems. This is a fundamental problem due to the explosion in the availability and use of such sensor network infrastructures and the need to operate efficiently in highly unpredictable conditions and be able to report the occurrence of events in real-time and reliably. Previous approaches to deal with this problem are inadequate because they result in under-utilisation of the system and significant waste of resources, or are reactive in nature. Furthermore, mobility brings a new set of challenges in that, real-time data delivery must be achieved in more demanding environments and to nodes (human observers) whose locations change in time. It is therefore crucial to build systems that are able to deliver time-critical events in real-time, despite possible failures, given the computational, communication and energy limitations of the sensor network, and in dynamic environments due to node mobility.
In particular, our work contributed to the following research problems:
1) Congestion control
Developed congestion control techniques to address the problem of burstiness in data streams in distributed stream processing systems with applications to sensor networks (this work has received a Best Paper Award at IPDPS'09). We have developed the methodology, studied and understood critical thresholds, developed theoretical modeling to understand how the traffic intensity changes with various parameters including application data rates, network topology and density and interference patterns among routing paths. Our approach uses simple, decentralised and low-overhead techniques.
2) Synergistic Distributed Real-Time Scheduling and Traffic Regulation
Designed and developed synergistic distributed real-time scheduling and traffic regulation techniques. Our main idea is to dynamically control the execution of the applications and delay the processing of bursty data streams when resource capacities are exceeded. We have implemented the distributed scheduling and traffic regulation mechanisms into our prototype.
3) Real-Time Querying of historical data
We addressed the problem of real-time querying of historical data on the sensor devices. We developed a suite of index structures and algorithms on flash-equipped sensor devices, which permit us to efficiently query historical data in real-time. Our techniques are designed to consider the unique read, write and wear constraints of the flash memories and the very limited data memory on the sensor devices; this allows us to efficiently retrieve historical data in real-time, which has been considered in the past to be prohibitively expensive.
4) Failures
Studied the problem of enabling uninterrupted delivery of messages between endpoints, subject to spatially correlated failures in addition to independent failures. We developed a failure model-independent algorithm for computing routing paths based on failure correlations using both a-priory failure statistics together with available real-time monitoring information. We also designed and implemented an overlay routing service that allows applications to construct and maintain highly resilient end-to-end paths.
4) Real-time Scheduling under Mobility
We studied the problem of distributed real-time scheduling under mobility. More specifically, we presented a two level scheduling scheme that effectively predicts application execution times and dynamically schedules application tasks to meet end-to-end real-time response requirements. We also studied the problem of congestion avoidance via traffic management in sensor networks with a mobile sink. More specifically, we have developed a low cost, low complexity routing scheme based on on-line measurements of the Received Signal Strength Indicator (RSSI), which effectively considers the paths'dynamic reliability variations during sink mobility and rate regulation approaches that effectively compensate lower quality flows for lost packets by receiving higher rate.
5) Event Sampling Mechanisms
We also developed event sampling techniques to choose a representative subset from the produced data streams that preserves the data characteristics, while respecting resource costs and application constraints. These allow us to improve the quality of the systems with user feedback and achieve rapid detection of events.
6) System Development and Software Components
Another important challenge we addressed is the development of the distributed sensor system and the appropriate software modules that work hand-in-hand with lower layer functionalities (i. e., scheduling at the OS level and communication protocols), and provides the appropriate programming primitives at the higher layer for the development of distributed applications in the system. Part of this work was performed with the colleagues from Nokia Research (this work has been featured by the MIT Technology Review, ReadWriteWeb and other forums as a prominent technology). We have developed and demoed Misco, a mobile MapReduce framework targeted for any cell phone, mobile device with Python capabilities, to support applications with real-time response requirements while hiding many of the complexities of the underlying system, namely, intermediate network connectivity, mobility, limited resource availability. The goal of this work, is to provide a usable and practical platform for developers of distributed mobile sensor-based applications, to support real-time demands, while hiding the low-level concerns of scheduling and fault-tolerance from the application developer.
The project has resulted in several scientific journal and conference publications. All the materials of the project are available through the website of the project. The project website is available at: http://www.cs. aueb. gr/~vana/rtd