Periodic Reporting for period 2 - IIMEO (Instantaneous Infrastructure Monitoring by Earth Observation)
Período documentado: 2023-12-01 hasta 2024-11-30
Critical infrastructures, e.g. transport, energy and water, are essential to our society. Disruptions can lead to supply bottlenecks, impairments to public safety or other dramatic consequences and should thus be recognised and fixed as soon as possible. Accordingly, critical infrastructure monitoring is of interest to many stakeholders, such as telecommunications and logistics companies, railway operators, energy suppliers, or state institutions such as civil protection agencies.
Current monitoring procedures are not able to monitor infrastructures both continuously and on a large scale. Satellites enable regular, weather-independent monitoring of critical infrastructure from space, however, current space-based systems are not able to provide rapid feedback on the condition of critical infrastructure after extreme weather events or natural disasters due to long response times or insufficient spatial and temporal resolution.
The idea behind IIMEO is to develop a system suitable for such monitoring tasks that can deliver accurate results within short response times and is also economically viable. The planned system follows the NewSpace approach, based on a constellation of small satellites in LEO equipped with a novel sensor combination of Synthetic Aperture Radar (SAR) and cameras as well as a novel on-board processor. The satellite constellation is intended to ensure short revisit times of the area of interest, while the on-board processor takes over part of the processing, significantly reducing the amount of data to be transmitted. This shall result in response times of less than one hour after user request. The current IIMEO project works towards such as system, both by gathering requirements and finding solutions to specific use cases, as well as by developing an airborne prototype implementing much of the functionality, including data acquisition, processing and presentation of results to a pilot user, to determine what a usable infrastructure monitoring system would look like. The prototype will be developed to keep the gap to a space-based prototype small, e.g. wherever possible hard- and software are designed to work on a satellite as well.
Monitoring of railway lines is the pilot use case and is elaborated in cooperation with a railway operator.
The prototype, including on-board processing, will be demonstrated as part of a final flight campaign. A roadmap will describe the further exploitation of results and outline further applications of the technology for other infrastructure systems.
A follow-up mission is envisaged to show monitoring of railways from space on a global scale.
Current monitoring procedures are not able to monitor infrastructures both continuously and on a large scale. Satellites enable regular, weather-independent monitoring of critical infrastructure from space, however, current space-based systems are not able to provide rapid feedback on the condition of critical infrastructure after extreme weather events or natural disasters due to long response times or insufficient spatial and temporal resolution.
The idea behind IIMEO is to develop a system suitable for such monitoring tasks that can deliver accurate results within short response times and is also economically viable. The planned system follows the NewSpace approach, based on a constellation of small satellites in LEO equipped with a novel sensor combination of Synthetic Aperture Radar (SAR) and cameras as well as a novel on-board processor. The satellite constellation is intended to ensure short revisit times of the area of interest, while the on-board processor takes over part of the processing, significantly reducing the amount of data to be transmitted. This shall result in response times of less than one hour after user request. The current IIMEO project works towards such as system, both by gathering requirements and finding solutions to specific use cases, as well as by developing an airborne prototype implementing much of the functionality, including data acquisition, processing and presentation of results to a pilot user, to determine what a usable infrastructure monitoring system would look like. The prototype will be developed to keep the gap to a space-based prototype small, e.g. wherever possible hard- and software are designed to work on a satellite as well.
Monitoring of railway lines is the pilot use case and is elaborated in cooperation with a railway operator.
The prototype, including on-board processing, will be demonstrated as part of a final flight campaign. A roadmap will describe the further exploitation of results and outline further applications of the technology for other infrastructure systems.
A follow-up mission is envisaged to show monitoring of railways from space on a global scale.
At project start, there was a rough idea of how to design an Earth observation system to achieve the above objectives, as well as a general idea that such a system is useful for monitoring railways or geographically similar distributed infrastructures.
We identified further use cases for such an infrastructure earth observation system - road, dyke and power line monitoring - and created specific requirements for our pilot use case, the monitoring of railway tracks, as well as a demonstration scenario to test these requirements. While the final project goal is to setup a satellite constellation, the first step is to develop and integrate our system into an airborne prototype to demonstrate the feasibility of the concept. To keep our developments as compatible as possible for use in a satellite, we have expanded our use case requirements accordingly. This is to ensure that our solution is also suitable for the communication bandwidth and computing capacity of a spacecraft and can withstand the environmental conditions in space.
Using the above, we updated our system design. This is now partitioned into the flight platform and a ground system. The latter has a user interface component as well as a processing platform, which communicates with the interface to accept tasks and present results. The processing platform is set up using standard components -- containers to encapsulate software for processing steps, a message passing middleware to let these components exchange information, and an S3 storage to store bulk data. For both the user interface and the platform infrastructure we developed working prototypes and verified most of them using data from measurement flights. On the non-user end the ground system communicates with our flight platform. This will contain an on-board processing computer, which we specified and made an engineering model of, which we environmentally tested to check that the same computer would also work in space even if it will only be mounted on our demonstrator airplane used in this project. We updated the design of that flight platform to include the sensing equipment (navigation systems, nadir/oblique RGB cameras, SAR), in two pods mounted below each of the planes wings. We designed and assembled the wing pods to include acquisition computers for the sensing equipment and developed synchronization between the SAR or cameras, respectively, and the navigation systems. We also tested the individual hardware components, i.e. prior to integration into the platform, and calibrated sensors in order to ensure useful data is recorded after plane take-off.
One of the main ideas is to use sensors with different properties to observe the infrastructure and to find obstructions on, in our pilot use case, railway tracks. To do so, we developed and implemented processing steps from sensor data acquisition to the output of obstacle detections. For the RGB images, these are the determination of the regions of interest based on map data, geo-referencing of images, image masking and the detection of tracks in those images to find anomalies by comparing detections to railway track maps. For SAR, the steps are image formation in pre-determined regions of interest to produce SAR images corresponding to geo-referenced tiles, as well as the detection of changes of these SAR image tiles against references images. Finally these almost independent processing chains are joined by fusing the detections from the SAR and the RGB chains. For the individual processing steps, as well as for the software infrastructure to run them on the on-board platform, we developed prototypes, which are currently being integrated into the on-board processing system.
We also recorded datasets. This resulted in a preliminary RGB image dataset showing railway tracks with various anomalies captured on a very large and very diverse model railway, RGB images of railway tracks from UAV flights, as well as a small dataset of overlapping SAR and VIS images of railway tracks, some with obstacles placed onto them.
We identified further use cases for such an infrastructure earth observation system - road, dyke and power line monitoring - and created specific requirements for our pilot use case, the monitoring of railway tracks, as well as a demonstration scenario to test these requirements. While the final project goal is to setup a satellite constellation, the first step is to develop and integrate our system into an airborne prototype to demonstrate the feasibility of the concept. To keep our developments as compatible as possible for use in a satellite, we have expanded our use case requirements accordingly. This is to ensure that our solution is also suitable for the communication bandwidth and computing capacity of a spacecraft and can withstand the environmental conditions in space.
Using the above, we updated our system design. This is now partitioned into the flight platform and a ground system. The latter has a user interface component as well as a processing platform, which communicates with the interface to accept tasks and present results. The processing platform is set up using standard components -- containers to encapsulate software for processing steps, a message passing middleware to let these components exchange information, and an S3 storage to store bulk data. For both the user interface and the platform infrastructure we developed working prototypes and verified most of them using data from measurement flights. On the non-user end the ground system communicates with our flight platform. This will contain an on-board processing computer, which we specified and made an engineering model of, which we environmentally tested to check that the same computer would also work in space even if it will only be mounted on our demonstrator airplane used in this project. We updated the design of that flight platform to include the sensing equipment (navigation systems, nadir/oblique RGB cameras, SAR), in two pods mounted below each of the planes wings. We designed and assembled the wing pods to include acquisition computers for the sensing equipment and developed synchronization between the SAR or cameras, respectively, and the navigation systems. We also tested the individual hardware components, i.e. prior to integration into the platform, and calibrated sensors in order to ensure useful data is recorded after plane take-off.
One of the main ideas is to use sensors with different properties to observe the infrastructure and to find obstructions on, in our pilot use case, railway tracks. To do so, we developed and implemented processing steps from sensor data acquisition to the output of obstacle detections. For the RGB images, these are the determination of the regions of interest based on map data, geo-referencing of images, image masking and the detection of tracks in those images to find anomalies by comparing detections to railway track maps. For SAR, the steps are image formation in pre-determined regions of interest to produce SAR images corresponding to geo-referenced tiles, as well as the detection of changes of these SAR image tiles against references images. Finally these almost independent processing chains are joined by fusing the detections from the SAR and the RGB chains. For the individual processing steps, as well as for the software infrastructure to run them on the on-board platform, we developed prototypes, which are currently being integrated into the on-board processing system.
We also recorded datasets. This resulted in a preliminary RGB image dataset showing railway tracks with various anomalies captured on a very large and very diverse model railway, RGB images of railway tracks from UAV flights, as well as a small dataset of overlapping SAR and VIS images of railway tracks, some with obstacles placed onto them.
We expect to deliver a concept for a satellite constellation for high-resolution and near real-time infrastructure monitoring including an innovative sensor concept as well as hard- and software to move processing on-board. With the demonstration of the entire chain from user-requests, acquisition, processing, to presenting the results fused from RGB and SAR to the user, this is beyond the current state of the art. At this stage of the project, we would like to keep individual improvements on the state of the art undisclosed to the public until their scientific publication.