Final Report Summary - SENSYF (Sentinels Synergy Framework)
Executive Summary:
The SenSyF project aimed at establishing a complete system for fully automated data acquisition and processing, including a specialised Sandbox Service with tools and development/validation platforms where Service developers would be able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase.
Now that it is completed, SenSyF allows the development and testing of new Services and Applications for Sentinel and GMES/Copernicus contributing missions data on a continuous basis. It is also able to assist in the delivery of higher-level products and services complementing the information provided by the (pre-) operational services. SenSyF is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited as has been demonstrated during the project. Using the same model behind the European Space Agency (ESA) Grid Processing on-Demand environment (G-POD), the SenSyF project enables the collaborative sharing of data and processing power from commercial or/and private clouds.
On top of the SenSyF Framework a selected set of demonstrative services have also been included in the project, to demonstrate the system’s potential, provide valuable development feedback for the framework improvement, and prove the overall concept by addressing specific services needs within the European and global settings.
Project Context and Objectives:
The GMES Space Component consists of the five foreseen Sentinel Earth Observation (EO) missions, developed specifically for GMES, as well as several other Contributing Missions - external missions offering part of their capacity to GMES. The first three Sentinel missions, dedicated to Marine and Land services, are planned for launch all starting from 2013. These missions will provide unprecedented amounts of EO data, which will pose substantial challenges on the data processing chains definitions. The diversity of the data available also provides a novel opportunity to extract more detailed and targeted information than available at the present. Within the GMES initiative, the current pre-operational Marine and Land services are implemented in MyOcean and Geoland2 projects, respectively. These services are complemented, by the GMES Initial Operation services in the Land and Emergency Response domains and by the follow up of G-Mosaic for the GMES Security services. The latest two provide rapid and follow-up products tailored for emergency and security applications.
The SenSyF operational objectives are twofold:
• In terms of infrastructure, it shall design, develop and operate an acquisition and processing framework allowing any potential user easy access to all Sentinel datasets, and in which the user can build its own novel Service for exploitation of Synergetic Sentinel data
• In terms of demonstration of the infrastructure capabilities, it shall develop a set of pre-selected Services that address not only the system capabilities but will also provide insightful feedback on it and at the same time assess their coverage of the specific needs identified as drivers for each Service
Earth Observation (EO) satellites generate huge amounts of data that are not easily integrated into processing chains outside the space agencies ground segments, and this is particularly true for the Sentinel missions. Very often, scientists or small and medium enterprises (SME) do not have access to computing and storage capacity to handle the amount of data required to implement, test and deliver end-user services based on EO data.
The SenSyF project consortium proposed a complete system for fully automated data acquisition and processing. The SenSyF project provides a specialized Sandbox Service with tools and development/validation platforms where developers are able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase. This project will allow for the development and testing of new processing chains and methods for Sentinel and GMES contributing mission data on a continuous basis, and the delivery of higher-level products and services complementing the information provided by the (pre-)operational services. This system is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited and demonstrated. The sandbox model furnishes a test environment very similar to the space agencies operational environments, where the applications are ran against large EO series of datasets, and where the “time- to-market” understood as the applications maturity and readiness for production can be streamlined. The distributed processing services are bridging the exploitation gap by offering access to EO data and processing power, and bringing the processors and applications closer to the data. By using the same model behind the European Space Agency (ESA) Grid Processing on-Demand environment, the SenSyF project will enable the collaborative sharing of data and processing power from commercial or/and private clouds. The SenSyF project will provide an infrastructure where SME and scientists can develop and deploy Earth Science application with a lower overall cost of data and infrastructure setup and maintenance.
Built upon the processing infrastructure a complete set of tools for data pre-processing are provided, comprising of the basic operations of data acquisition and retrieval, atomic manipulation tools (re-gridding, image composition, etc.) as well as high level common processing algorithms, such as image classification and segmentation. These common tools provide streamlined access to real and simulated Sentinels data, as well as to external auxiliary and validation data, as well as dissemination mechanisms for the services implemented, such as common set of analysis and visualisation tools for the users to access the data products provided.
The complete system is designed as a fully automatic data processing framework, which will ease the generation and testing of new and innovative applications. The system is oriented to automated processing and exploration of product synergies through rapid prototyping techniques, allowing for the smooth inclusion and testing of new algorithms.
Technically, the objectives for the SenSyF framework can be summarised as:
• Establish acquisition links from SenSyF to Sentinel and other EO data providers to browse and retrieve datasets;
• Establish a massive parallel infrastructure capitalising on G-POD existing infrastructure and experience;
• Provide a set of common tools for pre-processing, handling and combination of data from different EO data sources;
• Establish a generic internal data structure and interfaces for easy plug-in of externally developed modules (i.e. to be developed by users);
• Establish a dissemination channel with visualisation capabilities assess and transfer the results from the Services built on top of the framework.
On top of the Synergy Framework being proposed, a selected set of demonstrative services were selected, which are used to 1) demonstrate the system’s potential, 2) provide valuable development feedback for the framework improvement, and 3) prove the overall concept by addressing specific services needs within the European and global setting. The particular services provided within the current SenSyF project are:
S1 The Monitoring of Continental Fresh Water Service is a service dedicated to monitoring water quality and quantity for in-land water reservoirs;
S2 The Arctic-Alpine Growing Season Mapping Service will provide a service mapping the growing season in arctic and alpine areas of northern Europe;
S3 The Soil Freezing/Thawing Product Service will develop a fully automatic service based on Sentinel-1 and 3 data to classify ground as frozen or thawed;
S4 The Spectro-Temporal Integration Service will provide spectro-temporal integration of reflectance data from multiple different sensors, generating a multi-resolution spectro-temporal product;
S5 The Multitemporal Land Cover Classification and Change Detection Service will generate land cover and land cover change products from cloud-free mosaics;
S6 The Agriculture Support Service is an service dedicated to support farmers and associations in managing the agriculture irrigation process;
S7 The Landscape Management Service is devoted to the inclusion of remotely sensed data within Management Decision Support Systems, which would also include modelled outputs and ground-based information.
A common technical objective for each of the Services is that the development and operations cost have to be significantly lower than if the Service was developed on its own.
Project Results:
The project started with the definition of the basis for the SenSyF system, together with the experimental services that are proposed for the concept validation.
In the initial stages of the project, each of the seven services being provided with SenSyF were consulted, and a set of baseline user requirements derived from the users responses, and further consolidated into the SenSyF system requirements. Throughout this process of definition of requirements, the full SenSyF consortium was engaged in offline clarifications sessions, and active interchanges of knowledge, mostly between the services and the framework teams.
After its users requirements specification, SenSyF advanced towards the specification of framework and services. For the SenSyF framework, composed of the Cloud-based infrastructure and a set of tailored tools built on tool of it, and services, a complete set of specifications, involving algorithms and system definition, as well as data products, was produced. These specifications were then used as the foundation of the full SenSyF system, whose first version was produced after one year.
In parallel with the development of the SenSyF requirements and specifications, a set of activities was initiated towards the definition and procurement of test datasets for system validation. In SenSyF, a unified approach was pursued, to minimise the data dependency of the project in the early stages, when no Sentinels data is yet available. Furthermore, by restricting the number of test regions, data sharing and eventual collaborations between the internal SenSyF teams was fostered, where one service's output can be tested as the next service's input.
Once the design presented at CDR milestone was frozen, the implementation proceeded in two different fronts: on one side the development required for the SenSyF infrastructure and SenSyF Development Kit (SDK), and on the other side the development required for each of the individual Services. Starting from the harmonisation process described earlier, SDK developed the underlying tools for the Services to make use in their integration path (such as image cropping, tiling, orthorrectification...). The services developed (or migrated) their applications to be compliant with the interfaces of SenSyF, gaining easier access to data and processing resources in the deployment phase.
The development path was followed without major events, although the main problem faced was to convey the required paradigm change needed to deploy applications on the cloud to the Service developers. This led to some difficulties in the earlier stages of the development task, but was quickly resolved with dedicated training actions and the setup of a Help Desk to attend Service queries when dealing with integration and deployment issues.
After Service integration, several demonstration activities were carried out on each of the services, in order to demonstrate to the Service providers how the Cloud Bursting concept works and what are the costs associated with it. The results were presented at SAR meeting and the conclusions showed the great interest of Service providers in having access to infrastructure and tools as the ones provided by SenSyF.
The demonstration results were captured and used to drive a second round of user and service requirements specification, capitalising on the experience gained with the first version of system. A shorter cycle of design followed, and the implementation proceeded again in two different fronts: on one side the development required for the SenSyF infrastructure and SenSyF Development Kit (SDK), and on the other side the development required for each of the individual Services. Additional updates to the underlying tools for the Services were developed to make use in their integration path (such as improved tiling..) as well as to the User Front End, since it was identified as a weak point in the first version of the framework. The services (which already had their applications compliant with the interfaces of SenSyF), developed different algorithm evolutions that were identified as useful for V2.
At the end of the project, additional demonstration activities were carried out on each of the services, in order to demonstrate to the Service providers the improvements obtained in this second version of the system. The results were presented publicly at the Open Science 2.0 event at ESA-ESRIN.
Since the beginning of 2013 a dedicated SenSyF portal is available at sensyf.eu containing a project and team overview, and where updates can be consulted. Interested parties can also register to receive our newsletters on the site, and consult the project brochures and other promotion material. The SenSyF project and it's services' results have also been presented in a number of international conferences during the project lifetime.
Potential Impact:
The SenSyF project has successfully implemented a complete system for fully automated data acquisition and processing. It provides a specialized Sandbox Service with tools and development/validation platforms where developers are able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase. This project allows the development and testing of new processing chains and methods for Sentinel and Copernicus contributing mission data on a continuous basis, and the delivery of higher-level products and services complementing the information provided by the (pre-)operational services. This system is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited and demonstrated.
The SenSyF operational objectives were 1) in terms of infrastructure, it designed, developed and operated an acquisition and processing framework allowing any potential user easy access to all Sentinel datasets, and in which the user was able to build its own novel Service for exploitation of Synergetic Sentinel data, and 2) in terms of demonstration of the infrastructure capabilities, it developed a set of pre-selected Services that addressed not only the system capabilities but also provided insightful feedback on it and at the same time assessed their coverage of the specific needs identified as drivers for each Service.
At the end of the three years of the project, SenSyF has been able to:
Establish acquisition links from SenSyF to Sentinel and other EO data providers to browse and retrieve datasets;
Establish a massive parallel infrastructure capitalising on G-POD existing infrastructure and experience;
Provide a set of common tools for pre-processing, handling and combination of data from different EO data sources;
Establish a generic internal data structure and interfaces for easy plug-in of externally developed modules (i.e. to be developed by users);
Establish a dissemination channel with visualisation capabilities assess and transfer the results from the Services built on top of the framework.
In broad terms, the successful completion of SenSyF has demonstrated the validity of concept of Cloud-based framework for the processing of large amounts of data towards the operationalisation of services strongly depending of Earth Observation data, in particular of Copernicus' Sentinels data. Also on the technical side the tested solutions have paved the way to further optimisation and further system evolutions. Toward this end, a strong component of technical as well as business assessment effort was made towards the final stages of the projects, where both internal users and external potential users were called in to contribute with their views and proposed sustainable approaches.
List of Websites:
www.sensyf.eu
www.eo-sensyf.com
The SenSyF project aimed at establishing a complete system for fully automated data acquisition and processing, including a specialised Sandbox Service with tools and development/validation platforms where Service developers would be able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase.
Now that it is completed, SenSyF allows the development and testing of new Services and Applications for Sentinel and GMES/Copernicus contributing missions data on a continuous basis. It is also able to assist in the delivery of higher-level products and services complementing the information provided by the (pre-) operational services. SenSyF is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited as has been demonstrated during the project. Using the same model behind the European Space Agency (ESA) Grid Processing on-Demand environment (G-POD), the SenSyF project enables the collaborative sharing of data and processing power from commercial or/and private clouds.
On top of the SenSyF Framework a selected set of demonstrative services have also been included in the project, to demonstrate the system’s potential, provide valuable development feedback for the framework improvement, and prove the overall concept by addressing specific services needs within the European and global settings.
Project Context and Objectives:
The GMES Space Component consists of the five foreseen Sentinel Earth Observation (EO) missions, developed specifically for GMES, as well as several other Contributing Missions - external missions offering part of their capacity to GMES. The first three Sentinel missions, dedicated to Marine and Land services, are planned for launch all starting from 2013. These missions will provide unprecedented amounts of EO data, which will pose substantial challenges on the data processing chains definitions. The diversity of the data available also provides a novel opportunity to extract more detailed and targeted information than available at the present. Within the GMES initiative, the current pre-operational Marine and Land services are implemented in MyOcean and Geoland2 projects, respectively. These services are complemented, by the GMES Initial Operation services in the Land and Emergency Response domains and by the follow up of G-Mosaic for the GMES Security services. The latest two provide rapid and follow-up products tailored for emergency and security applications.
The SenSyF operational objectives are twofold:
• In terms of infrastructure, it shall design, develop and operate an acquisition and processing framework allowing any potential user easy access to all Sentinel datasets, and in which the user can build its own novel Service for exploitation of Synergetic Sentinel data
• In terms of demonstration of the infrastructure capabilities, it shall develop a set of pre-selected Services that address not only the system capabilities but will also provide insightful feedback on it and at the same time assess their coverage of the specific needs identified as drivers for each Service
Earth Observation (EO) satellites generate huge amounts of data that are not easily integrated into processing chains outside the space agencies ground segments, and this is particularly true for the Sentinel missions. Very often, scientists or small and medium enterprises (SME) do not have access to computing and storage capacity to handle the amount of data required to implement, test and deliver end-user services based on EO data.
The SenSyF project consortium proposed a complete system for fully automated data acquisition and processing. The SenSyF project provides a specialized Sandbox Service with tools and development/validation platforms where developers are able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase. This project will allow for the development and testing of new processing chains and methods for Sentinel and GMES contributing mission data on a continuous basis, and the delivery of higher-level products and services complementing the information provided by the (pre-)operational services. This system is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited and demonstrated. The sandbox model furnishes a test environment very similar to the space agencies operational environments, where the applications are ran against large EO series of datasets, and where the “time- to-market” understood as the applications maturity and readiness for production can be streamlined. The distributed processing services are bridging the exploitation gap by offering access to EO data and processing power, and bringing the processors and applications closer to the data. By using the same model behind the European Space Agency (ESA) Grid Processing on-Demand environment, the SenSyF project will enable the collaborative sharing of data and processing power from commercial or/and private clouds. The SenSyF project will provide an infrastructure where SME and scientists can develop and deploy Earth Science application with a lower overall cost of data and infrastructure setup and maintenance.
Built upon the processing infrastructure a complete set of tools for data pre-processing are provided, comprising of the basic operations of data acquisition and retrieval, atomic manipulation tools (re-gridding, image composition, etc.) as well as high level common processing algorithms, such as image classification and segmentation. These common tools provide streamlined access to real and simulated Sentinels data, as well as to external auxiliary and validation data, as well as dissemination mechanisms for the services implemented, such as common set of analysis and visualisation tools for the users to access the data products provided.
The complete system is designed as a fully automatic data processing framework, which will ease the generation and testing of new and innovative applications. The system is oriented to automated processing and exploration of product synergies through rapid prototyping techniques, allowing for the smooth inclusion and testing of new algorithms.
Technically, the objectives for the SenSyF framework can be summarised as:
• Establish acquisition links from SenSyF to Sentinel and other EO data providers to browse and retrieve datasets;
• Establish a massive parallel infrastructure capitalising on G-POD existing infrastructure and experience;
• Provide a set of common tools for pre-processing, handling and combination of data from different EO data sources;
• Establish a generic internal data structure and interfaces for easy plug-in of externally developed modules (i.e. to be developed by users);
• Establish a dissemination channel with visualisation capabilities assess and transfer the results from the Services built on top of the framework.
On top of the Synergy Framework being proposed, a selected set of demonstrative services were selected, which are used to 1) demonstrate the system’s potential, 2) provide valuable development feedback for the framework improvement, and 3) prove the overall concept by addressing specific services needs within the European and global setting. The particular services provided within the current SenSyF project are:
S1 The Monitoring of Continental Fresh Water Service is a service dedicated to monitoring water quality and quantity for in-land water reservoirs;
S2 The Arctic-Alpine Growing Season Mapping Service will provide a service mapping the growing season in arctic and alpine areas of northern Europe;
S3 The Soil Freezing/Thawing Product Service will develop a fully automatic service based on Sentinel-1 and 3 data to classify ground as frozen or thawed;
S4 The Spectro-Temporal Integration Service will provide spectro-temporal integration of reflectance data from multiple different sensors, generating a multi-resolution spectro-temporal product;
S5 The Multitemporal Land Cover Classification and Change Detection Service will generate land cover and land cover change products from cloud-free mosaics;
S6 The Agriculture Support Service is an service dedicated to support farmers and associations in managing the agriculture irrigation process;
S7 The Landscape Management Service is devoted to the inclusion of remotely sensed data within Management Decision Support Systems, which would also include modelled outputs and ground-based information.
A common technical objective for each of the Services is that the development and operations cost have to be significantly lower than if the Service was developed on its own.
Project Results:
The project started with the definition of the basis for the SenSyF system, together with the experimental services that are proposed for the concept validation.
In the initial stages of the project, each of the seven services being provided with SenSyF were consulted, and a set of baseline user requirements derived from the users responses, and further consolidated into the SenSyF system requirements. Throughout this process of definition of requirements, the full SenSyF consortium was engaged in offline clarifications sessions, and active interchanges of knowledge, mostly between the services and the framework teams.
After its users requirements specification, SenSyF advanced towards the specification of framework and services. For the SenSyF framework, composed of the Cloud-based infrastructure and a set of tailored tools built on tool of it, and services, a complete set of specifications, involving algorithms and system definition, as well as data products, was produced. These specifications were then used as the foundation of the full SenSyF system, whose first version was produced after one year.
In parallel with the development of the SenSyF requirements and specifications, a set of activities was initiated towards the definition and procurement of test datasets for system validation. In SenSyF, a unified approach was pursued, to minimise the data dependency of the project in the early stages, when no Sentinels data is yet available. Furthermore, by restricting the number of test regions, data sharing and eventual collaborations between the internal SenSyF teams was fostered, where one service's output can be tested as the next service's input.
Once the design presented at CDR milestone was frozen, the implementation proceeded in two different fronts: on one side the development required for the SenSyF infrastructure and SenSyF Development Kit (SDK), and on the other side the development required for each of the individual Services. Starting from the harmonisation process described earlier, SDK developed the underlying tools for the Services to make use in their integration path (such as image cropping, tiling, orthorrectification...). The services developed (or migrated) their applications to be compliant with the interfaces of SenSyF, gaining easier access to data and processing resources in the deployment phase.
The development path was followed without major events, although the main problem faced was to convey the required paradigm change needed to deploy applications on the cloud to the Service developers. This led to some difficulties in the earlier stages of the development task, but was quickly resolved with dedicated training actions and the setup of a Help Desk to attend Service queries when dealing with integration and deployment issues.
After Service integration, several demonstration activities were carried out on each of the services, in order to demonstrate to the Service providers how the Cloud Bursting concept works and what are the costs associated with it. The results were presented at SAR meeting and the conclusions showed the great interest of Service providers in having access to infrastructure and tools as the ones provided by SenSyF.
The demonstration results were captured and used to drive a second round of user and service requirements specification, capitalising on the experience gained with the first version of system. A shorter cycle of design followed, and the implementation proceeded again in two different fronts: on one side the development required for the SenSyF infrastructure and SenSyF Development Kit (SDK), and on the other side the development required for each of the individual Services. Additional updates to the underlying tools for the Services were developed to make use in their integration path (such as improved tiling..) as well as to the User Front End, since it was identified as a weak point in the first version of the framework. The services (which already had their applications compliant with the interfaces of SenSyF), developed different algorithm evolutions that were identified as useful for V2.
At the end of the project, additional demonstration activities were carried out on each of the services, in order to demonstrate to the Service providers the improvements obtained in this second version of the system. The results were presented publicly at the Open Science 2.0 event at ESA-ESRIN.
Since the beginning of 2013 a dedicated SenSyF portal is available at sensyf.eu containing a project and team overview, and where updates can be consulted. Interested parties can also register to receive our newsletters on the site, and consult the project brochures and other promotion material. The SenSyF project and it's services' results have also been presented in a number of international conferences during the project lifetime.
Potential Impact:
The SenSyF project has successfully implemented a complete system for fully automated data acquisition and processing. It provides a specialized Sandbox Service with tools and development/validation platforms where developers are able to implement and test their applications, and then tap into a distributed pool of cloud resources when ready for the exploitation phase. This project allows the development and testing of new processing chains and methods for Sentinel and Copernicus contributing mission data on a continuous basis, and the delivery of higher-level products and services complementing the information provided by the (pre-)operational services. This system is based on a dynamic parallel processing infrastructure, where the capabilities of grid computing applied to Sentinel data processing can be exploited and demonstrated.
The SenSyF operational objectives were 1) in terms of infrastructure, it designed, developed and operated an acquisition and processing framework allowing any potential user easy access to all Sentinel datasets, and in which the user was able to build its own novel Service for exploitation of Synergetic Sentinel data, and 2) in terms of demonstration of the infrastructure capabilities, it developed a set of pre-selected Services that addressed not only the system capabilities but also provided insightful feedback on it and at the same time assessed their coverage of the specific needs identified as drivers for each Service.
At the end of the three years of the project, SenSyF has been able to:
Establish acquisition links from SenSyF to Sentinel and other EO data providers to browse and retrieve datasets;
Establish a massive parallel infrastructure capitalising on G-POD existing infrastructure and experience;
Provide a set of common tools for pre-processing, handling and combination of data from different EO data sources;
Establish a generic internal data structure and interfaces for easy plug-in of externally developed modules (i.e. to be developed by users);
Establish a dissemination channel with visualisation capabilities assess and transfer the results from the Services built on top of the framework.
In broad terms, the successful completion of SenSyF has demonstrated the validity of concept of Cloud-based framework for the processing of large amounts of data towards the operationalisation of services strongly depending of Earth Observation data, in particular of Copernicus' Sentinels data. Also on the technical side the tested solutions have paved the way to further optimisation and further system evolutions. Toward this end, a strong component of technical as well as business assessment effort was made towards the final stages of the projects, where both internal users and external potential users were called in to contribute with their views and proposed sustainable approaches.
List of Websites:
www.sensyf.eu
www.eo-sensyf.com