Periodic Reporting for period 2 - openEO (openEO - a common, open source interface between Earth Observation data infrastructures and front-end applications)
Berichtszeitraum: 2018-10-01 bis 2020-11-30
With a new generation of Earth Observation (EO) monitoring satellites, launched within Europe's Copernicus programme, the user community faces significant Big Data challenges. In fact, the capacity of the Copernicus' Sentinel satellites to acquire data outstrips existing capacities to transmit, store, process, and analyse them. As a result, there is an urgent need to replace traditional workflows - which rely on distributing the data to thousands of users over the internet - with cloud computing approaches that bring the users and their software to the data instead. This is one of the central paradigms of the Big Data era. Many European organisations and initiatives have recognised this need at an early stage of the Copernicus programme, and are now working towards the establishment of back-ends capable of storing and processing Petabytes of Sentinel data. Since these endpoints differ regarding e.g. their data storage infrastructure, metadata standards or user interfaces, switching between back-ends is not a trivial task for the user community. The missing standardisation forms a significant entry barrier to cloud EO processing providers for the users.
Therefore, objectives of the openEO project have been to build a common, open source interface for standardising the communication between users and back-ends, and to develop interfaces for various clients and back-ends. openEO APIs were developed to standardise process chains via the programming libraries for Python, R, and JavaScript, by using web editors or via mobile applications. Furthermore, APIs have been developed to translate these standardised commands into the local syntaxes of chosen service providers (back-ends at VITO, EODC, mundialis, Sinergise, EURAC, JRC, CREODIAS, WEkEO DIAS and a proxy to Google Earth Engine). Serving as template for the connection of further back-ends, the existing open source APIs will simplify the use of cloud-based EO processing engines, allow switching between cloud-based back office providers and comparing them, and enable reproducible, open EO science. Thereby, openEO reduces the entry barriers for the adaptation of cloud computing technologies by a broad user community and paves the way for the federation of EO data infrastructure capabilities.
Other objectives have been to implement pre-defined processes for all aspects of the EO data life cycle and to develop use cases as proof-of-concept. 140+ processes have been implemented to find, access, and to process EO data. Additionally, user-defined functions can be processed on selected back-ends, using the openEO API. Five use cases have been developed to prove the usefulness of the developed processes and to validate the openEO interfaces.
Therefore, objectives of the openEO project have been to build a common, open source interface for standardising the communication between users and back-ends, and to develop interfaces for various clients and back-ends. openEO APIs were developed to standardise process chains via the programming libraries for Python, R, and JavaScript, by using web editors or via mobile applications. Furthermore, APIs have been developed to translate these standardised commands into the local syntaxes of chosen service providers (back-ends at VITO, EODC, mundialis, Sinergise, EURAC, JRC, CREODIAS, WEkEO DIAS and a proxy to Google Earth Engine). Serving as template for the connection of further back-ends, the existing open source APIs will simplify the use of cloud-based EO processing engines, allow switching between cloud-based back office providers and comparing them, and enable reproducible, open EO science. Thereby, openEO reduces the entry barriers for the adaptation of cloud computing technologies by a broad user community and paves the way for the federation of EO data infrastructure capabilities.
Other objectives have been to implement pre-defined processes for all aspects of the EO data life cycle and to develop use cases as proof-of-concept. 140+ processes have been implemented to find, access, and to process EO data. Additionally, user-defined functions can be processed on selected back-ends, using the openEO API. Five use cases have been developed to prove the usefulness of the developed processes and to validate the openEO interfaces.
The project's development started with providing a Proof of Concept. This was initiated within two weeks of intense collaboration, defining and implementing micro-services and user-defined functions to create first connections from python commands to endpoint processing. For the Proof of Concept, connections were realised from the Python, R, and JavaScript clients to seven endpoints, allowing the process of micro-services and user-defined functions for three previously defined test cases.
Standards of the endpoints metadata and user interfaces were analysed to allow the further development of openEO benefitting all partners. For involving the opinion of external users in these processes, several ways for communicating with the consortium have been established. A Zenodo community was created to provide all scientific publications and accepted project deliverables to the users. Additionally, during the second review period the consortium actively spread information about the developed standard, participating in or hosting 26 international conferences / workshops / seminars / hackathons or summer schools. Bilateral talks were organised with pilot users / external users or service providers about the use of openEO standard or the implementation of its APIs on additional platforms. The input of these discussions has been integrated into the further development. openEO APIs were further developed for CREODIAS and WEkEO DIAS as proof of concept.
In parallel with constantly upgrading the API in various code versions, five use cases have been developed. This helped with the definition of 140+ pre-defined processes from all aspects of the EO data life cycle, which were developed throughout the project to create a unified openEO syntax. These processes were implemented by the participating back-ends, as possible. Furthermore, the principle of processing user-defined functions has been discussed and implemented on several back-ends. An openEO API v1.0 was published in July 2020, representing the final version of the openEO project, but not the final version of the API. The state of repositories were constantly published in the open source platform GitHub.
Standards of the endpoints metadata and user interfaces were analysed to allow the further development of openEO benefitting all partners. For involving the opinion of external users in these processes, several ways for communicating with the consortium have been established. A Zenodo community was created to provide all scientific publications and accepted project deliverables to the users. Additionally, during the second review period the consortium actively spread information about the developed standard, participating in or hosting 26 international conferences / workshops / seminars / hackathons or summer schools. Bilateral talks were organised with pilot users / external users or service providers about the use of openEO standard or the implementation of its APIs on additional platforms. The input of these discussions has been integrated into the further development. openEO APIs were further developed for CREODIAS and WEkEO DIAS as proof of concept.
In parallel with constantly upgrading the API in various code versions, five use cases have been developed. This helped with the definition of 140+ pre-defined processes from all aspects of the EO data life cycle, which were developed throughout the project to create a unified openEO syntax. These processes were implemented by the participating back-ends, as possible. Furthermore, the principle of processing user-defined functions has been discussed and implemented on several back-ends. An openEO API v1.0 was published in July 2020, representing the final version of the openEO project, but not the final version of the API. The state of repositories were constantly published in the open source platform GitHub.
The implemented interface can be used by diverse clients: i) programming languages (Python, R, JavaScript), ii) graphical user interfaces (web editor, QGIS plugin), and iii) mobile applications (Android, iOS). Back-ends from different service providers can be accessed: VITO, EODC, mundialis, Sinergise, EURAC, JRC, Google Earth Engine, and the project external CREODIAS and WEkEO DIAS. Further discussions with additional external entities are ongoing about realising an openEO implementation on other platforms (e.g. DLR, EGI).
140+ pre-defined functions were developed to be used for the standardised communication. These functions cover following aspects:
- Authentication and user data management
- Query of EO data and processes available at the endpoints
- Band calculations and time series statistics
- Filtering and aggregation of specific dimensions
- UDF execution
- Zonal statistics
- Upload and use of vector data
- EO Data download
User-defined functions can be transferred via openEO to various back-ends for execution, allowing the execution of tailored workflows. Use cases have been implemented for defined pilot users to i) to validate the openEO standard and its potential to allow comparability between service providers, and to ii) be further used by at least the pilot users.
The openEO access to back-end providers represents a value adding services of EO data storage and processing services. It may empower both public and commercial users to use a standardised interface to realise EO data processing without having to deploy storage or processing facilities. This will enable them to reach out to a larger customer base. Users will be able to directly access back-end services and integrate them into their regular Python, R, or JavaScript workflows. The standardisation of a now highly fragmented market of often unique services to make them comparable will facilitate users to switch between endpoints.
Since openEO is developed and distributed under a permissive open source license, both industry and society will benefit from participating together to form openEO: There are no requirements or limitations on how the developed software can be used or redistributed. Permissive licenses do not control the license terms of future products, which build on openEO. Distributing openEO explicitly under the Apache license 2.0 will simultaneously guarantee the dissemination of the open source code without any changes in the original version.
140+ pre-defined functions were developed to be used for the standardised communication. These functions cover following aspects:
- Authentication and user data management
- Query of EO data and processes available at the endpoints
- Band calculations and time series statistics
- Filtering and aggregation of specific dimensions
- UDF execution
- Zonal statistics
- Upload and use of vector data
- EO Data download
User-defined functions can be transferred via openEO to various back-ends for execution, allowing the execution of tailored workflows. Use cases have been implemented for defined pilot users to i) to validate the openEO standard and its potential to allow comparability between service providers, and to ii) be further used by at least the pilot users.
The openEO access to back-end providers represents a value adding services of EO data storage and processing services. It may empower both public and commercial users to use a standardised interface to realise EO data processing without having to deploy storage or processing facilities. This will enable them to reach out to a larger customer base. Users will be able to directly access back-end services and integrate them into their regular Python, R, or JavaScript workflows. The standardisation of a now highly fragmented market of often unique services to make them comparable will facilitate users to switch between endpoints.
Since openEO is developed and distributed under a permissive open source license, both industry and society will benefit from participating together to form openEO: There are no requirements or limitations on how the developed software can be used or redistributed. Permissive licenses do not control the license terms of future products, which build on openEO. Distributing openEO explicitly under the Apache license 2.0 will simultaneously guarantee the dissemination of the open source code without any changes in the original version.