EO Big Data Shift

Activities are expected to address the adaptation of big data technologies to Copernicus user scenarios and should concentrate on the intermediate layers describe above. They shall enable Copernicus services, public and intermediate commercial users to engage with and serve their constituency with localised/specialised higher value services.

Activities should include the development of tools allowing for the chaining of different value adding activities increasing incrementally the information and knowledge content of EO and non EO data and possibly triggering new commercial initiatives. The aim would be to allow many users either public or private to provide advanced services to intermediate or end users without having to build up storage and processing capacities for Copernicus data and information but benefiting from the storage and processing services provided by ICT companies.

Big Data, activities shall bridge the gap between Earth observation and information technology sectors taking into account the user needs for EO Big Data and aiming at developing innovative solutions taking into account the needs of 1) non-expert users like policy makers involved in societal challenges, 2) experts involved, and 3) small and medium innovative enterprises. Activities shall be complementary to activities enabled by the ICT and research infrastructures work programmes which address generic challenges in the area of data mining, open linked data, web ontology, digital earth[[For example e-infrastructure for Research: Network (GÉANT), processing (PRACE), data network, Federation of research infrastructure with single sign on (eduGAIN).]].

Activities should address any relevant aspect of the data lifecycle which can solve EO big data challenges, in particular data management activities (e.g. collection, processing including online processing, quality control, documentation, dissemination, cataloguing, preservation, usage tracking, integration) and usage activities (e.g. discovery, analysis (including visual), product generation, user feedback, tagging, knowledge extraction, decision making). Activities are also expected to extensively use flexible coverage and open processing standards.

Activities shall rely on open source software/tools/modules/plug-ins and shall include small-scale demonstrations.

Proposers are advised to consult information on the Copernicus programme in general and linked actions within Copernicus including the Integrated Ground Segment at the Commission's website[[http://copernicus.eu]]. An information document is published together with this work programme[[http://ec.europa.eu/growth/sectors/space/research/horizon-2020]].

Proposers are invited to consult further information on availability of Copernicus Sentinel Data, access to Copernicus Contributing Mission data, as well as issues recommended to be detailed in the proposals at the Commission’s website[[http://www.copernicus.eu/main/data-access]].

In projects to be funded under this topic participation of industry, in particular SMEs, is encouraged.

The Commission considers that proposals requesting a contribution from the EU of between EUR 1 and 2 million would allow this specific challenge to be addressed appropriately. Nonetheless, this does not preclude submission and selection of proposals requesting other amounts.

Effective access to Copernicus dedicated mission data and Copernicus service information by public and private users is a sine qua non condition for the achievement of Copernicus' objectives.

In this context, Copernicus faces important challenges. First, the multiplicity of Copernicus partners involved in Copernicus dissemination activities requires both flexible and effective coordination. At the moment Copernicus dissemination infrastructure is built around different dissemination platforms operated by ESA, EUMETSAT, and service operators. In addition some Member States provide access to Sentinel data in the context of the collaborative ground segment. Second, the sheer volume of data and information to be disseminated and used puts Copernicus at the forefront of the big data challenges. This new paradigm requires a change of approach to data curation and dissemination, in the form of a technological leap to both ingest processing and make available the increased volume of Copernicus data and information considering both temporal and spatial resolutions. The Big Data paradigm offers new perspectives for data intensive activities where Europe could still close its technological gap with the US with huge industrial implications.

The free, full and open data policy will support the development of a strong Earth observation downstream service industry if an effective and scalable access system is implemented to meet the Big Data exploitation challenges and to address the full data cycle needs (e.g. standardised data query, retrieval, data exchange methods, processing and data fusion involving diverse datasets). Therefore, Europe needs to foster Copernicus access and dissemination services spurred by a vibrant European downstream sector taking advantage of the timely availability of the Copernicus data and information to provide innovative Earth observation information products on a worldwide basis, based on European Internet platforms using advanced big data technologies and serving a worldwide market.

The future Copernicus data access architecture is expected to follow the following broad approach:

• A back office service essentially ensuring access to Copernicus data and information and offering storage and processing capacities.
• Different front office services managed by intermediate users (public or private) would serve the need of other intermediate users or end-users via appropriate tool (e.g. search, visualisation, data analytics, knowledge extraction, animation of user communities, etc.
• Intermediate layers allowing the exploitation of the back office resources for the benefit of the various front offices as well as providing the necessary modules to foster EO data analytics and the chaining of value adding activities between different front offices.

This multi-layered approach would allow the mutualisation and efficient use of storage and processing capacities (generic or EO-specific) while providing flexibility to Copernicus services, Member States, intermediate commercial users to engage with and serve their constituency with localised/specialised higher value services.

• Enable value adding services on generic data and information storage and processing facilities which can allow public and commercial users effective production environment to interact with and serve their user base without deploying their own storage and processing facilities.
• Make access to the Copernicus data and information easy and user friendly through scalable dissemination and exploitation software based on international standards.
• Foster the establishment of interoperable access facilities to all EU Member States.
• Link with other big data initiatives.
• Provide user community tools including best-practices.
• Ensure resilience of the overall dissemination and exploitation system.
• Optimise the use of Copernicus data by non-traditional user communities to meet societal challenges.