Final Report Summary - SEN3APP (PROCESSING LINES AND OPERATIONAL SERVICES COMBINING SENTINEL AND IN-SITU DATA FOR TERRESTRIAL CRYOSPHERE AND BOREAL FOREST ZONE)
Executive Summary:
Development, implementation, operationalization and validation of Sentinel data processing lines for cryospheric (terrestrial) and land cover/phenology applications were main objectives of SEN3APP project. Operational capabilities of the Finnish Meteorological Institute (FMI) Sodankylä National Satellite Data Centre (NSDC) were applied to host part of the infrastructure and also complete processing lines. During the project implementing of Copernicus Collaborative Ground Segment to Sodankylä has been made in order to maximise the utilisation of Copernicus Sentinels for provided services and products of SEN3APP project by the project partners.
Twelve products participated in the SEN3APP demonstration phase. The products were generated by different SEN3APP partners. Several of the operational services are also continued after the official demonstration phase, and are still actively running. Products and services were generated and validated during the demonstration phase of the SEN3APP project.
Product validations were conducted by the service operator/ product developer. The validation datasets were mainly from years 2014-2016. The near real time (NRT) validation was piloted for the Baltic Sea FSC (Fractional Snow Cover) product. The SEN3APP system validation focused on monitoring the data production and dissemination.
Promotional materials brochures were prepared. Dissemination workshops in Helsinki, Finland and Vienna, Austria were successfully organized. Training material, entitled, “EU FP7 SEN3APP Product Descriptions & Data Access” regarding SEN3APP project was prepared. The SEN3APP portal which provides detailed information about how to download and access all the products was provided within the SEN3App project.
Market analysis was conducted using surveys, face to face and other communications such as email, tele-conference, etc. with some of the end-users of the SEN3APP project. Business models, service scenarios, exploitation and business plans were reported.
All SEN3APP products and services are made available to end-users via SEN3APP web portal.
Project Context and Objectives:
The Sentinel- satellite series aims at frequent global coverage of the Earth surface in full spectrum of remote sensing. This enables the use of well-established satellite products, built up with earlier more research oriented satellites, to be used for the benefit of people in six core areas of Copernicus: security, land monitoring, climate change, atmosphere monitoring, emergency management and marine environment monitoring. The SEN3APP- project addressed three of these, namely climate change, land monitoring and security. At the heart of many of the challenges for the global human community lies the climate change. The understanding of the processes requires vast amount of information. The cryosphere, especially seasonal snow cover, frozen ground and permafrost, plays a very important part in the Earth’s energy balance system. The inter- and intra-annual changes in the cryosphere, phenology and land cover have proven relevant not only for climate change studies, hydrology, hydropower, traffic etc. but they have an impact to and interact with many other environmental phenomena. For example, changes in the water quality (freshwater and coastal water) are significantly affected by the duration of winter conditions. Snow melting as well as the thawing and freezing of soil, induce and restrain different processes in the aquatic ecosystems. Changes of land cover are relevant in these processes as well.
Main objective of SEN3APP project was to develop and implement operational and validated Sentinel data processing lines for cryospheric (terrestrial) and land cover/phenology applications. Sub-objectives were
• Define a system and applications including interfaces
• Define processing line of optical system
• Define processing line of SAR system
• Develop tools for utilization of Sentinel satellite data for snow, glaciers, lake ice, soil and land cover change/phenology monitoring in boreal forest and sub-arctic zone
• Define external service interaction and access systems for user-controlled - service provision
• The satellite products and services adapted to Sentinel-satellites will be validated using the QA4EO framework.
The scientific and technical approach of SEN3APP were planned to achieve the project goal and objectives in a realistic, measurable and specific manner within 3 years. The work was divided into the following 8 Work Packages.
• WP1 Users’ Consultation and feedbacks
• WP2 System Architecture, Data Management and Processing
• WP3 Product and Service Generation
• WP4 Product and Service Validation
• WP5 Demonstration
• WP6 Dissemination and Exploitation
• WP7 Scientific and Technical coordination
• WP8 Management
Project Results:
1. SYSTEM ARCHITECURE, DATA MANAGEMENT AND PROCESSING
The SEN3APP system requirement specification covers the satellites and instruments to be utilized and the supported data formats, as well as the functional and non-functional requirements for the system.
Sentinel-1 is used as the Synthetic Aperture Radar (SAR) satellite data source in SEN3APP, offering dual polarisation C-band SAR data from two satellites, in four acquisition modes: Stripmap (SM), Interferometric Wide swath (IW), Extra Wide swath (EW) and Wave mode (WV). The main optical satellites used in the project are Sentinel-2 with its multi-spectral instrument (MSI) and Sentinel-3 with its Sea and Land Surface Temperature Instrument (SLSTR) and Ocean and Land Colour Instrument (OLCI).
The functional requirements for the system were mainly derived from the identified user requirements. For data access, the project used the ESA Sentinel Scientific Data Hub, the FMI Sodankylä Collaborative Ground Segment (including the national Sentinel mirror site and the Ground Segment with direct reception of Sentinel data) and the NASA download site for the MODIS atmospheric data.For input data, the specific product levels required for each instrument were determined. Spatial requirements for the data were determined as in the following. For snow, lake ice and phenology products based on Sentinel-1 and Sentinel-3: the Pan-European area (72°N/11°W – 35°N/50°E) and also regional coverage (at least the Alpine area and UK). Similar regional coverage requirements apply for Sentinel-2 data. For glacier products based on Sentinel-1 and Sentinel-2 data: selected glaciers or glacier regions all over the world, including Greenland periphery. For land cover products: the area of Finland.
Temporal requirements for the data were determined as in the following. For snow and lake ice products, near-real-time availability and processing of Sentinel-1, Sentinel-2 and Sentinel-3 data is required. Latency time should be no more than 12 hours, preferably in the order of six hours or faster. For snow products, the required temporal availability of data was determined to vary throughout a year. For phenology products, the season covers for Sentinel-3 (and Sentinel-2) from February until November. For glacier products, the requirements for Sentinel-1 depend on the glacier. For Sentinel-2 data, acquisitions are useful only in late summer (August – October) for glacier outline and snow/ice area product generation. For land cover products the period of interest for Sentinel-2 data is from April to July/August. The latency requirement is of 1-2 days.
Data/product provision requirements specified that for automated near-real-time product generation Sentinel products should be accessible via FTP or HTTP. For on demand generated glacier products, Sentinel data should be searchable and selectable via an online Web interface, and downloadable via FTP or HTTP.
Additionally, a number of non-functional requirements for the system were identified and concisely defined, namely for system availability, performance and storage, data throughput and security.
The internal system interfaces are arranged through a common data pool, which serves as an intermediate and as output data storage for the optical and SAR processing lines. The output products of the processing lines are written in GeoTIFF, NetCDF or Shapefile (GLMIS standard) files to the data pool, where they are picked up by the applications. The output of the applications is also written to the data pool, mainly in raster formats. The data is hosted in the data pool and served to end users via OGC WMS/WCS output interfaces.
The SEN3APP system architecture consists of four primary operating components and a data pool. The components are: Data ingestion processes, Optical and SAR pre-processing, Applications, and Product access interfaces. These are described in respective subsections of this document.
The data pool serves as an intermediate output data storage for the optical and SAR processing lines. It is also used by the applications to ingest and output data, and to serve the data to end users via output interfaces. To allow quick access, the data pool is using disks instead of tapes. Historical and archived data will be fetched from long term archives (LTA) and ingested into the data pool when needed.
Deployment options for the SEN3APP system include distributed and centralized approaches, either sufficiently expanding the existing operational systems, or extending highly scalable cloud-based data hosting services. During the project, the system has been deployed at the Sodankylä National Satellite Data Center (NSDC) of the Finnish Meteorological Institute (FMI), using a combination of new and existing infrastructure. The processors and processing chains are partially new developments and partially rely or have a heritage from previously existing software.
A second SEN3APP system deployment with ingestion, data pool and processor components is set up by ENVEO. This system uses Sentinel data that is outside the radius of the Sodankylä receiving station but relevant for product generation within SEN3APP. Regardless of the dual deployment scenario, all products are disseminated from a single source.
To speed up high-volume access to Sentinel data for a high number of concurrent national users, a national long-term archive is set up at the FMI Sodankylä centre. The archiving component consists of Long-Term Archive (LTA), a National mirror as a rolling on-line archive, and a catalogue service that provides metadata of the stored Sentinel data and value added products.
The SEN3APP processes are decentralized within various systems hosted by the Sodankylä NSDC. Depending on each process, it is performed in a virtual environment or in a cluster processing system. The available processing capacity is adequate to host all the defined processing chains.
As part of the NSDC, the Apache Hadoop based Calvalus system is serving as a re-processing and research platform that allows efficient validation of new algorithms and processors and generation of refined long-term time series of physical parameters of interest.
With regard to operating systems, the NSDC uses Red Hat Enterprise based virtual environment, where the users are able to select any operating system (OS) of choice. NSDC prefers UNIX/LINUX systems, and by default the virtual servers are CentOS 7. Calvalus runs on Hadoop 2.4 and Ubuntu 14.04 LTS is the preferred operating system for the processing nodes. Calvalus is capable of running linux-executable and BEAM based processor bundles.
Monitoring of processing systems, networks, infrastructure, services and disk space is arranged at the NSDC through Nagios (http://www.nagios.org/). The system components and processes can be attached to Nagios monitoring system which is maintained by the FMI ICT unit. Nagios sends alerts to specified user groups about identified problems or communication errors and also informs when the problems are solved.
All the SEN3APP processes are monitored during the office hours. Additionally, vital processes can be attached to FMI 24/7 h surveillance which is staffed around the clock daily.
Each process and service writes comprehensive log files. Logs are stored and can be used for problem solving and monitoring the system performance.
On the virtual environment running the SEN3APP Application processes, the processing can be controlled by a stream-based processing system. The Batch Processing System (BPS) allows executing processing jobs sequentially to restrict server workload.
Pre-processing lines were implemented for Sentinel-2 data and Suomi-NPP/VIIRS data. The Suomi-NPP/VIIRS data were used as a substitute for Sentinel-3 data, which was released for use only during the second last project month of project SEN3APP.
For reading Sentinel-2 data, the VTT-developed em_unpack tool was updated to input Sentinel-2 images. This tool:
• uncompressed the JPEG-2000 compressed per-band files of each granule,
• combines the images of each resolution (10, 20, and 60 m) into a multi-band file,
• mosaics together all granules of a UTM zone, and
• combines the sun/satellite angle grids per UTM zone.
When needed, the user can continue to re-projecting and combining the UTM zones with their preferred image processing tools like gdalwarp, ERDAS/ERMapper, ENVI and so forth.
For processing Sentinel-2 images over large areas, a set of tools were implemented for:
• Combination of the sun/satellite angle grids of UTM zones and re-projecting to another projection,
• Combination of Sentinel-2 image files of UTM zones and re-projecting to a chosen UTM zone or the Lambert Equal Area Azimuthal projection, and
• For estimation of a harmonized sun/satellite angle grids.
These tools were tested for a large-area test dataset covering the Boreal zone of Northern Europe.
For correcting the effects of varying aerosol in Sentinel-2 type images, a software tool was made for estimating the aerosol in the image. Program comp_aod calculates the Aerosol Optical Density (AOD) field using the blue band in 0.49 µm and the SWIR band in 2.19 µm. In Finland, the Corine-2012 land cover map is used to locate dark dense vegetation (mature forest) in AOD field calculation. AOD values are iterated by the Simplified Method for Atmospheric Correction (SMAC) for those areas and interpolated for remaining pixels. This AOD field is then used in the Envimon tool em_radio to make the atmospheric correction, i.e. to compute surface reflectance values for the pixels of the Sentinel-2 image.
The pre-processing line for Suomi-NPP/VIIRS (National Polar-orbiting Operational Environmental Satellite System Preparatory Project/Visible Infrared Imaging Radiometer Suite) data takes as input SDR data (Sensor Data Record). The processing line consists of tools for reading the input hdf5 (Hierarchical Data Format version 5) files, rectification tools, and atmospheric correction tools.
The hdf5 reading programs accept data from the FMI Sodankylä receiving station and data from the archive service CLASS (Comprehensive Large Array-data Stewardship System) of NOAA (National Oceanic and Atmospheric Administration). The rectification tools include resampling alternatives nearest neighbour and Gaussian-weighted average. The atmospheric correction tools utilize the SMAC program of CNES. Both constant AOD (Aerosol Optical Density) and variable AOD (by iteration with SMAC) are supported.
Cloud detection and masking methods were developed and implemented for Sentinel-2 and Suomi-NPP/VIIRS type of data. Program comp_cloud_s2 computes cloud and shadow mask from Sentinel-2 images and comp_cloud_VIIRS for Suomi NPP VIIRS images. Both are based on same algorithm and same software. The only difference is that comp_cloud_s2 contains the integration of the 10 m, 20 m, and 60 m Sentinel-2 bands for the processing.
The algorithm uses 3 visible bands (blue in 490 nm, green in 560 nm and red in 665 nm), one near-infrared (NIR) band (0.865 µm) and 3 short-wave infrared (SWIR) bands (1.38 µm, 1.61 µm and 2.19 µm). Cumulus cloud detection is based on high reflectance in the visible bands and the cirrus and high altitude cloud detection is based on high reflectance in cirrus band (1.38 µm). The resulting cloud mask contains also other classes like bare agricultural areas and snow and ice areas. Snow and ice is separated from the clouds by the Normalized Difference Snow Index using bands 1.38 µm and 1.61 µm and the bare agricultural areas are separated from clouds by the ratio between the NIR (865 nm) and the visible bands. Shadow detection is based on the ratio between blue (490 nm) and green (560 nm) bands. The resulting shadow mask contains also other classes like shore and shallow water areas and dark dense vegetation areas. These areas are separated from the shadow mask by differences in the spectral profiles. In post-processing, all one pixel sized objects are removed and the cloud and cirrus objects are dilated by 6 pixels and shadow objects by 4 pixels before the final cloud and shadow mask.
The Sentinel-1 (S1) pre-processing consists of a set of shell scripts that are grouped into 2 main modules:
• S1 GRD (Ground Range Detected) Processor and
• S1 SLC (Single Look Complex) Processor.
The scripts steer the processing of common tasks and base on COTS (Commercial Off The Shelf) GAMMA software tools (Table 1) as well as FOSS (Free and Open Source Software) GDAL (Geospatial Data Abstraction Library), and common Linux tools. The shell scripts are designed in a modular way to simplify maintenance and improve reliability of the scripts.
Prerequisites for the Sentinel-1 pre-processing line are:
• Installation of the latest release of the GAMMA Software packages ISP, DIFF, LAT (binaries).
• GDAL >= 1.10
• SEN3APP Sentinel-1 pre-processing shell scripts
ESA provides Sentinel data access to users via Data Hub Services (DHuS). There are two different services; Science Data Hub (SciHub) which is open for everyone and Collaborative Data Hub (ColHub) which is only available for Sentinel Collaborative Ground Segments (CGS). FMI, as a CGS, uses ColHub to mirror Sentinel 1, 2, 3 and 5P data to Sodankylä National Satellite Data Center (NSDC).
In addition to the DHuS, Sodankylä NSDC uses the near-real time Sentinel-1 data as a part of the Collaborative Acquisition Station (CAS) activities. The Sentinel-1A and 1B data is received and processed using the existing infrastructure and ingested into the SEN3APP system. Sentinel-3 was launched during the SEN3APP project but the data was not available. Thus, the Suomi NPP (S-NPP) satellite data is used for developing corresponding SEN3APP processing lines and applications. Sodankylä NSDC receives S-NPP on daily basis and the data is provided in near-real time (NRT) through the disk system.
Landsat 8 satellite data was used to demonstrate and develop SEN3APP processing chains until Sentinel 2 data was available. USGS data service is used to fetch Landsat 8 data to Sodankylä NSDC and ingested to SEN3APP system. Currently the service is only available through a semi-automatic bulk download application (BDA). From the beginning of the year 2016, Sentinel-2 data is ingested from ColHub to NSDC to be used in SEN3APP applications.
ESA has released its DHuS software as an open source that FMI NSDC uses to distribute local Sentinel data to SEN3APP applications and other users. FinHub, available at https://finhub.nsdc.fmi.fi is open for everyone who has need for faster and non-restricted access (no parallel download limits) to Sentinel data.
The Sodankylä NSDC has built a web portal (http://saana.nsdc.fmi.fi/apollo-portal/) for data distribution (other than Sentinel L1 data) from a single point of access. The SEN3APP products are being made available through the portal linked from the project website. The portal is based on ERDAS APOLLO, an OGC/ISO-based web application that is developed by Hexagon Geospatial and implemented on a geospatial platform that enables high performance OGC web services. The data on the service is accessible with a web browser or with a GIS tool.
The web service allows users to browse and search the data from a catalogue. The data content can be restricted based on the user. Some of the data can be made available only after logging in to the system. The data can be shown on the WMS or downloaded to the user’s computer. The data can also be opened directly with 3rd party software such as Google Earth. The WMS service offered allows basic tools to select the user desired data.
The data interfaces offered employ the Open Geospatial Consortium’s (OGC) Web Map Service (WMS) and Web Coverage Service (WCS) specifications. Technically, all the metadata interfaces are XML-based and work over the HTTP protocol. The SEN3APP web portal offers example scripts to help users to request the data without using the ERDAS APOLLO GUI. The scripts are similar to those provided by the CryoLand portal and can be used to automate the access to the WCS interface to gather information about the service and receive data.
User controlled services of SEN3APP are realized by giving external users access to virtual machines hosted on the infrastructure of the Arctic Research Centre (ARC) of the Finnish Meteorological Institute (FMI). The machines can be used to run specialized processing lines for custom applications. In addition to the processing capacity, the machines have a fast connection to the SEN3APP Near-Real Time (NRT) data pool as well as to the data archive.
The processing is mainly done on virtual machines. A typical virtual machine for running the processing has a 64bit Intel dual-core 2,5 GHz CPU and is equipped with 4 GB RAM. The processing capabilities can be easily expanded when using virtualization. Also resources that are not used at the moment can be used by other processors. Depending on the operational level, a virtual machine can be granted higher privileges that allow using more resources compared to other machines.
The processing machines have access to the common storage (Lustre) where users can find the source data and products for processing. This intermediate data storage contains only the latest data of the whole data centre. The NRT data is available for the users from a rolling archive. The data on the rolling archive is staged on the drives for fast access upon request. From the rolling archive the data is moved to a long term archive (LTA) where it is first placed on disks and eventually to tapes. Data from the LTA is also available for users but the recovery from tapes takes more time than from disks. NSDC is currently renewing its storage system. The new storage, planned to be active at the beginning of 2017, will have increased storage capacity and instead of tapes, all the data is stored to Solid State Disks (SSD) that provide faster data access.
Backup will be performed for the processing machines usually once per week or as agreed with the user. The services are targeted to run 99.9 % of time and the users are informed about the outages due to routine maintenance of the computer systems or in case of unexpected malfunctions. External users can access an on-line ticketing service that is used to request for support and report problems regarding the processing services. The used software is JIRA that is developed by Atlassian for issue and project tracking.
2. PRODUCT AND SERVICE GENERATION
Land cover change and phenology for boreal zone:
The following products are produced:
• Crop/vegetation classification
• Phenology Product
• Water Body Map
• Freeze/Thaw
Snow for boreal forest and mountain zone:
The following products were established within SEN3APP:
• Fractional Snow Cover extent for Northern Hemisphere from optical data
• Combined (2-layer, 25km) Northern Hemisphere snow cover product with optical FSC and PMW SWE –layers
• High resolution (5km) Pan-European SWE product (augmented using optical FSC data)
• Wet snow cover for the Alpine area from Sentinel-1 data
• Regional and Pan-European FSC product from synergistic Sentinel-3 SLSTR/OLCI data
• Extended Baltic Sea drainage basin direct broadcast FSC based on NPP VIIRS/Sentinel-3 SLSTR
Glaciers:
Glacier products from Sentinel data are generated on demand for particular areas of interest identified by users. Users show interest in the following glacier products, each generated from high resolution satellite data, as available from Sentinel-1 and Sentinel 2:
• Glacier outlines
• Snow and ice areas on glaciers
• Ice surface velocity maps
3. PRODUCT AND SERVICE VALIDATION
To use the satellite data based products the user needs naturally information about the product itself, but also about its uncertainty. The QA4EO (Quality Assurance for Earth Observations) initiative was established to provide guidelines how to report uncertainty of satellite (and other earth observation) data. Since the start of the project the QA4EO framework has not been actively updated. The framework still provides a good starting point to create documentation to accompany the data, which provides the user enough information to evaluate the suitability of the data to the use.
The QA4EO key principle is stated as: “Data and derived products shall have associated with them a fully traceable indicator of their quality.” The two core requirements in this statement are that the dataset has:
• Quality indicator: “A Quality Indicator (QI) shall provide sufficient information to allow all ausers to readily evaluate the “fitness for purpose” of the data or derived product.”
• Traceability: “A QI shall be based on a documented and quantifiable assessment of evidence demonstrating the level of traceability to internationally agreed (where possible SI) standards”
The SEN3APP validation documentation focuses on the quality assurance of the satellite data products. The documentation assumes that the user is familiar with the product content, although references are given to documentation describing the product generation. The validation is presented in a manner that allows the user, also to follow the protocol if the data or similar data is available. The main content of the validation documentation are:
• General information: Document ID, contact information etc.
• Main results of the validation, i.e. the accuracy of the product
• Datasets used
• Methods used
• Additional information on the validation
• References
4. DEMONSTRATION
Twelve products participated in the SEN3APP demonstration phase from July 2015 – May 2016:
1.Crop / vegetation classification (pilot)
2.Phenology Product (under development)
3.Fractional Snow Cover (FSC) for NH (operational)
4.High resolution Pan-European SWE product (operational)
5.Regional wet snow cover (pilot)
6.Regional FSC (operational)
7.Pan-European FSC (operational)
8.Extended Baltic Sea drainage basin direct broadcast FSC (operational)
9.Glacier outlines (on demand)
10.Glacier ice surface velocity (on demand)
11.Snow / ice areas on glaciers (on demand)
12.Lake ice extent (operational)
The products are generated by different SEN3APP partners. Several of the operational services were also continued after the official demonstration phase, and are still actively running.
The SEN3APP products and services are provided to users through two geoportals: the FMIARC GeoPortal, installed at FMI, and the CryoLand GeoPortal hosted by ENVEO. Both portals are accessible through the SEN3APP Portal (http://sen3app.fmi.fi/index.php?style=main&page=Products).
The following products are distributed through FMIARC GeoPortal, and are available using WCS and WMS.
• Fractional Snow Cover Extent for Northern Hemisphere from Optical Data
• High Resolution (5km) Pan-European SWE Product (Augmented Using Optical FSC)
• Extended Baltic Sea Drainage Basin Direct Broadcast FSC Based on NPP VIIRS/Sentinel-3 SLSTR
• Phenology
• Crop / Vegetation Classification
• Ice Velocity
Some of the products are distributed through the CryoLand GeoPortal, but sample products can also be accessed from the FMIARC GeoPortal:
• Regional Wet Snow Cover from Sentinel-1 Data
• Regional and Pan-European Fractional Snow Cover Product from Synergistic Sentinel-3 SLSTR/OLCI Data
For some key end users, products are tailored to their needs and provided directly via file transfer protocol (FTP), or sent via e-mail. This includes also products generated only on demand.
Potential Impact:
Promotional materials brochures were prepared. These brochures were distributed thorough meetings, workshop and conferences by the project partners. 2 Dissemination workshops in Helsinki, Finland and Vienna, Austria were successfully organized. 1st Dissemination workshop was held on November 19, 2016 at the FMI in Helsinki, Finland. The participants of workshop were mainly from Finland as focus on two end-users MAVI and FORTUM and the position of the Sodankyla NSDC in the project. However the whole workshop is broadcasted online on YouTube during the day. About 25–30 persons have participated the workshop in person and about 6 viewers in average have followed the broadcast with a peak of ten viewers.
2nd dissemination workshop was held April 18, 2016 at the ZAMG in Vienna, Austria. 33 people has participated to the workshop in person. The objectives of 2nd dissemination workshop were as follow
1. To disseminate products and services provided by SEN3APP project to wider end-user community possible future end-users which has scope of interest as same as SEN3APP project
2. To present demonstrated services and products of SEN3APP project
3. To give hands-on training to use services and products provided by SEN3APP project.
In order to achieve the objectives above listed we selected Vienna and timing during the same week EGU (egu.eu) general assembly was held. We co-organized with the COST Action ES1404 (harmosnow.eu) which has over 100 persons from 28 countries. The idea of co-organizing 2nd dissemination workshop with this COST Action and during EGU conference was to increase number of participation and interest international level.
We prepared training material, entitled, “EU FP7 SEN3APP Product Descriptions & Data Access” regarding SEN3APP project. In that training document, the products generated within the EU FP7 project SEN3APP are described and information on how to access the products is provided. Downloading of SEN3APP products via the FMIARC GeoPortal (http://saana.nsdc.fmi.fi/fmiarc-geoportal) and the CryoLand GeoPortal (http://neso1.cryoland.enveo.at/cryoclient/) is described.
SEN3APP products are disseminated from two interfaces, FMI Erdas Apollo system and CryoLand system. The FMI Erdas Apollo system is a data dissemination instance of the Finnish National Satellite Data Centre located in Sodankylä. The CryoLand system is powered by the EOxServer software which was developed within the EU FP7 project CryoLand (2011-2015). The CryoLand system is hosted by ENVEO.
Sodankylä site now hosts the Finnish Collaborative Ground Segment which is providing ESA Sentinel data. Sodankylä National Satellite Data Centre focuses on fast delivery remote sensing product generation for scientific and commercial uses. The data centre’s high performance computer arrays are capable of processing vast amounts of satellite data to value adding products to various users. The products can be delivered directly to customer or to large data archives for bulk use. Satellite data and products can be transferred to users with no delays using high speed optical fibre.
The FMI Erdas Apollo System is powered by Erdas Apollo. Erdas Apollo is an interoperable OGC/ISO-based commercial application that implements an out-of-the-box service-oriented architecture (SOA). It is implemented on a geospatial platform that enables high performance OGC web services from the FMIARC catalog. Users can interact with the FMIARC catalog directly using a Web Coverage Service (WCS) interface, a Web Map Service (WMS) interface, the Web Feature Service (WFS) interface, and the Web Registry Service (WRS) interface. All of these interfaces are compliant with the standards established by the Open Geospatial Consortium (OGC).
The CryoLand GeoPortal is the online data access point for snow and land ice services developed and established during the EU FP7 project CryoLand (No. 262925, 2011 - 2015). Products in the CryoLand system are generated and provided by different operators based on freely available satellite data, but for the end-user, all products are accessible via a centralized portal. The CryoLand system has a basic FTP Server hosted by ENVEO for storing most of the generated snow and land ice products. Some products are only made available via the FTP server of other product providers. These external FTP servers are also linked with the CryoLand system. The integration of the different products into the CryoLand system and the user interfaces are controlled via the Open Source EOxServer software (http://eoxserver.org) running on a virtual machine hosted by ENVEO.
The SEN3APP portal provides detailed information about how to download and access all the products that are provided within the SEN3App project. SEN3App products page is created under the SEN3App webpage (http://sen3app.fmi.fi/) and available from http://sen3app.fmi.fi?page=Products. In the page, the products generated within the project, are described and information on how to access the products is provided. For each product, a specific page, where the detailed information and previews can be found, is created and available from the links in the product page.
Preliminary stakeholders defined at the beginning of the project and can be listed as follow
• Agency for Rural Affairs (MAVI)
• Tyrol State Government (Hydrography and Hydrology)
• Central Institute for Meteorology and Geodynamics (ZAMG)
• MetOffice (United Kingdom)
• Fortum Energy
• Freshwater Centre of the Finnish Environment Institute
During the project time, especially during 2nd dissemination workshop possible stakeholders identified as follow:
• Institute of Hydrology, Slovak Academy of Sciences (SAS)
• Institute for applied mathematics (CNR-IAC), Italy
• Middle East Technical University (METU), Turkey
• University of Bern, Switzerland
• University of Oslo, Norway
• University of Leuven, Belgium
• University of Silesia, Poland
• University of Oulu, Finland
• alpS Centre for Climate Change Adaptation
• Estonian Environment Agency
• Slovak Hydrometeorological Institute
• Anadolu University, Eskisehir, Turkey
• WSL Institute for Snow and Avalanche Research SLF, Switzerland
• National Institute of Meteorology and Hydrology, Bulgaria
• Institute of Geography, Romanian Academy
• alpS Centre for Climate Change Adaptation
• Geological Survey of Spain (IGME)
The SEN3APP project is designed to lead to a set of long-term operational services. SEN3APP services will appear as a feasible extension of the current business of the project partners. The current business portfolio of preoperational and operational services delivered by project partners includes
• Snow cover mapping and monitoring for the hydropower industry (production and energy trading)
• Snow cover and glacier mapping and monitoring for hydropower planning and geotechnical work
• Snow cover monitoring for hydrological services
• Snow extent and water equivalent monitoring for ecosystem management
• Glacier monitoring for natural hazard assessment and mitigation (glacial lakes and glacier instabilities)
• Lake ice and river ice products including flood extent during ice jams
In SEN3APP, project partners gain further knowhow in (1) developing methods for processing and using of Sentinel data (2) providing services and products based on Sentinel data. This knowhow and extended service and product portfolio will support project partners’ current and future businesses. New applications are expected to open up through the SEN3APP developments.
Users will have access to various service layers. Free (at the point of access) layers would be supported by public organizations. Individual users or organizations requiring more detail would pay for a subscription service, while specialized companies (particularly those involved in the energy market and in water management) would pay more for tailored solutions.
In SEN3APP, we developed a system which is (a) capable of ingesting the extremely high-volume data of the Sentinel-1,-2 and -3 satellites provided by the Sodankylä ground segment interfaces (b) process the data to create value added product and services (c) integrate the products and services with data from other external sources, and offer both the raw and the processed data to users, via an accessible user interface and standard-based data interfaces, which provide intelligent search mechanisms and allow ordering of customized products and services. Sodankylä National Satellite Data Centre (NSDC) is in the centre of the SEN3APP system. Business scenarios of the developed system are planned to be
1. Customer oriented services
a. Customized high performance processing in processing clusters, Long-term archiving of the data and high speed data delivery.
b. Flexible product generation and hosting of external processing chains in virtual environments
c. Platform as a Service (Paas) and Infrastructure as a Service (IaaS)
d. Imaging and acquisition planning service
2. Direct broadcast and near real time services
a. Copernicus Collaborative Acquisition Station to downlink Sentinel-1 pass-through for quasi.real-time-applications like Baltic Sea Ice service
3. Applications
a. Terrestrial Cryosphere (e.g. Snow and ice maps generation)
b. Meteorological applications
c. Research and development
The activities and tasks of the Sodankylä National Satellite Data Centre (NSDC) is being developed in various projects such as SEN3APP. Overall goal is to develop the NSDC as the centre for satellite data and products in Finland. The centre that serves Sentinel data users with free Sentinel data and provides processing services and data products for national satellite data users and international partners developed during SEN3APP project.
The main barrier for establishing an operational service based on the pilot products developed in SEN3APP is the lacking funding for finishing the development work and establishing the operational service (i.e. setting up sustainable technology, work flow, administration, contracts etc.). The funding opportunities are constantly sought for. There are no other evident obstacles for the development of the operational service. The stakeholder, i.e. MAVI, has strong interest to develop the methodology; the expertise for service development exists and the technology is ready.
Critical factors for the service are:
• The renewal of the platform and process for subsidies control does not support the uptake of the remote sensing data.
• Timely and spatially extensive (covering entire Finland) delivery of the Sentinel-1 and -2 data.
• Contractual agreement on the division of responsibilities and sustained resources.
Efficient use of remote sensing is strongly connected to the reform in the subsidies control process. The advancement in remote sensing is therefore also dependent on the success of the other process.
Sentinel satellites are still young and e.g. the Sentinel-1 imaging program has gone through some changes in the first couple of years and is still under review. With the launch of Sentinel-1b the temporal coverage was improved. The current bottleneck for the commercialisation of Sentinel-1 based NRT services is the ground segment that struggles to provide the data on a reliable base. Use of the optical data is dependent also on the successful launch of the Sentinel-2B satellite as, especially in the northern latitudes, the constraints from the cloud cover are significantly reduced with two operating satellites. Agreement on by the contributing parties needs to be signed to ensure the sustainability of the service.
List of Websites:
http://sen3app.fmi.fi
Project Manager: Dr. Ali Nadir Arslan (ali.nadir.arslan@fmi.fi)
Financial Adminstrator: Mrs. Sari Jokiniemi (sari.jokiniemi@fmi.fi)
Development, implementation, operationalization and validation of Sentinel data processing lines for cryospheric (terrestrial) and land cover/phenology applications were main objectives of SEN3APP project. Operational capabilities of the Finnish Meteorological Institute (FMI) Sodankylä National Satellite Data Centre (NSDC) were applied to host part of the infrastructure and also complete processing lines. During the project implementing of Copernicus Collaborative Ground Segment to Sodankylä has been made in order to maximise the utilisation of Copernicus Sentinels for provided services and products of SEN3APP project by the project partners.
Twelve products participated in the SEN3APP demonstration phase. The products were generated by different SEN3APP partners. Several of the operational services are also continued after the official demonstration phase, and are still actively running. Products and services were generated and validated during the demonstration phase of the SEN3APP project.
Product validations were conducted by the service operator/ product developer. The validation datasets were mainly from years 2014-2016. The near real time (NRT) validation was piloted for the Baltic Sea FSC (Fractional Snow Cover) product. The SEN3APP system validation focused on monitoring the data production and dissemination.
Promotional materials brochures were prepared. Dissemination workshops in Helsinki, Finland and Vienna, Austria were successfully organized. Training material, entitled, “EU FP7 SEN3APP Product Descriptions & Data Access” regarding SEN3APP project was prepared. The SEN3APP portal which provides detailed information about how to download and access all the products was provided within the SEN3App project.
Market analysis was conducted using surveys, face to face and other communications such as email, tele-conference, etc. with some of the end-users of the SEN3APP project. Business models, service scenarios, exploitation and business plans were reported.
All SEN3APP products and services are made available to end-users via SEN3APP web portal.
Project Context and Objectives:
The Sentinel- satellite series aims at frequent global coverage of the Earth surface in full spectrum of remote sensing. This enables the use of well-established satellite products, built up with earlier more research oriented satellites, to be used for the benefit of people in six core areas of Copernicus: security, land monitoring, climate change, atmosphere monitoring, emergency management and marine environment monitoring. The SEN3APP- project addressed three of these, namely climate change, land monitoring and security. At the heart of many of the challenges for the global human community lies the climate change. The understanding of the processes requires vast amount of information. The cryosphere, especially seasonal snow cover, frozen ground and permafrost, plays a very important part in the Earth’s energy balance system. The inter- and intra-annual changes in the cryosphere, phenology and land cover have proven relevant not only for climate change studies, hydrology, hydropower, traffic etc. but they have an impact to and interact with many other environmental phenomena. For example, changes in the water quality (freshwater and coastal water) are significantly affected by the duration of winter conditions. Snow melting as well as the thawing and freezing of soil, induce and restrain different processes in the aquatic ecosystems. Changes of land cover are relevant in these processes as well.
Main objective of SEN3APP project was to develop and implement operational and validated Sentinel data processing lines for cryospheric (terrestrial) and land cover/phenology applications. Sub-objectives were
• Define a system and applications including interfaces
• Define processing line of optical system
• Define processing line of SAR system
• Develop tools for utilization of Sentinel satellite data for snow, glaciers, lake ice, soil and land cover change/phenology monitoring in boreal forest and sub-arctic zone
• Define external service interaction and access systems for user-controlled - service provision
• The satellite products and services adapted to Sentinel-satellites will be validated using the QA4EO framework.
The scientific and technical approach of SEN3APP were planned to achieve the project goal and objectives in a realistic, measurable and specific manner within 3 years. The work was divided into the following 8 Work Packages.
• WP1 Users’ Consultation and feedbacks
• WP2 System Architecture, Data Management and Processing
• WP3 Product and Service Generation
• WP4 Product and Service Validation
• WP5 Demonstration
• WP6 Dissemination and Exploitation
• WP7 Scientific and Technical coordination
• WP8 Management
Project Results:
1. SYSTEM ARCHITECURE, DATA MANAGEMENT AND PROCESSING
The SEN3APP system requirement specification covers the satellites and instruments to be utilized and the supported data formats, as well as the functional and non-functional requirements for the system.
Sentinel-1 is used as the Synthetic Aperture Radar (SAR) satellite data source in SEN3APP, offering dual polarisation C-band SAR data from two satellites, in four acquisition modes: Stripmap (SM), Interferometric Wide swath (IW), Extra Wide swath (EW) and Wave mode (WV). The main optical satellites used in the project are Sentinel-2 with its multi-spectral instrument (MSI) and Sentinel-3 with its Sea and Land Surface Temperature Instrument (SLSTR) and Ocean and Land Colour Instrument (OLCI).
The functional requirements for the system were mainly derived from the identified user requirements. For data access, the project used the ESA Sentinel Scientific Data Hub, the FMI Sodankylä Collaborative Ground Segment (including the national Sentinel mirror site and the Ground Segment with direct reception of Sentinel data) and the NASA download site for the MODIS atmospheric data.For input data, the specific product levels required for each instrument were determined. Spatial requirements for the data were determined as in the following. For snow, lake ice and phenology products based on Sentinel-1 and Sentinel-3: the Pan-European area (72°N/11°W – 35°N/50°E) and also regional coverage (at least the Alpine area and UK). Similar regional coverage requirements apply for Sentinel-2 data. For glacier products based on Sentinel-1 and Sentinel-2 data: selected glaciers or glacier regions all over the world, including Greenland periphery. For land cover products: the area of Finland.
Temporal requirements for the data were determined as in the following. For snow and lake ice products, near-real-time availability and processing of Sentinel-1, Sentinel-2 and Sentinel-3 data is required. Latency time should be no more than 12 hours, preferably in the order of six hours or faster. For snow products, the required temporal availability of data was determined to vary throughout a year. For phenology products, the season covers for Sentinel-3 (and Sentinel-2) from February until November. For glacier products, the requirements for Sentinel-1 depend on the glacier. For Sentinel-2 data, acquisitions are useful only in late summer (August – October) for glacier outline and snow/ice area product generation. For land cover products the period of interest for Sentinel-2 data is from April to July/August. The latency requirement is of 1-2 days.
Data/product provision requirements specified that for automated near-real-time product generation Sentinel products should be accessible via FTP or HTTP. For on demand generated glacier products, Sentinel data should be searchable and selectable via an online Web interface, and downloadable via FTP or HTTP.
Additionally, a number of non-functional requirements for the system were identified and concisely defined, namely for system availability, performance and storage, data throughput and security.
The internal system interfaces are arranged through a common data pool, which serves as an intermediate and as output data storage for the optical and SAR processing lines. The output products of the processing lines are written in GeoTIFF, NetCDF or Shapefile (GLMIS standard) files to the data pool, where they are picked up by the applications. The output of the applications is also written to the data pool, mainly in raster formats. The data is hosted in the data pool and served to end users via OGC WMS/WCS output interfaces.
The SEN3APP system architecture consists of four primary operating components and a data pool. The components are: Data ingestion processes, Optical and SAR pre-processing, Applications, and Product access interfaces. These are described in respective subsections of this document.
The data pool serves as an intermediate output data storage for the optical and SAR processing lines. It is also used by the applications to ingest and output data, and to serve the data to end users via output interfaces. To allow quick access, the data pool is using disks instead of tapes. Historical and archived data will be fetched from long term archives (LTA) and ingested into the data pool when needed.
Deployment options for the SEN3APP system include distributed and centralized approaches, either sufficiently expanding the existing operational systems, or extending highly scalable cloud-based data hosting services. During the project, the system has been deployed at the Sodankylä National Satellite Data Center (NSDC) of the Finnish Meteorological Institute (FMI), using a combination of new and existing infrastructure. The processors and processing chains are partially new developments and partially rely or have a heritage from previously existing software.
A second SEN3APP system deployment with ingestion, data pool and processor components is set up by ENVEO. This system uses Sentinel data that is outside the radius of the Sodankylä receiving station but relevant for product generation within SEN3APP. Regardless of the dual deployment scenario, all products are disseminated from a single source.
To speed up high-volume access to Sentinel data for a high number of concurrent national users, a national long-term archive is set up at the FMI Sodankylä centre. The archiving component consists of Long-Term Archive (LTA), a National mirror as a rolling on-line archive, and a catalogue service that provides metadata of the stored Sentinel data and value added products.
The SEN3APP processes are decentralized within various systems hosted by the Sodankylä NSDC. Depending on each process, it is performed in a virtual environment or in a cluster processing system. The available processing capacity is adequate to host all the defined processing chains.
As part of the NSDC, the Apache Hadoop based Calvalus system is serving as a re-processing and research platform that allows efficient validation of new algorithms and processors and generation of refined long-term time series of physical parameters of interest.
With regard to operating systems, the NSDC uses Red Hat Enterprise based virtual environment, where the users are able to select any operating system (OS) of choice. NSDC prefers UNIX/LINUX systems, and by default the virtual servers are CentOS 7. Calvalus runs on Hadoop 2.4 and Ubuntu 14.04 LTS is the preferred operating system for the processing nodes. Calvalus is capable of running linux-executable and BEAM based processor bundles.
Monitoring of processing systems, networks, infrastructure, services and disk space is arranged at the NSDC through Nagios (http://www.nagios.org/). The system components and processes can be attached to Nagios monitoring system which is maintained by the FMI ICT unit. Nagios sends alerts to specified user groups about identified problems or communication errors and also informs when the problems are solved.
All the SEN3APP processes are monitored during the office hours. Additionally, vital processes can be attached to FMI 24/7 h surveillance which is staffed around the clock daily.
Each process and service writes comprehensive log files. Logs are stored and can be used for problem solving and monitoring the system performance.
On the virtual environment running the SEN3APP Application processes, the processing can be controlled by a stream-based processing system. The Batch Processing System (BPS) allows executing processing jobs sequentially to restrict server workload.
Pre-processing lines were implemented for Sentinel-2 data and Suomi-NPP/VIIRS data. The Suomi-NPP/VIIRS data were used as a substitute for Sentinel-3 data, which was released for use only during the second last project month of project SEN3APP.
For reading Sentinel-2 data, the VTT-developed em_unpack tool was updated to input Sentinel-2 images. This tool:
• uncompressed the JPEG-2000 compressed per-band files of each granule,
• combines the images of each resolution (10, 20, and 60 m) into a multi-band file,
• mosaics together all granules of a UTM zone, and
• combines the sun/satellite angle grids per UTM zone.
When needed, the user can continue to re-projecting and combining the UTM zones with their preferred image processing tools like gdalwarp, ERDAS/ERMapper, ENVI and so forth.
For processing Sentinel-2 images over large areas, a set of tools were implemented for:
• Combination of the sun/satellite angle grids of UTM zones and re-projecting to another projection,
• Combination of Sentinel-2 image files of UTM zones and re-projecting to a chosen UTM zone or the Lambert Equal Area Azimuthal projection, and
• For estimation of a harmonized sun/satellite angle grids.
These tools were tested for a large-area test dataset covering the Boreal zone of Northern Europe.
For correcting the effects of varying aerosol in Sentinel-2 type images, a software tool was made for estimating the aerosol in the image. Program comp_aod calculates the Aerosol Optical Density (AOD) field using the blue band in 0.49 µm and the SWIR band in 2.19 µm. In Finland, the Corine-2012 land cover map is used to locate dark dense vegetation (mature forest) in AOD field calculation. AOD values are iterated by the Simplified Method for Atmospheric Correction (SMAC) for those areas and interpolated for remaining pixels. This AOD field is then used in the Envimon tool em_radio to make the atmospheric correction, i.e. to compute surface reflectance values for the pixels of the Sentinel-2 image.
The pre-processing line for Suomi-NPP/VIIRS (National Polar-orbiting Operational Environmental Satellite System Preparatory Project/Visible Infrared Imaging Radiometer Suite) data takes as input SDR data (Sensor Data Record). The processing line consists of tools for reading the input hdf5 (Hierarchical Data Format version 5) files, rectification tools, and atmospheric correction tools.
The hdf5 reading programs accept data from the FMI Sodankylä receiving station and data from the archive service CLASS (Comprehensive Large Array-data Stewardship System) of NOAA (National Oceanic and Atmospheric Administration). The rectification tools include resampling alternatives nearest neighbour and Gaussian-weighted average. The atmospheric correction tools utilize the SMAC program of CNES. Both constant AOD (Aerosol Optical Density) and variable AOD (by iteration with SMAC) are supported.
Cloud detection and masking methods were developed and implemented for Sentinel-2 and Suomi-NPP/VIIRS type of data. Program comp_cloud_s2 computes cloud and shadow mask from Sentinel-2 images and comp_cloud_VIIRS for Suomi NPP VIIRS images. Both are based on same algorithm and same software. The only difference is that comp_cloud_s2 contains the integration of the 10 m, 20 m, and 60 m Sentinel-2 bands for the processing.
The algorithm uses 3 visible bands (blue in 490 nm, green in 560 nm and red in 665 nm), one near-infrared (NIR) band (0.865 µm) and 3 short-wave infrared (SWIR) bands (1.38 µm, 1.61 µm and 2.19 µm). Cumulus cloud detection is based on high reflectance in the visible bands and the cirrus and high altitude cloud detection is based on high reflectance in cirrus band (1.38 µm). The resulting cloud mask contains also other classes like bare agricultural areas and snow and ice areas. Snow and ice is separated from the clouds by the Normalized Difference Snow Index using bands 1.38 µm and 1.61 µm and the bare agricultural areas are separated from clouds by the ratio between the NIR (865 nm) and the visible bands. Shadow detection is based on the ratio between blue (490 nm) and green (560 nm) bands. The resulting shadow mask contains also other classes like shore and shallow water areas and dark dense vegetation areas. These areas are separated from the shadow mask by differences in the spectral profiles. In post-processing, all one pixel sized objects are removed and the cloud and cirrus objects are dilated by 6 pixels and shadow objects by 4 pixels before the final cloud and shadow mask.
The Sentinel-1 (S1) pre-processing consists of a set of shell scripts that are grouped into 2 main modules:
• S1 GRD (Ground Range Detected) Processor and
• S1 SLC (Single Look Complex) Processor.
The scripts steer the processing of common tasks and base on COTS (Commercial Off The Shelf) GAMMA software tools (Table 1) as well as FOSS (Free and Open Source Software) GDAL (Geospatial Data Abstraction Library), and common Linux tools. The shell scripts are designed in a modular way to simplify maintenance and improve reliability of the scripts.
Prerequisites for the Sentinel-1 pre-processing line are:
• Installation of the latest release of the GAMMA Software packages ISP, DIFF, LAT (binaries).
• GDAL >= 1.10
• SEN3APP Sentinel-1 pre-processing shell scripts
ESA provides Sentinel data access to users via Data Hub Services (DHuS). There are two different services; Science Data Hub (SciHub) which is open for everyone and Collaborative Data Hub (ColHub) which is only available for Sentinel Collaborative Ground Segments (CGS). FMI, as a CGS, uses ColHub to mirror Sentinel 1, 2, 3 and 5P data to Sodankylä National Satellite Data Center (NSDC).
In addition to the DHuS, Sodankylä NSDC uses the near-real time Sentinel-1 data as a part of the Collaborative Acquisition Station (CAS) activities. The Sentinel-1A and 1B data is received and processed using the existing infrastructure and ingested into the SEN3APP system. Sentinel-3 was launched during the SEN3APP project but the data was not available. Thus, the Suomi NPP (S-NPP) satellite data is used for developing corresponding SEN3APP processing lines and applications. Sodankylä NSDC receives S-NPP on daily basis and the data is provided in near-real time (NRT) through the disk system.
Landsat 8 satellite data was used to demonstrate and develop SEN3APP processing chains until Sentinel 2 data was available. USGS data service is used to fetch Landsat 8 data to Sodankylä NSDC and ingested to SEN3APP system. Currently the service is only available through a semi-automatic bulk download application (BDA). From the beginning of the year 2016, Sentinel-2 data is ingested from ColHub to NSDC to be used in SEN3APP applications.
ESA has released its DHuS software as an open source that FMI NSDC uses to distribute local Sentinel data to SEN3APP applications and other users. FinHub, available at https://finhub.nsdc.fmi.fi is open for everyone who has need for faster and non-restricted access (no parallel download limits) to Sentinel data.
The Sodankylä NSDC has built a web portal (http://saana.nsdc.fmi.fi/apollo-portal/) for data distribution (other than Sentinel L1 data) from a single point of access. The SEN3APP products are being made available through the portal linked from the project website. The portal is based on ERDAS APOLLO, an OGC/ISO-based web application that is developed by Hexagon Geospatial and implemented on a geospatial platform that enables high performance OGC web services. The data on the service is accessible with a web browser or with a GIS tool.
The web service allows users to browse and search the data from a catalogue. The data content can be restricted based on the user. Some of the data can be made available only after logging in to the system. The data can be shown on the WMS or downloaded to the user’s computer. The data can also be opened directly with 3rd party software such as Google Earth. The WMS service offered allows basic tools to select the user desired data.
The data interfaces offered employ the Open Geospatial Consortium’s (OGC) Web Map Service (WMS) and Web Coverage Service (WCS) specifications. Technically, all the metadata interfaces are XML-based and work over the HTTP protocol. The SEN3APP web portal offers example scripts to help users to request the data without using the ERDAS APOLLO GUI. The scripts are similar to those provided by the CryoLand portal and can be used to automate the access to the WCS interface to gather information about the service and receive data.
User controlled services of SEN3APP are realized by giving external users access to virtual machines hosted on the infrastructure of the Arctic Research Centre (ARC) of the Finnish Meteorological Institute (FMI). The machines can be used to run specialized processing lines for custom applications. In addition to the processing capacity, the machines have a fast connection to the SEN3APP Near-Real Time (NRT) data pool as well as to the data archive.
The processing is mainly done on virtual machines. A typical virtual machine for running the processing has a 64bit Intel dual-core 2,5 GHz CPU and is equipped with 4 GB RAM. The processing capabilities can be easily expanded when using virtualization. Also resources that are not used at the moment can be used by other processors. Depending on the operational level, a virtual machine can be granted higher privileges that allow using more resources compared to other machines.
The processing machines have access to the common storage (Lustre) where users can find the source data and products for processing. This intermediate data storage contains only the latest data of the whole data centre. The NRT data is available for the users from a rolling archive. The data on the rolling archive is staged on the drives for fast access upon request. From the rolling archive the data is moved to a long term archive (LTA) where it is first placed on disks and eventually to tapes. Data from the LTA is also available for users but the recovery from tapes takes more time than from disks. NSDC is currently renewing its storage system. The new storage, planned to be active at the beginning of 2017, will have increased storage capacity and instead of tapes, all the data is stored to Solid State Disks (SSD) that provide faster data access.
Backup will be performed for the processing machines usually once per week or as agreed with the user. The services are targeted to run 99.9 % of time and the users are informed about the outages due to routine maintenance of the computer systems or in case of unexpected malfunctions. External users can access an on-line ticketing service that is used to request for support and report problems regarding the processing services. The used software is JIRA that is developed by Atlassian for issue and project tracking.
2. PRODUCT AND SERVICE GENERATION
Land cover change and phenology for boreal zone:
The following products are produced:
• Crop/vegetation classification
• Phenology Product
• Water Body Map
• Freeze/Thaw
Snow for boreal forest and mountain zone:
The following products were established within SEN3APP:
• Fractional Snow Cover extent for Northern Hemisphere from optical data
• Combined (2-layer, 25km) Northern Hemisphere snow cover product with optical FSC and PMW SWE –layers
• High resolution (5km) Pan-European SWE product (augmented using optical FSC data)
• Wet snow cover for the Alpine area from Sentinel-1 data
• Regional and Pan-European FSC product from synergistic Sentinel-3 SLSTR/OLCI data
• Extended Baltic Sea drainage basin direct broadcast FSC based on NPP VIIRS/Sentinel-3 SLSTR
Glaciers:
Glacier products from Sentinel data are generated on demand for particular areas of interest identified by users. Users show interest in the following glacier products, each generated from high resolution satellite data, as available from Sentinel-1 and Sentinel 2:
• Glacier outlines
• Snow and ice areas on glaciers
• Ice surface velocity maps
3. PRODUCT AND SERVICE VALIDATION
To use the satellite data based products the user needs naturally information about the product itself, but also about its uncertainty. The QA4EO (Quality Assurance for Earth Observations) initiative was established to provide guidelines how to report uncertainty of satellite (and other earth observation) data. Since the start of the project the QA4EO framework has not been actively updated. The framework still provides a good starting point to create documentation to accompany the data, which provides the user enough information to evaluate the suitability of the data to the use.
The QA4EO key principle is stated as: “Data and derived products shall have associated with them a fully traceable indicator of their quality.” The two core requirements in this statement are that the dataset has:
• Quality indicator: “A Quality Indicator (QI) shall provide sufficient information to allow all ausers to readily evaluate the “fitness for purpose” of the data or derived product.”
• Traceability: “A QI shall be based on a documented and quantifiable assessment of evidence demonstrating the level of traceability to internationally agreed (where possible SI) standards”
The SEN3APP validation documentation focuses on the quality assurance of the satellite data products. The documentation assumes that the user is familiar with the product content, although references are given to documentation describing the product generation. The validation is presented in a manner that allows the user, also to follow the protocol if the data or similar data is available. The main content of the validation documentation are:
• General information: Document ID, contact information etc.
• Main results of the validation, i.e. the accuracy of the product
• Datasets used
• Methods used
• Additional information on the validation
• References
4. DEMONSTRATION
Twelve products participated in the SEN3APP demonstration phase from July 2015 – May 2016:
1.Crop / vegetation classification (pilot)
2.Phenology Product (under development)
3.Fractional Snow Cover (FSC) for NH (operational)
4.High resolution Pan-European SWE product (operational)
5.Regional wet snow cover (pilot)
6.Regional FSC (operational)
7.Pan-European FSC (operational)
8.Extended Baltic Sea drainage basin direct broadcast FSC (operational)
9.Glacier outlines (on demand)
10.Glacier ice surface velocity (on demand)
11.Snow / ice areas on glaciers (on demand)
12.Lake ice extent (operational)
The products are generated by different SEN3APP partners. Several of the operational services were also continued after the official demonstration phase, and are still actively running.
The SEN3APP products and services are provided to users through two geoportals: the FMIARC GeoPortal, installed at FMI, and the CryoLand GeoPortal hosted by ENVEO. Both portals are accessible through the SEN3APP Portal (http://sen3app.fmi.fi/index.php?style=main&page=Products).
The following products are distributed through FMIARC GeoPortal, and are available using WCS and WMS.
• Fractional Snow Cover Extent for Northern Hemisphere from Optical Data
• High Resolution (5km) Pan-European SWE Product (Augmented Using Optical FSC)
• Extended Baltic Sea Drainage Basin Direct Broadcast FSC Based on NPP VIIRS/Sentinel-3 SLSTR
• Phenology
• Crop / Vegetation Classification
• Ice Velocity
Some of the products are distributed through the CryoLand GeoPortal, but sample products can also be accessed from the FMIARC GeoPortal:
• Regional Wet Snow Cover from Sentinel-1 Data
• Regional and Pan-European Fractional Snow Cover Product from Synergistic Sentinel-3 SLSTR/OLCI Data
For some key end users, products are tailored to their needs and provided directly via file transfer protocol (FTP), or sent via e-mail. This includes also products generated only on demand.
Potential Impact:
Promotional materials brochures were prepared. These brochures were distributed thorough meetings, workshop and conferences by the project partners. 2 Dissemination workshops in Helsinki, Finland and Vienna, Austria were successfully organized. 1st Dissemination workshop was held on November 19, 2016 at the FMI in Helsinki, Finland. The participants of workshop were mainly from Finland as focus on two end-users MAVI and FORTUM and the position of the Sodankyla NSDC in the project. However the whole workshop is broadcasted online on YouTube during the day. About 25–30 persons have participated the workshop in person and about 6 viewers in average have followed the broadcast with a peak of ten viewers.
2nd dissemination workshop was held April 18, 2016 at the ZAMG in Vienna, Austria. 33 people has participated to the workshop in person. The objectives of 2nd dissemination workshop were as follow
1. To disseminate products and services provided by SEN3APP project to wider end-user community possible future end-users which has scope of interest as same as SEN3APP project
2. To present demonstrated services and products of SEN3APP project
3. To give hands-on training to use services and products provided by SEN3APP project.
In order to achieve the objectives above listed we selected Vienna and timing during the same week EGU (egu.eu) general assembly was held. We co-organized with the COST Action ES1404 (harmosnow.eu) which has over 100 persons from 28 countries. The idea of co-organizing 2nd dissemination workshop with this COST Action and during EGU conference was to increase number of participation and interest international level.
We prepared training material, entitled, “EU FP7 SEN3APP Product Descriptions & Data Access” regarding SEN3APP project. In that training document, the products generated within the EU FP7 project SEN3APP are described and information on how to access the products is provided. Downloading of SEN3APP products via the FMIARC GeoPortal (http://saana.nsdc.fmi.fi/fmiarc-geoportal) and the CryoLand GeoPortal (http://neso1.cryoland.enveo.at/cryoclient/) is described.
SEN3APP products are disseminated from two interfaces, FMI Erdas Apollo system and CryoLand system. The FMI Erdas Apollo system is a data dissemination instance of the Finnish National Satellite Data Centre located in Sodankylä. The CryoLand system is powered by the EOxServer software which was developed within the EU FP7 project CryoLand (2011-2015). The CryoLand system is hosted by ENVEO.
Sodankylä site now hosts the Finnish Collaborative Ground Segment which is providing ESA Sentinel data. Sodankylä National Satellite Data Centre focuses on fast delivery remote sensing product generation for scientific and commercial uses. The data centre’s high performance computer arrays are capable of processing vast amounts of satellite data to value adding products to various users. The products can be delivered directly to customer or to large data archives for bulk use. Satellite data and products can be transferred to users with no delays using high speed optical fibre.
The FMI Erdas Apollo System is powered by Erdas Apollo. Erdas Apollo is an interoperable OGC/ISO-based commercial application that implements an out-of-the-box service-oriented architecture (SOA). It is implemented on a geospatial platform that enables high performance OGC web services from the FMIARC catalog. Users can interact with the FMIARC catalog directly using a Web Coverage Service (WCS) interface, a Web Map Service (WMS) interface, the Web Feature Service (WFS) interface, and the Web Registry Service (WRS) interface. All of these interfaces are compliant with the standards established by the Open Geospatial Consortium (OGC).
The CryoLand GeoPortal is the online data access point for snow and land ice services developed and established during the EU FP7 project CryoLand (No. 262925, 2011 - 2015). Products in the CryoLand system are generated and provided by different operators based on freely available satellite data, but for the end-user, all products are accessible via a centralized portal. The CryoLand system has a basic FTP Server hosted by ENVEO for storing most of the generated snow and land ice products. Some products are only made available via the FTP server of other product providers. These external FTP servers are also linked with the CryoLand system. The integration of the different products into the CryoLand system and the user interfaces are controlled via the Open Source EOxServer software (http://eoxserver.org) running on a virtual machine hosted by ENVEO.
The SEN3APP portal provides detailed information about how to download and access all the products that are provided within the SEN3App project. SEN3App products page is created under the SEN3App webpage (http://sen3app.fmi.fi/) and available from http://sen3app.fmi.fi?page=Products. In the page, the products generated within the project, are described and information on how to access the products is provided. For each product, a specific page, where the detailed information and previews can be found, is created and available from the links in the product page.
Preliminary stakeholders defined at the beginning of the project and can be listed as follow
• Agency for Rural Affairs (MAVI)
• Tyrol State Government (Hydrography and Hydrology)
• Central Institute for Meteorology and Geodynamics (ZAMG)
• MetOffice (United Kingdom)
• Fortum Energy
• Freshwater Centre of the Finnish Environment Institute
During the project time, especially during 2nd dissemination workshop possible stakeholders identified as follow:
• Institute of Hydrology, Slovak Academy of Sciences (SAS)
• Institute for applied mathematics (CNR-IAC), Italy
• Middle East Technical University (METU), Turkey
• University of Bern, Switzerland
• University of Oslo, Norway
• University of Leuven, Belgium
• University of Silesia, Poland
• University of Oulu, Finland
• alpS Centre for Climate Change Adaptation
• Estonian Environment Agency
• Slovak Hydrometeorological Institute
• Anadolu University, Eskisehir, Turkey
• WSL Institute for Snow and Avalanche Research SLF, Switzerland
• National Institute of Meteorology and Hydrology, Bulgaria
• Institute of Geography, Romanian Academy
• alpS Centre for Climate Change Adaptation
• Geological Survey of Spain (IGME)
The SEN3APP project is designed to lead to a set of long-term operational services. SEN3APP services will appear as a feasible extension of the current business of the project partners. The current business portfolio of preoperational and operational services delivered by project partners includes
• Snow cover mapping and monitoring for the hydropower industry (production and energy trading)
• Snow cover and glacier mapping and monitoring for hydropower planning and geotechnical work
• Snow cover monitoring for hydrological services
• Snow extent and water equivalent monitoring for ecosystem management
• Glacier monitoring for natural hazard assessment and mitigation (glacial lakes and glacier instabilities)
• Lake ice and river ice products including flood extent during ice jams
In SEN3APP, project partners gain further knowhow in (1) developing methods for processing and using of Sentinel data (2) providing services and products based on Sentinel data. This knowhow and extended service and product portfolio will support project partners’ current and future businesses. New applications are expected to open up through the SEN3APP developments.
Users will have access to various service layers. Free (at the point of access) layers would be supported by public organizations. Individual users or organizations requiring more detail would pay for a subscription service, while specialized companies (particularly those involved in the energy market and in water management) would pay more for tailored solutions.
In SEN3APP, we developed a system which is (a) capable of ingesting the extremely high-volume data of the Sentinel-1,-2 and -3 satellites provided by the Sodankylä ground segment interfaces (b) process the data to create value added product and services (c) integrate the products and services with data from other external sources, and offer both the raw and the processed data to users, via an accessible user interface and standard-based data interfaces, which provide intelligent search mechanisms and allow ordering of customized products and services. Sodankylä National Satellite Data Centre (NSDC) is in the centre of the SEN3APP system. Business scenarios of the developed system are planned to be
1. Customer oriented services
a. Customized high performance processing in processing clusters, Long-term archiving of the data and high speed data delivery.
b. Flexible product generation and hosting of external processing chains in virtual environments
c. Platform as a Service (Paas) and Infrastructure as a Service (IaaS)
d. Imaging and acquisition planning service
2. Direct broadcast and near real time services
a. Copernicus Collaborative Acquisition Station to downlink Sentinel-1 pass-through for quasi.real-time-applications like Baltic Sea Ice service
3. Applications
a. Terrestrial Cryosphere (e.g. Snow and ice maps generation)
b. Meteorological applications
c. Research and development
The activities and tasks of the Sodankylä National Satellite Data Centre (NSDC) is being developed in various projects such as SEN3APP. Overall goal is to develop the NSDC as the centre for satellite data and products in Finland. The centre that serves Sentinel data users with free Sentinel data and provides processing services and data products for national satellite data users and international partners developed during SEN3APP project.
The main barrier for establishing an operational service based on the pilot products developed in SEN3APP is the lacking funding for finishing the development work and establishing the operational service (i.e. setting up sustainable technology, work flow, administration, contracts etc.). The funding opportunities are constantly sought for. There are no other evident obstacles for the development of the operational service. The stakeholder, i.e. MAVI, has strong interest to develop the methodology; the expertise for service development exists and the technology is ready.
Critical factors for the service are:
• The renewal of the platform and process for subsidies control does not support the uptake of the remote sensing data.
• Timely and spatially extensive (covering entire Finland) delivery of the Sentinel-1 and -2 data.
• Contractual agreement on the division of responsibilities and sustained resources.
Efficient use of remote sensing is strongly connected to the reform in the subsidies control process. The advancement in remote sensing is therefore also dependent on the success of the other process.
Sentinel satellites are still young and e.g. the Sentinel-1 imaging program has gone through some changes in the first couple of years and is still under review. With the launch of Sentinel-1b the temporal coverage was improved. The current bottleneck for the commercialisation of Sentinel-1 based NRT services is the ground segment that struggles to provide the data on a reliable base. Use of the optical data is dependent also on the successful launch of the Sentinel-2B satellite as, especially in the northern latitudes, the constraints from the cloud cover are significantly reduced with two operating satellites. Agreement on by the contributing parties needs to be signed to ensure the sustainability of the service.
List of Websites:
http://sen3app.fmi.fi
Project Manager: Dr. Ali Nadir Arslan (ali.nadir.arslan@fmi.fi)
Financial Adminstrator: Mrs. Sari Jokiniemi (sari.jokiniemi@fmi.fi)
