Final Report Summary - CORE-CLIMAX (COordinating Earth observation data validation for RE-analysis for CLIMAte ServiceS)
Executive Summary:
The COordinating Earth observation data validation for RE-analysis for CLIMAte ServiceS (CORE-CLIMAX) project aimed to substantiate how Copernicus observations and products (from both in situ and space components) can contribute to climate change analyses, by establishing the extent to which observations complement existing Climate Data Records (CDRs).
The CORE-CLIMAX consortium comprises seven European institutions (University of Twente, Faculty for Geo-information Science and Earth Observation (ITC); European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT); European Centre for Medium-Range Weather Forecasts (ECMWF); German Weather Service (DWD); Flemish Institute for Technological Research (VITO); Finnish Meteorological Institute (FMI); Meteo-France (MTF)) and two International Cooperation Partner Countries institutions (Chinese Academy of Sciences, Institute of Tibetan Plateau Research (ITP); Chinese Academy of Sciences, Cold and Arid Regions Environmental and Engineering Research Institute (CAREERI)), each responsible for measuring, monitoring and disseminating various aspects of climate change related to their particular areas of expertise. The consortium aims to provide coordination for producing and validating Essential Climate Variables (ECVs) with the objective of reanalysing existing data to develop a clearer picture of global climate change and its implications.
The CORE-CLIMAX consortium is coordinated by the University of Twente. Together with EUMETSAT coordination is made with the European Space Agency’s Climate Change Initiative (ESA-CCI) through several key national ECV producers and climate change service providers (DWD, FMI, MTF) - along with the major reanalysis centre in Europe, ECMWF, and satellite data processing and validation centres (VITO). The two ICPC participants (ITP, CAREERI) are tasked with maintaining all of the activities at the major climate observation stations situated on the Tibetan plateau. As such these participants provide an essential role in coordinating the validation of ECVs and reanalysis results. Furthermore, these participants are also responsible for monitoring climate variations at the Tibetan Plateau (together with surrounding areas also known as the Third Pole Environment). With such a wide range of expertise spread throughout key geographical locations, CORE-CLIMAX was able to implement an internationally recognised mechanism for the coordination of ECV Climate Data Records (CDR) generation. The varying geographical locations of the consortium’s participants also facilitated links with space nations holding satellite data repositories.
To achieve the aim of the project, CORE-CLIMAX has assessed the European capability to provide essential climate data records, prepared the white book for structured process to derive CDRs, coordinated a harmonized approach for validating Essential Climate Variable (ECV) CDRs, identified the integration of CDRs into the reanalysis chain, and formulated a process to compare the results of different reanalysis techniques.
ECV capacity assessment
It is essential to assess the capability of the existing CDR development activities to ensure the prolonged generation of high-quality ECV CDRs. As such they can help to produce the underpinning science that supports decisions on mitigation and adaptation for a changing Earth climate. For an assessment of the existing capacity in the most objective way possible, we need tools that provide a basis for information preservation, expectations, and a metric for progress towards completeness. The System Maturity Matrix (SMM) is such an assessment tool, which assesses whether the CDR generation procedures have been compliant with best practices developed and accumulated by the scientific and engineering communities. Also, a prototype of Application Performance Metric (APM) was developed to meet the need of giving advice to data users which data record can be used for which application.
White book for the structured delivering of ECV CDRs
The white book describes the structured process to derive ECV data records, based on the architecture for climate monitoring from space. This architecture outlines major processes to generate long-term climate data records as well as data records which are updated with higher frequencies, the so-called interim climate data records. The architecture for climate monitoring contains the major activities describing the value adding chain for climate data down to the usage of the data in applications and subsequently for decision making. The CORE-CLIMAX process description focusses on the activity of creating and maintaining short/medium term and long term CDRs. Specifically, each step in generating a CDR was detailed using the IDEF0 (Icam DEFinition for Function Modeling, where 'ICAM' is an acronym for Integrated Computer Aided Manufacturing) standard.
Generic Validation Processes for ECV CDRs
To establish the traceability of a validation process, one is requested to document each step of the validation process. Starting from documenting how the reference dataset is generated, the generic process requires the assessment of the independence levels of the reference dataset. Furthermore, it is suggested to implement both the self-assessment and independent/external assessment of the CDR products and the validation process, and to assure the understanding of the data quality. For a complete validation process, one should also consider the consistency of the validated CDRs (e.g. compared with the physically inter-related variables). The last step is to sustain the established validation facilities and procedures. This final step means achieving an operational validation level, at which validation activities and data release are regularly implemented.
Reanalysis Feedback Mechanism to CDR updates
The advent of operational climate services increases the need for producers of CDRs to know about the needs of reanalysis, particularly regarding desirable CDR updates. The reanalysis needs can be effectively communicated to CDR producers via quality assessment procedures. It is through this communication (e.g. an iterative loop linking "Comparison & Assessment" and "Response by CDR provider") that (a) CDR weaknesses are reported, and (b) the CDR producer's response can be developed to a level at which CDR update plans can be formulated.
Reanalysis intercomparison
Reanalysis intercomparison activities are a key component of characterizing reanalysis uncertainties. As such, they yield information that assists users in deciding which reanalysis product might be most suitable for their particular application. The CORE-CLIMAX project developed a set of procedures for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs. To do so, five categories of comparisons are identified: 1) Descriptive product comparison; 2) Comparison with third party observation-based CDRs; 3) Intercomparison between different reanalyses; 4) thematic comparison (i.e. how well can the reanalysis products be applied to understand a particular problem); 5) Internal metrics comparison.
Recommendations
CORE-CLIMAX can be compared to ISO procedures where it comes to quality standards concerning CDR/ECV/reanalysis products, i.e. time series. As such the project constitutes an important building block for climate services.
With respect to the Copernicus Climate Change Service (C3S) the systematic application and further development of the CORE-CLIMAX system maturity matrix (SMM) and the spin-off Application Performance Metric (APM) were strongly endorsed by participants of the C3S Climate Data Store workshop in July 2015. However, currently no implementation plan, e.g. as part of the C3S Evaluation & Quality Control (EQC) pillar, exists for these functionalities. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation of C3S. Particularly, the SMM and APM can provide starting points for the EQC function of the C3S to assess the technical and scientific quality of the service including the value to users.
More training is required in order to capitalize on what has been achieved in CORE-CLIMAX project and more generally for the success of the C3S, which hints both at improving university education as well as other education activities (e.g. ESA training activities).
Currently many CDR’s are not ready yet for application, consistent high-resolution data is a challenge and more research is urgently needed. Linking ECV’s to Essential Water Variables is the next step from closing the gaps from CDRs to sectorial applications, the generation of different data sets should be related to high impact phenomena like severe droughts and flooding events.
At the closure review meeting at Research Executive Agency in Brussels, on 22 June 2015, the external reviewer commented that CORE-CLIMAX has produced results way beyond its original scope. It has set a new paradigm for data handling and data quality regarding the production of climate services. These issues are very sensitive and the best practice approach of the project has created community acceptance.
Project Context and Objectives:
CORE-CLIMAX coordinated the identification of available physical measurements, which can be reconciled with previously existing data records, to form long time series. It aimed to substantiate how Copernicus observations and products can contribute to climate change analyses, by establishing the extent to which Copernicus observations complement existing Climate Data Records (CDR).
With GCOS, Copernicus and ESA CCI projects, and EUMETSAT including its Satellite Application Facility (SAF) network, coordination took place with specific efforts to be undertaken by new Copernicus projects to further upgrade their product catalogues to include this climate relevant validation and information and lay the observational basis for service activities. CORE-CLIMAX identified the integration of ECVs into the reanalysis chain by proposing a feedback mechanism ensuring that the results of the re-analysis process get appropriately reflected into updates of the ECVs. Together with intercomparing different reanalyses, CORE-CLIMAX contributed to establish a European truly coupled gridded re-analysis which incorporates full exchanges and interactions between atmosphere, ocean, land, including the hydrological cycle.
Specific objectives:
1 Coordinate with Copernicus ongoing activities and contribute to the formulation of the Copernicus climate service theme (GCOS, Copernicus and climate change projects, ESA CCI projects, EUMETSAT including its SAF network and EUMETNET as part of the European Meteorological Infrastructure);
2 Propose a structured process for delivering ECVs through the stepped and quality controlled elaboration of CDR, the latter being derived from prioritisation of the most appropriate input data sets;
3 Propose a validation process aiming at qualifying the accuracy of the climate variables;
4 Propose a feedback mechanism ensuring that the results of the re-analysis process get appropriately reflected into updates of the CDR;
5 Propose a process to compare reanalyses
Project Results:
WP1 Project management
This workpackage concerned the project management and as such there are no Science and Technology results to report on.
WP2 European ECV capability and structured ECV process
The work in work package two was divided into two major tasks addressing a European assessment for the capacity to generate climate data records suitable for monitoring of climate variability and change and to document the process to current best knowledge on how climate data records are being generated.
2.1. Analysis of ECV capability and prioritisation for Copernicus climate service
Introduction
In support to the establishment of the Copernicus Climate Change Service and other international activities such as the establishment of the architecture for climate monitoring from space conducted by major space agencies one major objective of the CORE-CLIMAX project was to systematically assess the capacity of ongoing European activities in the area of generation and provision of climate data records. With respect to the Copernicus Climate Change Service also the role of in situ data and model-based reanalysis need to be considered. As the in-situ data sets are generally under national responsibility the CORE-CLIMAX assessment did not cover all in-situ data records generated by European providers. However, the inclusion of some of the best known in-situ data records in the assessment has demonstrated the value of the tools developed in CORE-CLIMAX and encourages more nationally managed data records to be included in a future assessment.
Capacity Assessment: Concept and Methods
For an assessment of the European capacity in the most objective way possible tools were developed that provided a basis for information preservation, expectations, and a metric for progress to completeness. The maturity matrix approach proposed by Bates and Privette (2012) offered a systematic mean to assess if the data record generation follows best practices in the areas science, information preservation and usage of the data. Some example uses of the matrix maturity are the assessments of data records developed in the NOAA Climate Data Record program and in the 2nd phase of SCOPE-CM to measure progress in the projects. For both these cases, maturity assessments were first done as self-assessments. External assessments could be done in a form of audit.
Basically, three different aspects of our capacity to generate data records need to be considered:
• Scientific, engineering and information preservation practices;
• Usage of products including feedback and update mechanisms;
• Quality of products with respect to applications.
Assessing if data record generation follows best practices provides an internal view on strengths and weaknesses of the processes to generate, preserve and improve climate data records for agencies and each individual data record provider. It also provides a general information to the community concerning the status of individual data records as well as collective information on the state of all existing records, highlighting areas for development and improvement. The assessment of quality of products is facilitating an external view on data records trying to answer the most important user question: Is the quality good enough for my application?
The CORE-CLIMAX project defined three major elements for its capacity assessment:
• Data record descriptions that contain technical specifications and also information on quality;
• A System Maturity Matrix (SMM) that evaluates if the production of a data record follows best practices for science, engineering, information preservation and facilitation of usage, and;
• An Application Performance Metric (APM) that attempts to evaluate the performance of an ECV CDR with respect to a specific application. To be able to apply the APM, user requirements for each application are needed to be compared to the actual technical specifications and validation results.
The three elements of the capacity assessment were designed to be independent of each other and represent means to support an assessment but do not provide the assessment results per se. The SMM is designed to be principally used without considering specific applications. With this the SMM does not depend on user requirements for specific applications and their change over time. In contrast the APM should facilitate a comparison of the real technical features of a data record and results of validation and other data quality assessment activities to user requirements for an application. It basically provides summary information on how close a specific data record is at fulfilling the requirements of a specific application. A prototype APM was developed during the capacity assessment workshop because the need of giving advice to data users which data record can be used for which application. This need is manifested for instance in the huge amount of information provided on validation of data records that is unlikely to be processed by institutions that want to use the data records. The APM is intended to support institutions in making choices among different existing data records without the need to assess the full documentation of all potential data records. However, it shall be noted that the APM is a new concept that was a spin-off result from the CORE-CLIMAX capacity assessment workshop. The development of an applicable tool was beyond the scope of the CORE-CLIMAX project. It is expected that the APM will be further adapted in the future to become fully functional. For details on data set description, as well as the SMM and APM tools please see Deliverable 2.25.
Capacity Assessment: Results
The CORE-CLIMAX capacity assessment experienced great community support (EUM SAF, ESA CCI, Copernicus Services, National weather services, EU projects) leading to a successful assessment. The assessment methodology outlined above was applied to 40+ data records including satellite, in situ and re-analysis data records. It can be noted that the self-assessments of the data records were performed with high honesty making the results reliable for further usage. During the review of the assessment report (Deliverable 2.25) some data providers updated the assigned maturity to lower values. Seeing other assessments obviously led to self-regulatory process to even more realistic results.
Assessment Workshop
The CORE-CLIMAX Essential Climate Variable (ECV) Capacity Assessment was initialised by a workshop that was held at EUMETSAT Head Quarters in Darmstadt on 21 – 23 January 2014. The workshop gathered about 40 participants representing all relevant European climate data record producers including EUMETSAT SAF network, ESA CCI projects, Copernicus Services, EU projects and EUMETSAT member states weather services as well as global stakeholders (EC, CGMS, CEOS, WMO). The workshop covered data records for Land, Ocean, and Atmosphere constructed from satellite and in situ data as well as reanalysis. The workshop participants were asked to do a self-assessment of their data records using the SMM prior to the workshop and there were 30 data sets (23 are based on satellite measurements, 6 are based on in situ measurements, and one is reanalysis based) with SMM results for analyses during the workshop.
The workshop participants endorsed the SMM as a useful tool for assessing completeness and identifying weaknesses in the developed systems which produce CDRs from satellites and in situ measurements and reanalyses. The presentations and breakout group discussions provided a common understanding of most of the elements in the SMM among different communities and programs and based on these understandings further improvements have been proposed for the SMM and its instruction manual. However, the workshop suggested that the process of introducing the concept of SMM in different communities needs to be open and inclusive. As the SSM is based on “best practices”, reviews and changes to the SMM are expected on longer time scales (~10 years), but it is expected to have a review and eventual change prior to an implementation to a major service such as the Copernicus Climate Change Service. It was recommended to use the SMM in the review process of existing CDR programs such as EUMETSAT-SAF network and ESA-CCI.
Assessment Completeness
The assessment tried to be as comprehensive as possible. Enough major European climate data providers participated in the assessment to allow conclusions about the applicability of the SMM for a capacity assessment. The assessment has been performed for satellite and in situ data records as well as weather prediction model-based reanalysis output. Among the satellite data providers are EUMETSAT including its Satellite Application Facility Network, the ESA Climate Change Initiative, Copernicus Atmosphere and Land Services.
It has to be emphasized that such an assessment had been done for the first time for in situ data records and the great response from the dataset developers is highly appreciated. Major European players such as Met Office Hadley Centre being responsible for the Hadley Centre Observational data sets (HadObs), Deutscher Wetterdienst having international responsibility for the Global Precipitation Climatology Centre (GPCC), the two leading the European Climate Assessment & Dataset (ECA&D) project, and the Alfred Wegener Institute hosting the Baseline Surface Radiation Network archive participated in the assessment. Their inputs to the fruitful discussions have very much contributed towards adapting the initially satellite oriented SMM to a matrix that is more generally applicable.
There is only one global reanalysis that is being produced in Europe (ERA Interim) and that has participated in the assessment. It is certainly possible to extend the assessment to regional reanalyses such as from the EU FP-7 UERRA project in the future. It is suggested to perform SMM assessments for such datasets in the framework of the Copernicus Climate Change Service.
Although the assessment was done in a self-assessment mode, an audit-type assessment for some of the randomly picked datasets revealed that most of the data providers have been very honest in assigning SMM scores to their products. This provides an overall large credibility of the self-assessment results presented here.
Value and Potential Usage
The value of the self-assessment has several aspects. Firstly, it provides a consistent view on strengths and weaknesses of the process of generating, preserving and improving CDRs to each individual CDR producer, agencies and the EC. In particular, the assessment fulfilled its promise to be useful for the data providers by providing guidance to further development of their data products. The assessment made some data providers for the first time thinking about issues such as software maintenance which are a cost sensitive long-term item for an operational service, in particular for data records based on satellite data. Data providers are encouraged to repeat the self-assessment periodically, e.g. annually, that enables progress monitoring of a guided development.
Secondly, the presented assessment provides detailed information on the status of of European activities to provide climate data records for GCOS ECVs. It may provide guidance for climate services where eventually to concentrate their investments for climate data record generation activities. For instance, data records showing maturity levels 3 and 4 (forming the initial operations capability) in many categories are certainly those that have good chance to be developed into full operations capability within the next 3-5 years. Others showing maturity levels 1 and 2 (being at research level) may be invested in but rather for doing more research and engineering development than transfer to operational services.
Thirdly, the assessment provides information to the general user community of CDRs, including science and services, on the status of individual records and the collective state of all records. It also provides this information for the first time across different observing systems, i.e. users can compare between satellite and in situ data sets on the different aspects covered by the maturity assessment. By this transparency and openness towards the user of CDRs has been significantly enhanced by the CORE-CLIMAX project.
Fourthly, the collective effort of the assessment to compile consistent descriptions of the participating data records has better facilitated Europe’s contributions to international activities on the evaluation of climate models using observations such as Obs4Mips that require standardised data record descriptions.
2.2. Preparation of a white book for structured process to derive ECV data records
This task covers a description of the structured process to derive ECV data records. The starting point for this description is the architecture for climate monitoring from space that outlines major processes to generate long term climate data records but also data records which are updated with higher timeliness as the so called interim climate data records.
Figure 1: Decomposition of the four pillar of the architecture for climate monitoring from space into individual activities.
Technically the structured description as delivered in D2.26 uses the IDEF0 standard (Icam DEFinition for Function Modeling, where 'ICAM' is an acronym for Integrated Computer Aided Manufacturing) to create flow diagrams that go deeper into details with each level of the considered process.
Figure 1 describes the top level adapted from the architecture for climate monitoring from space that contains the major activities describing the value adding chain for climate data down to the usage of the data in applications and subsequently for decision making. The CORE-CLIMAX process description concentrates on the elements A2 and A3 in Figure 1 that describe the production of data records. As an example Figure 2 shows the detailed description of the activities and outputs of the A3 box of Figure 1. The Deliverable 2.26 contains for each of the sub boxes shown in Figure 2 further descriptions.
Figure 2: Schematic representation of the process to generate a long climate data record.
In addition to the data record generation elements also a more generic description of the associated life cycle including all necessary review processes became part of Deliverable 2.26. Figure 3 shows the top level life cycle diagram for illustration. The life cycle has as major input user requirements as for instance provided by the Global Climate Observing System (GCOS). The requirements are analysed and converted into so called Product Requirements which represent a development target for the CDR. Such a step is necessary because user requirements can be apparently unreachable in particular if historic instruments are being used. By developing product requirements related to specific applications historic data are put into the right perspective.
The life cycle also demonstrates that each step of the generation of a climate data record is finalised by a formal review step that documents the status of the development and prevents any development and/or production of data on an unsound basis of methodology or data. The process for the final data record release review is finally sketched in Figure 4 that expands the process behind the L5 box in Figure 3.
Figure 3: Schematic representation of the life cycle for the development of Climate Data Records from initial user requirements to the final CDR.
Figure 4: Schematic representation of the Data Release Review that precedes the publication of climate data records.
WP3 Validation process
The generation of ECVs needs to put strong emphasis on the generation of fully described, error-characterized and consistent satellite-based ECV products. For example, generation of many ECVs (such as in the ESA CCI projects) requires ancillary information about the state of the atmosphere (e.g. cloud screening for SST, atmospheric correction for space-borne altimeters). As such, the consistency between the various ECV products (e.g. cloud flags in one ECV and non-flag in another one) extends to ensuring consistency in the approaches of CDR generation. The validation process as the integral components of the CDR processing chain, is a critical step to ensure consistency. In WP3, there are two main tasks:
• Task 3.1. Coordinating a harmonized approach for validating each ECV/CDR
• Task 3.2. Analysis of ECV/CDR validation network and strategy
In the following, the main science and technology results from these two tasks will be briefly introduced in terms of the three deliverables:
• Deliverable 3.1. Protocol for verifying, monitoring, calibrating and validating FCDRs and TCDRs
• Deliverable 3.2. Generic list for validation strategy for CDRs/ECVs
• Deliverable 3.3. Assessment report on consistency of the CDRs/ECVs
3.1. Protocol for verifying, monitoring, calibrating and validating FCDRs and TCDRs of the ECV CDRs
The need for minimal uncertainty in climate monitoring, together with the need to combine data from a variety of sources (space and in-situ), and emerging products with data assimilation, have placed “traceability” and its quantification at the top of the agenda for climate monitoring (Dowell et al 2013). Traceability is defined as the property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations/validations each contributing to the measurement uncertainty. It implies that a reference standard needs to be established. In CORE-CLIMAX, the System Maturity Matrix proposed in WP2 has included this in the uncertainty characterization sub-matrix. In addition, quality indicator has to be determined. The quality indicator (QI) is a means of providing a user of data or derived product with sufficient information to assess its suitability for a particular application (i.e. fitness for purpose). In CORE-CLIMAX, the Application Performance Matrix proposed in WP2 can be utilized as such indicator. This information should be based on a quantitative assessment of its traceability to an agreed reference or measurement standard, but can be presented as numeric or a text descriptor, providing the quantitative linkage is defined.
As the purpose of calibration and validation is to provide confidence in the quality of FCDRs/TCDRs and their uncertainties, it is important to provide sufficient information on how the Cal/Val activities are carried out, at each stage of the data process chain. It allows all end-users to evaluate a data product’s suitability for their particular application (e.g. ‘fitness for purpose’). To achieve this purpose, it is suggested that the documentation, at each stage of the data process chain (e.g. Cal/Val in the case discussed here), should follow guidelines or best practices, which aims to achieve transparency of approach (e.g. traceability) and improve efficiency. This has been covered by the SMM with the “Metadata”, “User Documentation” and “Uncertainty Characterization” sub-matrices.
As the existing GCOS principles, requirements of the Climate Data Records from Environmental Satellites: Interim Report of the National Research Council NRC, QA4EO, CEOS-WGCV and similar efforts are aiming to provide common reference standards for CDR monitoring and generation in a community-desired-endorsed manner. It is important for CORE-CLIMAX to adhere to these common reference standards. Therefore, the Cal/Val protocols for FCDRs/TCDRs can be proposed as follow:
1. Generation of FCDRs should adhere to community-desired-endorsed principles, requirements and standards, and be assessed with SMM and APM;
2. Validation of TCDRs should adhere to community-desired-endorsed principles, requirements and standards (e.g. CEOS working group on Cal/Val-WGCV protocols), and be assessed with SMM and APM;
3. The Cal/Val activity under planning should be documented, adhering to community-desired-endorsed principles, requirements and standards (e.g. QA4EO, GCOS-153). The documented Cal/Val plan should be evaluated with the SMM sub-matrices to understand how the Cal/Val plan is traceable;
4. When the Cal/Val activities related to FCDRs/TCDRs are beyond the assessment of the calibration and validation thematic areas proposed in this protocol, feedbacks should be given.
3.2. Generic Validation Strategy for ECV CDRs
The Committee on Earth Observing Satellites (CEOS) working group for Cal/Val (WGCV) defines validation as the process of assessing, by independent means, the quality of the data products derived from satellite observations. This can be called product validation. The product validation ensures that the quality of the products is properly assessed, through quantification of the uncertainties in both the data itself and the measurement system deployed for generating the data. It includes a quantitative understanding and characterization of the measurement system and its bias in time and space. In this context, validation can be considered a process that encompasses the entire system, from sensor to product.
Figure 5 shows two typical validation concepts: the scaling method and the direct comparison method. The scaling method is shown in the left panel of Figure 5. The scaling method uses an intermediate Very High Resolution (VHR) satellite data layer (or airborne campaign data) to compare the ground measurements with products at coarser spatial resolution. This permits reducing the uncertainties and the difficulties during the integration of several punctual ground measurements over a common area (or an Elementary Surface Area, ESU) to be used for the validation of the product at a pixel level. This is valid especially for products around 100m of resolution or more, for which it is very difficult to integrate several measurements to reach an ESU of that dimension also taking into consideration the landscape heterogeneity. This is the case for most of the terrestrial ECV CDRs (e.g. land use, LAI).
The consolidated and qualified campaign data (yellow) boxes in Figure 5 indicate the elements needed for validation, including the satellite data, the ancillary/auxiliary data and ground (reference) measurements. The left (green) boxes represent the FCDR processing and TCDR generation (the retrieved quantity or the retrieval) while the right (blue) ones are the processing of campaign data to produce VHR reference validation layers (“true quantity”). In the FCDR processing, raw satellite measurements are geolocated and atmospherically corrected first when relevant and then homogenized and inter-calibrated, to generate calibrated radiances, backscatter of active instruments or radio occultation bending angles. Afterwards, by means of data assimilation or modeling (e.g. radiative transfer model or specific retrieval algorithms), TCDR products are retrieved. In the right (blue) boxes, level 1 data are used to derive level 2 data products at very high resolution. Afterwards, the ground measurements are processed and “transferred” directly to the level 2 data product to represent the validation layers. Finally in the bottom (red) box, the TCDR products are validated by the use of the previously generated VHR reference validation layers.
Figure 5: Validation of ECV CDRs as an expansion of the validation function “D” in Figure 1 showing two typical direct validation concepts, the scaling method (left) and the direct comparison method (right).
It is noted that the generations of both FCDRs and TCDRs are recursive because when improved information becomes available (e.g. better algorithms or improved calibration/validation information) the observations are re-processed to generate improved CDRs. As understanding of sensor calibration issues and as the modeling of the radiative transfer from the Earth and atmosphere improves, products can be generated via reprocessing.
The main advantage of the scaling scheme is the fact that these intermediate layers are very close in terms of quality and resolution to the ground measurements (since they are obtained by VHR data), thus the uncertainties due to the ESU integration can be reduced because now the integration is applied to surfaces at the same spatial scale as the satellite pixels. One should realize that the optimal ESU size is determined by the level of within ESU variability that can be tolerated by the validation protocol and the effort available to conduct measures. The size of ESU within a reference region can vary with various factors (e.g. surface condition, instrument field of view and spatial sampling design etc.) [Fernandes et al. 2014]. Nevertheless, as shown in the blue column of Figure 5, the level 2 data retrieval, the transfer processing, and the aggregation introduce uncertainties that have to be monitored.
In the direct comparison method, the ground measurements are directly compared with the TCDR product retrieved for standard processing. The elaboration needed is the processing/integration of ground measurements to generate ESU comparable with the TCDR product. This method is more “direct” with respect to the other, but the ground measurement processing for the ESU generation can introduce additional uncertainties that again have to be estimated and monitored. Moreover, this processing is applicable if the campaign ground measurements have followed a specific protocol and the area to be covered by the ESU is comparable in size with the product pixel.
From the description of validation concept for ECV CDRs, there are at least three components needed for implementing a validation study (i.e. the purple-red highlighted TCDR box in Figure 5): the validation requirements, the generated TCDR itself, and the established reference data. In addition, one needs to select methods to implement validation. Based on how different European initiatives approached the validation of CDR and ECV products on these aspects, a generic validation process is proposed, together with a quality indicator. The proposed generic validation process for CDRs/ECVs may include:
1. The generation of independent reference datasets;
2. Assessing independence levels of reference datasets (section 3.3);
3. Self-assessment;
4. Independent assessment;
5. External review/evaluation of self-assessment validation practices;
6. Assessing independence levels of point 4 and 5 (section 3.3);
7. Consistency check for inter-related CDRs/ECVs;
8. Sustaining established processes and methods.
The generic validation process was matched with the quality indicator of “Validation” and “Formal Validation Report” of system maturity matrix as shown in Table 1.
Table 1. Matching the generic validation process with the “Validation” and “Formal Validation Report”
3.3. Consistency validation of the ECV CDRs
The climate data store (CDS) for Copernicus Climate Change Services (CCCS) envisions that it shall include essential climate variables (ECVs), uncertainty estimates, reanalysis, multi-model data (e.g. seasonal forecasts and up-to-date climate projections), and in-situ and satellite data. Furthermore, CDS should contain only ‘climate compliant’ data, as defined by Evaluation and Quality Control working group of CCCS. This implicates that one may find certain ECV variable from different sources, and there is a need to do consistency check to determine which dataset are fit for certain particular purpose. On the other side, when there is a certain ECV variable under evaluation, one would like to collect as many as possible independent datasets, under the constraints that they are measuring the same thing, to do (inter)comparison, in order to understand comprehensively the associated uncertainty.
For the in-situ datasets, the homogeneity adjustment plays a critical role in producing long-term climate data. There are direct and indirect methodologies to do homogeneity detection and adjustment [Peterson et al. 1998]. However, it is still difficult to avoid all the inhomogeneity caused by, for example, changes in instrumentation, station moves, changes in the local environment (e.g. building construction), or the introduction of different observing practices like a new formula for calculating mean daily temperature or different observation times. The existence of independent datasets and data assimilation technique provide unprecedented opportunities to spot “bugs” in the in-situ datasets. Figure 6 shows the use of reanalysis data to detect erroneous in-situ measurement.
Figure 6: Example for applying reanalysis for historical quality control: observed daily 10m wind speed at the station Seehausen, Germany (top left) match well with the frequency distribution from the interpolated ERA 20 C wind fields (top right) in recent years whereas 1980-1985 (lower left) the observed station values (lower right) are found to be suspicious.
For the satellite, reanalysis, and multi-model datasets, the same principle of consistency validation as discussed above (e.g. collect as many as possible available independent datasets) is applicable. On the other hand, this principle needs to be constrained with the reference data selection procedure. The major purpose of the consistency validation is to identify what is missing in regards to following the generic validation process (e.g. validation gaps). To achieve this purpose, it is suggested to collect comprehensively the relevant datasets/documents produced by data providers, end users and any other third parties. And, consistency validation is a key for identifying such gaps.
WP4 Reanalysis feedback mechanism to CDR updates
4.1 “Assessing CDR data quality by comparison with existing reanalysis”
The deliverable for this Task, together with Task 4.2 was the document D4.42 “Design of support infrastructure for CDR quality assessment in a reanalysis environment” which was successfully completed by the team and accepted by the Commission. The design was informed by a number of working visits that enabled consultation on proposals and practical testing of infrastructure prototypes. Some of these are highlighted here:
Joey Comeaux from NCAR visited ECMWF to support work on D4.42. The findings were summarized in a visit report. Key results were as follows:
(1) The unique identification of observations is of utmost importance; this identification should flow, untouched, from the data provider to the reanalysis center to allow for iterative update (enriching the original observation archive from assessment in reanalysis environment).
(2) Unintended differences in formats and values in the data when transferred from a data provider to a reanalysis center, even slight, are important to spot and fix early on, before a whole archive has been transferred. To check that, a dedicated method may be needed - probably it could be inspired by standard quality control assurance practices (systematic checks coupled with random checks).
(3) The integration of metadata information with observation data is desirable, before ingest by the reanalysis center. Consequently, such integration should be done by the observation provider rather than the reanalysis center.
Furthermore, the visit by ECMWF to KNMI allowed to approach the WG-GRUAN (Global Reference Upper-Air Network) which held a meeting at KNMI that same week. A visit report was written up. The GRUAN is expected to grow and potentially become a prime provider of near-real-time CDR for upper-air observations in years to come. The finding of interest for D4.42 was that product developers within WG-GRUAN are requesting quantitative feedback on their observation quality, as can be provided by the reanalysis process.
ECMWF visit D5.3-1 determined that the ozonesonde data holdings of the World Ozone and Ultraviolet Data Centre (WOUDC) remain the prime candidate for assessment of atmospheric ozone in-situ data within a reanalysis context. The procedures explored by Joey Comeaux in prior CORE-CLIMAX visit D4.2-1 provide a useful template/starting point. It remains to be seen whether any adaptation is needed when considering ozone rather than temperature data.
ECMWF visit D4.1-3 for which a report was issued, determined that the wind energy community collects a large number of in situ measurements, which could be prime candidates for validation of reanalyses at the local scales. However, due to the economic model under which the wind energy community is placed (concessions and permits), the value of the collected measurement data represents the first return of a concession/permit investment, and consequently these observational data are not exchanged between competitors, even for a fee. Consequently, in a climate service view, this community cannot be expected to contribute any observations at this point.
Visit D4.2-2 was planned with a provider of surface in situ observations, but in May 2013 we heard of recent developments owing to an imminent reprocessing of imagery data from the GOES series satellites. Data from GOES represented so far a glaring omission, such long time-series being not reprocessed, thus failing to realize the value of these observations for climate applications. We thus decided to invite a representative of the center who will do such reprocessing to discuss how to best assist in their efforts and understand how their work fits in a procedural approach to reprocess observations for reanalysis applications.
The resulting visit D4.2-2 to ECMWF by Steve Wanzong from CIMSS, from which a visit report was issued, determined that a satellite data provider who reprocesses imagery data into atmospheric motion vector (AMV) winds would have great interest into improving the feedback received from users, ideally by users making available their own quantitative comparisons of reprocessed data with their own application.
In the case of the reanalysis application, this means opening up the observation feedback archive and populating it with AMV products and corresponding reanalysis feedback as they are received.
ECMWF looked for a way to test the ideas to be presented in deliverable D4.42. To this end, ECMWF prepared a proposal for the Climate Monitoring Satellite Application Facility (CM-SAF) for a visiting scientist to compare with existing reanalyses the quality controlled SSM/T-2 data set acquired within the previous reporting period, from EUMETSAT, within Task 4.2. This proposal involved Paul Poli and Viju John as lead co-investigators, both from Core-Climax (note, without seeking funding for either one), and would fund a visiting scientist to compare SSM/T-2 brightness temperature data with existing reanalyses. This proposal was approved and represented an important test case of the procedure (deliverable D4.42). The activity is described further below and also resulted in the report “Pre-assimilation feedback on a Fundamental Climate Data Record of brightness temperatures from Special Sensor Microwave Imagers: A step towards MIPs4Obs?” which was approved for publication as ERA Report Series 19. It is co-authored by ECMWF, DWD (Satellite-based climate monitoring), and EUMETSAT. The document contains a feedback step from the FCDR producer, the CM-SAF in this case.
In a move toward comparison of reanalysis/ ground station data, so as to determine at which temporal-spatial scales the reanalysis data add useful information, DWD started analyzing frequency distributions and scales of variability in independent ground station data and reanalysis data.
In preparation for visiting the Global Runoff Data Centre (D4.2-4) the Global Runoff Data Base (GRDB) product provided by the Global Runoff Data Centre was investigated for the procedure intended for D4.42 (assessment by comparison - not assimilation). One issue that arose was that we were not able to establish a simple mechanism to identify retrospective updates of the database values, either by version numbering or by other means - the values obtained by different users depend on the date/time of the actual downloads. This may be acceptable for research-type exercises of limited extent requiring one-off downloads, but presents difficulties for coordinated inter-comparisons and for applications requiring periodic incremental downloads (e.g. near-real-time reanalyses). To explore these issues further, the Core-Climax System Maturity Matrix has been applied to the GRDB - done jointly with a colleague from the University of Uppsala.
A historical dataset has been evaluated following the procedure intended for D4.42 (assessment by comparison - not assimilation). It is a dataset of radiances collected by the IRIS instrument onboard the Nimbus-4 satellite in 1970-1971.
4.2: “Assessing CDR data quality by assimilation into reanalysis”
Task 4.2 complemented the activities of Task 4.1 in producing Deliverable D4.42. In support of this, EUMETSAT worked on 2 sets of input data for the planned analysis, drawing on activities performed outside the CORE-CLIMAX project:
• A quality controlled SSM/T-2 data;
• An analysis on homogenising Meteosat First Generation (MFG) infrared (IR) and water vapor (WV) channels based on High-resolution Infrared Sounder (HIRS) measurements
both served as input to Task 4.2. In support of this task, deliverable D4.42 EUMETSAT and ECMWF worked to transfer by FTP a quality controlled SSM/T-2 data set generated under the auspices of EU FP7 ERA-CLIM into the ECMWF File Storage system. This dataset covers approximately 35 satellite-years, DMSP F-11 (1993-2000), DMSP F-12 (1994-2002), DMSP F-14 (1997-2006), and DMSP F-15 (2000-2008). The dataset thus exchanged was not to be assimilated into the next reanalysis production but represented an excellent test case for a procedure whereby a dataset was assessed for its quality in an experimental reanalysis.
Deliverable D4.42 was drafted in several iterations, incorporating new insights from the visits in the order they were conducted. Particularly valuable was the experience with observational CDRs provided by EUMETSAT from the SSMI series of instruments (a Fundamental CDR, FCDR, DOI-referenced, CM-SAF product) and from AMV reprocessing activities (a Thematic CDR, TCDR, derived from European operational satellites). Initial results of the comparison of the FCDR with reanalyses were presented at the 6th EUMETSAT Working Group on Data Record Generation held in Darmstadt, Germany, on 12-13 June 2014. This feedback to the CDR team was highly appreciated and the document (in preparation at the time) formalized the process.
Of the remaining visits formulated for WP4, D4.1-2 D4.2-3 D4.2-6 D4.2-7 D4.3-2 and D4.3-4 were conducted by a Core-Climax Coordination Meeting Towards a Global Archive of Historical In Situ Snow Data, held at ECMWF on 17-19 November 2014. Among those invited were Eric Brun (Meteo-France), Matthew Menne (NOAA NCDC), Olga Bulygina (World Data Centre for Meteorology from the Russian Federation), Bob Dattore (NCAR), and Ross Brown (Environment Canada). The Chinese Meteorological Administration were also invited and investigated whether they could send someone to this meeting (on the understanding that a separate bi-lateral visit could be arranged if necessary). Having made originally provisions in the project plan for both visits and hosting events, we adjusted as anticipated in the DoW to whichever proved most practical for each. At the time, out of all the 11 exchanges conducted or committed, only 3 consisted in hosting, the other 8 had been visits. So hosting all the 8 remaining visits was a fair adjustment.
During initial consultation with the various parties, the invitation process revealed that some large institutions see the setting up of a new, uni-variate archive (here, of snow depth or snow fall, and snow water equivalent) as a threat to their large multi-variate archives. The fear is that large archives with aligned records would be gradually disintegrated into smaller entities, each covering only isolated aspects of the observational problem. This concern is fully legitimate as eventually reanalysis attempts to solve the representation of past time-series as a fully multi-variate problem. To win participation of the sceptics to this initiative we indicated that 1) thematic problems (snow here) need to be solved thematically (i.e. by bringing over snow experts around the discussion table), 2) it is unlikely that any of the existing archives covers all the snow data found in the others (this was also agreed by everyone), and 3) with today's technologies there are ways to avoid physically copying large data amounts and forfeiting data ownership, as long as the data are uniquely identified in each collection and a central catalogue can be set-up to allow finding/discovering data (which could then still be served by the relevant, currently existing archive).
Following the extensive preparations, including adjustments to the candidate participants contacted above, ECMWF hosted the “Core-Climax Coordination Meeting Towards a Global Archive of Historical In Situ Snow Data” organized in support of WP4, from 17 to 19 November 2014. This meeting was supported by Core-Climax visits D4.1-2 (Expert invited: Eric Brun, from Météo-France), D4.2-3 (Expert invited: Matthew Menne, from the U.S. National Oceanic and Atmospheric Administration National Climatic Data Center), D4.2-7 (Expert invited: Steve Worley from the U.S. National Center for Atmospheric Research, also representing the National Snow and Ice Data Center), D4.3-2 (Expert invited: Yu Yu from China Meteorological Administration), D4.3-4 (Expert invited: Ross Brown from Environment Canada), and D4.2-6 (Expert invited: Olga Bulygina, from the World Data Centre for Meteorology Federal Service for Hydrometeorology and Environmental Monitoring of Russia). One visit which had been scheduled was cancelled at the last minute (D4.2-6) with the Expert failing to get a visa in time for travel, in spite of our efforts to send all the paperwork more than 2 months before the meeting (ECMWF sent the invitation letter on 10 September 2014 for a meeting starting on 17 November 2014); the Expert sent presentations that were given at the meeting on her behalf by a colleague from the same institution, Alexander Sterin.
The meeting attracted a large audience, including from the project itself (FMI) but also from other interested entities such as the Global Precipitation Climatology Centre. The meeting ran smoothly over the course of 3 days. All presentations as well as the meeting report have been posted by ITC on the Core-Climax project website, at http://www.coreclimax.eu/?q=Snow. The report starts by a one-page executive summary of ‘key points’, for those interested in fast reading of the main outcomes.
4.3 “Provision of state-of-the-art reanalysis gridded data for use as ancillary information in CDR generation”
The deliverable for Task 4.3 was the document D4.41 “Procedure for feeding back improved ancillary data to assist CDR updates” which underwent a number of drafts. The procedures developed during the Task and were informed by a number of working visits that enabled consultation on proposals and practical testing of infrastructure prototypes.
ECMWF visited KNMI who uses global reanalysis data as input to regional reanalysis activities (process to derive CDR at regional resolution). A visit report document was issued. The key findings were that regional-scale CDR activities would benefit from the following information from global reanalyses:
• information about random and systematic uncertainties,
• ensemble of reanalysis realizations,
• information about atmospheric composition (~40-year timespan) and potentially atmosphere-ocean coupling
• regular presence of mechanisms to monitor/improve co-ordination
ECMWF visit D4.1-3 was also conducted, from which a visit report was issued. Addressing about 200 attendees Paul Poli delivered the opening presentation of the European Wind Energy Association (EWEA) Technical Workshop Resource Assessment 2013 in Dublin, and presented how the global reanalyses are conducted. In return, he collected, from panel and offline discussions, the following key information points, relevant to realize user needs and understand the value of global reanalyses for producers of kilometre-scale products from the wind energy community:
• The wind energy community are advanced users of observational and climate reanalysis data records.
• Members of this community generate kilometre-scale local products over 10-20 years from global reanalyses to cover both climate and short time-scales, downscaling with meso-scale and micro-scale models.
• They would appreciate improvements in the horizontal and temporal resolution of global reanalysis products (like the current MERRA reanalysis 2D fields: hourly).
• The wind community would realize cost savings if current reanalyses of the last decades could be delivered with timeliness shorter than the current 2 to 3 month delay.
• A global reanalysis at a resolution of 10~km or better would better serve the wind energy community.
• The wind community experience in using reanalysis and observational records is distributed in dozens of consultancy companies, with expertise in data mining, quality control of both observational and reanalysis time-series, and optimal combination or merging of observations with climate reanalysis data records.
Additional information was gathered from ECMWF visit D5.3-1 regarding how regional-scale CDR activities would benefit from the following development of global reanalyses. In the view of a climate service processing chain, their needs are:
• access to observational quality control information (e.g. first-guess checks, bias corrections), as being developed in ECMWF’s Observation Feedback Archive,
• mechanisms to exchange knowledge, especially about observations,
• ensembles of boundary conditions from global reanalysis datasets.
The product of Task 4.3 deliverable D4.41 was finalized by ECMWF, drawing on findings of visits D4.1-2 and D4.1-3 as well as direct exchanges with selected projects from the European Space Agency (ESA) Climate Change Initiative (CCI).
4.4: “Guidance of CDR updates”
The deliverable for Task 4.4 was the document D4.43 which underwent a number of drafts and was delivered on 22 December 2014. The guidance developed during the Task and was informed by a number of working visits that enabled consultation on proposals and refinement of the ideas. Some of these are highlighted here:
Visit D4.3-6 to ECMWF by Dave Santek from CIMSS was carried out, and a comprehensive report summarizing the visit findings was drafted. Representatives from EUMETSAT were also able to participate (under funding external to WP4) to enhance co-ordination with analogous European activities.
The visit determined that two important satellite instrument datasets have now matured:
• 30 years of imagery data from the AVHRR instruments have been reprocessed into atmospheric motion vector (AMV) wind products by CIMSS and EUMETSAT. (This covers 11 satellites, from NOAA-7 in 1982 to NOAA-18 in 2011, as well as MetOp-A in 2007-2012)
• EUMETSAT has reprocessed 20 years of Meteosat First Generation AMVs for the period 1982-2001 and reprocessing Meteosat Second Generation AMV covering 2004-date is ongoing
The visit also determined that several programmatic issues need to be tackled to improve the reprocessing of imagery data into atmospheric motion vector (AMV) winds and serve the needs of reanalyses:
• Co-ordinated effort to better understand the implications of differences between AMV reprocessing methods at different centers
• A unified global AMV processing code, supporting both real-time and historical processing for all satellites, would allow consistency in regional datasets, applying algorithms of choice whenever needed, and possibly realizing synergies and cost savings by consolidating expertise in developments. Co-ordinated scientific/technical development would be required.
• Reprocessing of additional long-term imagery datasets could be extremely valuable to cover recent satellite missions which do not include AMV processing as a main mission obligation (e.g. MODIS), and to extend backward the start of the “satellite era” in reanalyses (e.g. GOES-1 to -7, Nimbus)
Visit D4.3-6 was planned at NOAA Headquarters, with a view to discuss the issue of GOES reprocessing, which had never been conducted so far. However, in May 2013 we heard that CIMSS received internal funding to reprocess NOAA GOES imagery data. We thus coupled this visit with that of a scientific expert in D4.2-2 from the same institution to discuss all aspects of this reprocessing.
In support of deliverable D4.43 Visit D4.3-3 was carried out. It has enabled to better understand how to feed information about upper-air observation quality back to the data provider who assembles these observations from paper and microfilm records.
The report from visit D4.3-3 was drafted and is available from the Core-Climax co-ordinating team. The main conclusions are that the experience gained at a non-reanalysis producer center (U. Vienna) and at ECMWF using the in situ CDR (CHUAN V1.7) and its recent ERA-CLIM extension need consolidation before feeding back to the data producer. Information on the types and makes of historical radiosonde ascents found in a metadatabase (the so-called VAPOR database) will be fed back to the data producer so as to aid future users of the CHUAN CDR, e.g. to validate break-point detection in the CHUAN upper-air network, and could also potentially be used to support a self-adaptive bias correction scheme in the forthcoming new reanalysis at ECMWF. Generally this illustrates the importance of “detective work” on the metadata to improve the quality of a CDR.
Work on preparing large swaths of sample observation feedback from reanalysis was progressed to support the procedure delivered in D4.43.
ECMWF and EUMETSAT exchanged a document listing the satellite CDRs to be used in future reanalyses.
The ECMWF visit D4.3-5 to NASA Headquarters produced a report. The main conclusions are that Obs4MIPs cannot grow into a full-fledge super-observation archive because of its founding principles, which are to bring observations to the modellers in the shape they are used to, namely gridded products. This precludes extending Obs4MIPs to in situ sparse observations or observation-only data types such as brightness temperatures. A logical reading of this limitation would be to start thinking of a reverse approach to map systematically climate and reanalysis model estimates into observation space.
WP5 Intercomparing reanalysis results
In WP5 a worldwide survey about user needs in private and public sector both up- and downstream in regard to reanalyses and climate services was made. Practical means for intercomparing reanalyses, using feedback data and assessing black list were identified in wide international collaborations. WP5 produced 3 project reports, 4 posters, 6 abstracts and 6 submitted manuscripts. Two of the manuscripts are now published as peer-reviewed articles, two are under revision and two are rejected. WP5 ongoing work and results were presented in workshops, conferences and meetings worldwide. WP 5 is expected to have a high impact on future operational climate change services.
5.1 “Summarizing uncertainties related to reanalyses based on a survey on user requirements supported by peer reviewed literature”
Science: Data from the survey: 2578 respondents answered the check-box questions and several hundred also wrote free form responses. Publications: one report, one poster, two abstracts, one rejected manuscript (Gregow et al. 2013, Figure 7 shows an example), one published manuscript (Gregow et al. 2015a), one submitted manuscript (Gregow et al. 2015 b). Presentations: several oral presentations.
Task 5.1 focused on identifying uncertainties in reanalysis products and how these affect climate services/ scientific community. To be able to build a survey, an investigation on earlier surveys concerning reanalyses or climate services and/or research were made.
Figure 7: Correlation of ERA-Interim (1979-2012) wind gust product to sum of geostrophic and ageostrophic isallobaric wind (SWIND) from the paper Gregow et al. 2013 that was rejected. The figure is produced by Pauli Jokinen.
In Task 5.1 an online survey entitled “Reanalysis User and Application Survey” was made. The final version of the survey included only 11 sections to be answered. These sections resulted in around 360 multiple-choice questions (Fig. 8).
Figure 8: Description of the survey sections and emphasis of these. This figure is from the manuscript submitted by Gregow et al. to BAMS in June 2015. That manuscript focuses on the boxes shaded by light blue color.
The survey was distributed by WMO globally to regional offices, by help of NOAA and Paul Poli it was posted on the web page reanalysis.org. Additionally FMI, ECMWF and DWD distributed the survey nationally through their contacts, e.g. the Copernicus user forum, and for the known stakeholders. The links were posted on reanalysis.org ecmwf.int and ewea.org. The first analysis results of the questionnaire were distributed to the partners on 12th of February 2014.
An analysis of the survey and a literature review of reanalyses were compiled. When the analysis and the report D5.52 entitled “Reanalyses and user needs with respect to Climate Change Services” was in its final form by 31.3.2014 and was immediately sent to ECMWF by personal request from ECMWF director general Alan Thorpe to Hilppa Gregow on the Copernicus workshop 17.-18.2.2014.
Foreground of the survey material is shared among FMI, DWD and ECMWF. It is an excel file including all responses that can be analyzed in any statistical way. We have written a notification concerning Intellectual Property Rights in the excel file.
5.2: “Collecting and synthesizing information from current reanalyses”
Science: Publications: One report, two abstracts, two posters, Kaiser-Weiss et al. 2015 (accepted).
Task 5.2 focused on the process of building a database of diagnostics for reanalyses to allow quantitative comparison between reanalyses, as well as between reanalyses and CDR and other Copernicus observations. The aim was to yield information that assists users in deciding which reanalysis product might be most suitable for their particular application. With increased resolution of the reanalyses and applications, there is a growing need for evaluation with observations.
In Task 5.2 ECMWF and DWD synthesized the information from current reanalyses and investigated how the reanalyses can best be compared. The analysis of feedback statistics was identified as the appropriate method to identify the temporal-spatial scales for confident usage, as well as for comparison between different analyses. A methodological overview was carried out over formal processes for intercomparison of regional and global reanalysis. A pilot study was performed to demonstrate feasibility and potential value of feedback statistics. The latter yield the fit of reanalyses to assimilated observations, and are thus suitable as a means for comparing regional with global reanalysis. These findings were reported in D5.53 “Procedure for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs” This report presents a set of procedures for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs. Five categories of comparisons were identified. These were accompanied by two complexity ratings. The first rates the complexity of conducting the procedure (simple, moderate, difficult), and the second rates the complexity of interpreting the results (simple, moderate, difficult):
1. descriptive product comparison (simple to conduct, simple to interpret)
2. comparison with third party observation-based CDRs (moderate to conduct, moderate to interpret)
3. inter-comparison between different reanalyses (moderate to conduct, moderate/difficult to interpret)
4. thematic comparison (difficult to conduct, difficult to interpret)
5. internal metrics comparison (difficult to conduct, moderate to interpret)
5.3: “Summarizing Tasks 5.1 and Tasks 5.2 and highlighting the uncertainties and gaps that need addressing to deliver climate services”
Science: Publications: one report, one poster, four abstracts, one accepted manuscript (Jokinen et al. 2015), one submitted manuscript (Gregow et al. 2015 b). Presentations: several oral presentations.
In preparation of a deeper analysis of the survey a new manuscript (Gregow at al. 2015b) focusing on future climate services as supported by reanalyses was started (in June 2014). At that time also the groups Water, Energy, Agriculture and Forests were chosen to be examined more closely. The 2578 respondents were mostly users of global reanalyses and in particular ECMWF, NCEP and NASA reanalyses. An innovative procedure was devised to organize users’ requests into classes of priorities of similar significance (Table 2a&b). The groups (excluding energy) emphasized the importance of activities such as seasonal forecasting and verification, applied weather and climate research for impact assessment and/or statistical impact analyses for improving weather warnings and their criteria to be part of the future climate services (Table 2b).
Table 2a. Statistical significant differences for average scores for different statements about reanalysis data between two groups of respondents, namely respondents having vs. not having fresh water resources and management (WAT), agriculture and food production (AGR), forests (FOR), or energy (ENE) as a field or a subject of their work. For details, see the Gregow et al. (2015b, submitted to BAMS)
If a statement is agreed more strongly by the subgroup members than by the outsiders, then the following notation is used: ^ (p<0.05) ^^ (p>0.01) ^^^ (p<0.001). If it is disagreed more strongly, then v (p<0.05) vv (p>0.01) vvv (p<0.001).
Table 2b. Statistical significant differences for average scores for different suggestions for future climate service tasks or activities between two groups of respondents, namely respondents having vs. not having fresh water resources and management (WAT), agriculture and food production (AGR), forests (FOR), or energy (ENE) as a field or a subject of their work. For details, see the footnotes. (Copy from Gregow et al. 2015b submitted to BAMS).
If a statement is agreed more strongly by the subgroup members than by the outsiders, then the following notation is used: ^ (p<0.05) ^^ (p>0.01) ^^^ (p<0.001). If it is disagreed more strongly, then v (p<0.05) vv (p>0.01) vvv (p<0.001).
Next we show examples of use rate of data (Tables 3 and 4) that are part of the supplementary material in the submitted manuscript by Gregow et al. 2015b.
Table 3. Applications and methods used among sectors WAT (water), AGR (agriculture), FOR (forests), ENE (energy) and ALL (all 28 sectors). “R” indicates reanalysis data and “G” gridded in situ data. Color codes are analysed separately for ALL and each field of work for “R” and “G”. Green indicates the highest percentage (67-100%) of responses, yellow indicates the medium share (34-66%) and red the lowest share (1-33%) of responses within each main field of work. (from Gregow et al. 2015b submitted to BAMS)
Table 4. Essential climate variable (ECV) distribution concerning use of atmospheric and terrestrial reanalysis data (RD) and other data (OD) (in situ, radar, satellite). For the color codes see Table 3. (from Gregow et al. 2015b submitted to BAMS)
In addition to the manuscript that was prepared a synthesis report summarizing Tasks 5.1 and 5.2 and thus Deliverables D5.51-D5.53 was made. This task was coordinated with WP3 (task 3.2: Analysis of ECV/CDR validation network and strategy) where the consistency of gridded data was examined (FMI also involved). ECMWF visit D5.3-1 determined that the ozone sonde data holdings of the World Ozone and Ultraviolet Data Centre (WOUDC) remain the prime candidate for assessment of atmospheric ozone in-situ data within a reanalysis context.
Additional information was gathered from ECMWF visit D5.3-1 regarding how regional-scale CDR activities would benefit from the following development of global reanalyses. In the view of a climate service processing chain, their needs are:
• access to observational quality control information (e.g. first-guess checks, bias corrections), as being developed in ECMWF’s Observation Feedback Archive,
• mechanisms to exchange knowledge, especially about observations,
• ensembles of boundary conditions from global reanalysis datasets.
Deliverable D5.53 was influenced by W4 and this has been taken into account in the drafting of a white paper outlining the need for reanalysis intercomparison guidance and a task team at an international level. One indirect outcome was the creation of a WCRP Task Team for Intercomparison of ReAnalyses.
WP6 Dissemination, outreach and capacity building
This workpackage was about “Dissemination, outreach and capacity building” and as such there are no Science and Technology results to report on.
Potential Impact:
WP1 Project management
This workpackage concerned the project management and as such there are no “Impact and dissemination activities nor exploitation of results” to report on.
WP2 European ECV capability and structured ECV process
Work and results of Task 2.1 had specific impact at different levels of the European community engaged in the production or usage of climate data records. Impact was realised at national, European and global level (see below for specific examples). The impact mostly became possible by the way how the maturity assessment was conducted by the CORE-CLIMAX project which was highly inclusive for the participants and also had a big educative process behind that was explaining in detail the benefits of performing a maturity assessment.
Specific examples for currently known impacts at national level are:
• Deutscher Wetterdienst as one of the CORE-CLIMAX partners has used the maturity assessment to analyse its in situ data record provision. This had the consequence of introducing a version control for data sets and also to provide annual updates of the data records;
• The Met Office, United Kingdom, as participant in the CORE-CLIMAX assessment has realised the importance of software maintenance for the systems generating very prominent climate data records such as the Hadley Centre Sea Surface Temperature data set (HadSST). The consequence of that analysis was a recruitment of a software engineer to improve the software maintenance in the future;
• The Deutsche Klimarechenzentrum GmbH, Hamburg, Germany has used the CORE-CLMAX maturity model to start the definition of a similar maturity model for climate model data stewardship;
At European level:
• At the beginning of the CORE-CLIMAX project, ESA was pursuing similar activities within the ESA Climate Change Initiative (ESA CCI). The CORE-CLIMAX project was successful in insisting that ESA should join the activities in CORE-CLIMAX which ended in almost complete participation of ESA CCI data records in the CORE-CLIMAX assessment;
• The Framework 7 project QA4ECV has taken up the CORE-CLIMAX maturity matrix into its more general Quality Assurance framework. The maturity matrix will be applied towards the end of the project to assess all climate data records produced as part of QA4ECV;
• The Horizon 2020 project GAIA-CLIM is using the maturity matrix development within its concept to evaluate the adequacy of surface based reference measurements for the characterisation of satellite measurements and derived products. It is within the remits of this project that a further utilisation of the maturity matrix approach for in situ data will be further fostered;
• The Horizon 2020 project EUSTACE is using the maturity matrix approach to monitor the development of global surface temperature climate data records to be developed within the project. This will enable to clearly demonstrate progress due to the project work;
At global level:
• The WMO Sustained Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM, http://www.scope-cm.org/) initiative uses the maturity matrix to monitor progress for its ten global projects on climate data generation;
• In the context of the preparation of the next Climate Model Intercomparison Project (CMIP-6) the obs4mips activity, that prepares the provision of evaluation data sets for climate modelling groups, uses the CORE-CLIMAX Maturity Matrix as starting point for the development of a so called Model Evaluation Readiness Level Matrix to support the evaluation of the suitability of proposed climate data records for climate model evaluation;
• The CEOS/CGMS Working Group Climate has the potential to assess the content of the global inventory of GCOS Essential Climate Variable data records (currently 220 data records) using the CORE-CLIMAX methodology. As most of the data records produced in Europe were participating in the CORE-CLIMAX assessment the results will be added to the inventory by EUMETSAT (which takes responsibility of the inventory on behalf of CEOS and CGMS) after the project end.
Some potential impacts could not be realised due to resource issues. For instance the usage of the System Maturity Matrix and the distribution of the assessment results using the technology developed in the parallel project CHARMe could not be realised due to the short duration of the projects. Both projects were and the participants are still very interested to include CORE-CLIMAX assessment results into the commentary meta data feed of CHARMe. It is suggested that the results of both projects shall be at least utilised in the framework of the Copernicus Climate Change Service but it could be useful for other Copernicus services as well.
With respect to the Copernicus Climate Change Service the systematic application and further development of the CORE-CLIMAX system maturity matrix and the spin-off Application Performance Metric were strongly endorsed by participants of a Copernicus Climate Change Service workshop on the content of the Climate Data Store in July 2015. However, currently no implementation plan, e.g. as part of the Quality Control & Evaluation (QC&E) pillar, exists for this functionality. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation in the climate change service.
Compared to the impact of Task 2.1 the impact of Task 2.2 is not really measurable at the end of the project. This is due to the fact the associated deliverable D2.26 is only released at the end of the project. In addition, the relative big success of the maturity assessment and associated impact including all required communications around it has also led to a reduced community engagement for Task 2.2. This is mainly due to the relatively short duration of the project that does not fit long-term community engagement into the production of a document on the structured processes. However, the structured process diagrams are useful to other EU FP-7 and H2020 projects. For instance the QA4ECV project has started drafting so called traceability diagrams that sketch execution pathways for algorithms which represent further sublevels of the process diagrams provided by CORE-CLIMAX. In addition the Horizon 2020 project FIDUCEO aims at publication of so called best practice documents that can certainly make use of the structured process diagrams developed by CORE-CLIMAX.
WP3 Validation process
The most evidenced impact is indicated in the 37th CEOS-WGCV plenary minutes, which was quoted as: “Before developing a cross-cutting project on maturity matrices across CEOS, WGCV needs to look at the work of the CORE-CLIMAX group where they are already looking into maturity matrices approaches including extensions to the original Maturity Matrix, called the “System Maturity Matrix” and the Application Performance Matrix (APM). They are also looking at Uncertainty. What is the interest for WGCV in maturity matrices? What can WGCV learn from the CORE-CLIMAX study? Is a simple number based rating sufficient to meet WGCV’s requirement and those of the user, or is a different approach warranted?”
The CORE-CLIMAX efforts in developing a quality indicator for Cal/Val was recognized by CEOS-WGCV and provoked discussions among the community on how to incorporate this concept into the practical operation of CEOS-WGCV.
Dissemination has been done via participating in various of workshops, conferences and plenaries:
1) The GEPWs (GEO European Project Workshops, Barcelona & Athena, 2013 & 2014);
2) IGWCO (GEO Integrated Global Water Cycle Observations Community of Practice (CoP) Meetings, 2013 & 2014);
3) The 37th CEOS WGCV (working group on calibration and validation) Plenary (2014);
4) The GEO-X Plenary and the 2013 Ministerial Summit (2013);
5) The CORE-CLIMAX Collocation Meeting, Brussel (2013)
6) The CORE-CLIMAX ECV Capacity Assessment Workshop, EUMETSAT (2014)
7) The CORE-CLIMAX Capacity Building Workshop in Finish Meteorological Institute (FMI), Helsinki, Finland (2014)
8) The CORE-CLIMAX Capacity Building Workshop as the side event of the 14th EMS Annual Meeting & 10th European Conference on Applied Climatology (ECAC) (2014)
9) The CORE-CLIMAX Collocation Meeting, Brussel (2014)
10) The main Science and Technology result was published in an international peer-reviewed journal (International Journal of Applied Earth Observations and Geoinformation), 2015
WP4 reanalysis feedback mechanism to CDR update
As far as the impact concerns, WP4 Deliverables influenced the WP5 Deliverable D5.53 and this has been taken into account in the drafting of a white paper outlining the need for reanalysis intercomparison guidance and a task team at an international level. One indirect outcome was the creation of a WCRP Task Team for Intercomparison of ReAnalyses.
The ideas spelled out in the deliverable D4.42 “Design of support infrastructure for CDR quality assessment in a reanalysis environment” promoted the value of the observational feedback and collocation, and were instrumental in the virtual observatory concept now being worked in an H2020 project.
Dissemination activities, apart from the listed publications, workshops and conferences in Template A2 in this work-package consisted for a large part of working visits to several international institutions. These are listed in Template A2 under “visits”.
WP5 Intercomparing reanalysis results
WP5 has potentially still a high impact on development of Copernicus Climate Change Services (C3S). These are the main conclusions that were made during WP5 concerning C3S:
Data Store: Users of reanalysis products would benefit from improved data access. Explanations for nominal resolution and feature resolution should be available. Enhanced tools for data discovery, extraction and subsetting would reduce data transfer volumes. Ready-made graphical products would make the service more time efficient. Sectoral Information System: Up- and down-stream users have varying needs in the private and public sectors and thus a constant dialogue between the users would advance understanding of the changing needs. Faster release of the datasets would be a priority for providing better 24/7 climate services. Evaluation and Quality Control: The current level of uncertainty regarding biases in reanalysis datasets and resolution effects between the reanalyses in both the time and space dimensions, are identified as impediments to climate service and research. Five categories of comparison were identified which are fundamental in evaluating and quality-controlling the use of reanalyses. The categories are: descriptive product comparison; comparison with third party observation-based CDRs; intercomparison between different reanalyses; thematic comparison; internal metrics comparison. Each category benefits from the findings of the other categories. Outreach and Dissemination: Targeted training (e.g. e-learning) for different user subgroups would be appreciated. Examples of climate service case studies using alternative datasets as input could be useful in capacity building. A more comprehensive set of diagnostics would help the users to factor in the uncertainties inherent in the reanalysis data. Capacity-building for the five identified EQC intercomparison categories, with special emphasis on targeting the understanding of internal reanalysis metrics, is a priority for outreach.
Co-ordination: A need for co-ordination is clear, not least to mobilize the participants, to collate/disseminate findings, and to promote the sharing/use of common software tools. Efficiency is also enhanced when reanalysis intercomparisons are preceded by standalone evaluation of individual reanalyses under the responsibility of each producer. It could be argued that a minimum level of self-evaluation quality should be reached and documented prior to undertaking extensive intercomparison – this level could be characterized by achieving threshold levels (to be defined) in the Core-Climax System Maturity Matrix. More generally, the timeline of the intercomparison workflow will need a set of triggers (decision points) in order to establish when the various activities can be usefully undertaken.
Operational services Climate change services are essential for supporting decision making. In addition to inter-comparing and developing reanalysis products for boosting the delivery of climate services, interaction between science, business and policy making should be improved. For determining the cause for the new observed records, it would be crucial to have the observation, weather and climate services working closely using e.g. “mobility as a service” approach. By combining the chain from data assimilation to reanalysis production, product development, interactive tools, operational climate service, climate change research and communication, it will be possible to upgrade the climate services to an operational phase and thus advance the development Copernicus services as part of the Global Framework for Climate Services.
WP6 Dissemination, outreach and capacity building
This workpackage concerned the “Dissemination, outreach and capacity building” activities and had as objectives the coordination with (European and international) stakeholders and a dissemination and/or outreach activity.
Thematic coordinating with international activities took place through the organization of the following coordination forum workshops:
▪ Coordination forum Workshop 1 (D6-61): Kick-off Meeting “COordinating Earth observation data validation for RE-analysis for CLIMAte SercieS” (CORE-CLIMAX) at ITC premises, 17-18 January 2013, Enschede, The Netherlands
▪ Coordination forum Workshop 2 (D6-62): Co-location Meeting EC FP7, ESA and EUMETSAT ECV-related Projects at REA premises, 15-16 May 2013, Unit S2 – Space Research, COV2 19-SDR2, B-1049 Brussels, Belgium
▪ Coordination forum Workshop 3 (D6-63): CORE-CLIMAX Essential Climate Variable (ECV) Capacity Assessment Workshop at EUMETSAT headquarters, 21-23 January 2014, Darmstadt, Germany
▪ Co-location Meeting “Coordinating European efforts in generating climate data records for Copernicus Climate Change Services (C3S)”, at REA premises, 15-16 January 2015, Unit B1 – Space Research, COV2 17/149, B-1049 Brussels, Belgium
In addition thematic coordinating with international activities was also done by participation in international workshops and conferences, which are listed in Template A2.
Apart from the dissemination done by (scientific) publications, presentations, meetings, visits, posters, abstracts, an important component consisted of the organization of two Capacity Building workshops:
▪ 1st Capacity Building Workshop (D6-65), 19-20 March 2014 at FMI premises, Helsinki, Finland
▪ 2nd Capacity Building Workshop (D6-66), 10 October 2014 at ESM/ECAC conference, Prague, Czech Republic
Details hereof can be found on the project’s website and in Template A2.
Project as a whole: The CORE-CLIMAX paradigm
Following up on a previous collocation meeting of mid-2013, a second collocation meeting was organised by the CORE-CLIMAX project at the Research Executive Agency in Brussels on 15 and 16 January 2015. The aim of this meeting was to coordinate with the different H2020, FP7, ESA & EUMETSAT ECV-related projects the processes of generating, validating and updating (including reanalysis) ECVs and to understand the gaps between the ECVs and user needs in the newly established Copernicus Climate Change Services (C3S). Specific questions related to these processes were addressed: what needs to happen next, what would be needed for that to happen, which are issues of concern, and how can these potentially be solved?
European structured ECV capability
Concerning the “European structured ECV capability” it is important to look back on the success from the Joint Research Centre (JRC) workshop in 2009 to the establishment of the C3S today. For the establishment of the C3S the obstacles are the transition to operations and funding, e.g. issues with Long Term Data Preservation (LTDP) versus Climate Change Initiatives (CCI). Also, CDRs generated from scientific activities such as the CCI and the projects will need to be funded by the service or transitioned to other operational budget such as the Satellite Application Facilities (SAFs).
For the transfer of FP7/H2020 project results into C3S there is a need to establish and maintain a regular dialogue between C3S and the projects to enable best uptake as possible. Concerning the transfer of knowledge from projects to operational activities such as SAFs, best practice guides need to be brought into work, continuous efforts should be invested to achieve this. Such efforts should not and is not feasible to rely on one-off projects.
Validation process in ECVs
A better engagement of metrologists is needed to define best practices. It is recommended to link to the European metrology community as much as possible and to secure funding for independent pre-launch and long term post-launch Calibration/Validation (CAL/VAL) studies.
Innovative validation methods and infrastructures are needed to permit a cross-cutting quality monitoring (i.e. considering several variables at the same time and assessing the consistency between variables).
It is recommended to (1) develop the use of data assimilation for validation as the obtained analyses account for the synergies of the various upstream products and provide statistics that can be used to monitor the quality of the assimilated observations; (2) create or consolidate in situ reference data sets by measuring as many variables as possible at the same location, using several relevant sensors.
Finally, there is a need for better international collaboration. The European efforts in developing reference datasets need to be coordinated with the international community in as much as global products considered.
Reanalysis feedback mechanisms
The group recognized that reanalysis-based CDR assessment has proved valuable and endorsed the idea of widening its use.
It is recommended to have short feedback loops in the assessment process and to facilitate 3rd-party access to the quantitative datum-level feedback information.
The main needs are for (1) sustained infrastructure to compute/archive/analyse reanalysis feedback, (2) more personnel with the skills to conduct reanalysis-based CDR assessment, and (3) systems for the CDR providers to make visible to users the assessment findings and any ensuing CDR update plans.
Lack of capacity is a potential obstacle, which could be addressed by programs to build more capacity.
Inter-comparing reanalysis results
The main objective of inter-comparing re-analyses is to estimate the uncertainties of the various reanalysis products.
This is needed to help the users to decide whether re-analyses are generally fit for the user’s purposes, to help users to decide which product to use, and last but not least to build trust in the reanalysis products.
Typically, users firstly require an overview of available products. Possibly they are looking for any alternatives to familiar products. In any cases, they need help to pick the right product. Maybe they just think of the easiest to download or the highest resolution and would benefit from being pointed to other criteria. In the future, specially targeted inter-comparisons could provide such criteria.
It is important to note that usually users do not have the resources to do inter-comparisons, or to estimate uncertainties for their particular applications on their own. They might lack the know-how or the resources to do so. Thus, there is a gap to be bridged for a full use of C3S re-analyses data sets, where re-analyses inter-comparisons could help.
A concern is that various users have quite different applications and backgrounds. In other words, we need various bridges and have to cover a range of inter-comparison methods and independent reference data. Strengthening the community efforts could help to address a wide range of questions related to uncertainty and could also increase user involvement. Inter-comparison of re-analyses would as such feed into C3S guidance and user support.
Beyond CORE-CLIMAX – final Recommendations
CORE-CLIMAX can be compared to ISO procedures where it comes to quality standards concerning CDR/ECV/reanalysis products, i.e. time series. As such the project constitutes an important building block for climate services.
With respect to the C3S the systematic application and further development of the CORE-CLIMAX system maturity matrix and the spin-off Application Performance Metric were strongly endorsed by participants of the C3S Climate Data Store workshop in July 2015. However, currently no implementation plan, e.g. as part of the Evaluation & Quality Control (QC&E) pillar, exists for these functionalities. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation of C3S. Particularly, the SMM and APM can provide starting points for the EQC function of the C3S to assess the technical and scientific quality of the service including the value to users.
More training is required in order to capitalize on what has been achieved in CORE-CLIMAX project and more generally for the success of the C3S, which hints both at improving university education as well as other education activities (e.g. ESA training activities).
Currently many CDR’s are not ready yet for application, consistent high-resolution data is a challenge and more research is urgently needed. Linking ECV’s to Essential Water Variables is the next step from closing the gaps from CDRs to sectorial applications, the generation of different data sets should be related to high impact phenomena like severe droughts and flooding events.
At the closure review meeting at Research Executive Agency in Brussels, on 22 June 2015, the external reviewer commented that CORE-CLIMAX has produced results way beyond its original scope. It has set a new paradigm for data handling and data quality regarding the production of climate services. These issues are very sensitive and the best practice approach of the project has created community acceptance.
List of Websites:
Public website address: www.coreclimax.eu
The COordinating Earth observation data validation for RE-analysis for CLIMAte ServiceS (CORE-CLIMAX) project aimed to substantiate how Copernicus observations and products (from both in situ and space components) can contribute to climate change analyses, by establishing the extent to which observations complement existing Climate Data Records (CDRs).
The CORE-CLIMAX consortium comprises seven European institutions (University of Twente, Faculty for Geo-information Science and Earth Observation (ITC); European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT); European Centre for Medium-Range Weather Forecasts (ECMWF); German Weather Service (DWD); Flemish Institute for Technological Research (VITO); Finnish Meteorological Institute (FMI); Meteo-France (MTF)) and two International Cooperation Partner Countries institutions (Chinese Academy of Sciences, Institute of Tibetan Plateau Research (ITP); Chinese Academy of Sciences, Cold and Arid Regions Environmental and Engineering Research Institute (CAREERI)), each responsible for measuring, monitoring and disseminating various aspects of climate change related to their particular areas of expertise. The consortium aims to provide coordination for producing and validating Essential Climate Variables (ECVs) with the objective of reanalysing existing data to develop a clearer picture of global climate change and its implications.
The CORE-CLIMAX consortium is coordinated by the University of Twente. Together with EUMETSAT coordination is made with the European Space Agency’s Climate Change Initiative (ESA-CCI) through several key national ECV producers and climate change service providers (DWD, FMI, MTF) - along with the major reanalysis centre in Europe, ECMWF, and satellite data processing and validation centres (VITO). The two ICPC participants (ITP, CAREERI) are tasked with maintaining all of the activities at the major climate observation stations situated on the Tibetan plateau. As such these participants provide an essential role in coordinating the validation of ECVs and reanalysis results. Furthermore, these participants are also responsible for monitoring climate variations at the Tibetan Plateau (together with surrounding areas also known as the Third Pole Environment). With such a wide range of expertise spread throughout key geographical locations, CORE-CLIMAX was able to implement an internationally recognised mechanism for the coordination of ECV Climate Data Records (CDR) generation. The varying geographical locations of the consortium’s participants also facilitated links with space nations holding satellite data repositories.
To achieve the aim of the project, CORE-CLIMAX has assessed the European capability to provide essential climate data records, prepared the white book for structured process to derive CDRs, coordinated a harmonized approach for validating Essential Climate Variable (ECV) CDRs, identified the integration of CDRs into the reanalysis chain, and formulated a process to compare the results of different reanalysis techniques.
ECV capacity assessment
It is essential to assess the capability of the existing CDR development activities to ensure the prolonged generation of high-quality ECV CDRs. As such they can help to produce the underpinning science that supports decisions on mitigation and adaptation for a changing Earth climate. For an assessment of the existing capacity in the most objective way possible, we need tools that provide a basis for information preservation, expectations, and a metric for progress towards completeness. The System Maturity Matrix (SMM) is such an assessment tool, which assesses whether the CDR generation procedures have been compliant with best practices developed and accumulated by the scientific and engineering communities. Also, a prototype of Application Performance Metric (APM) was developed to meet the need of giving advice to data users which data record can be used for which application.
White book for the structured delivering of ECV CDRs
The white book describes the structured process to derive ECV data records, based on the architecture for climate monitoring from space. This architecture outlines major processes to generate long-term climate data records as well as data records which are updated with higher frequencies, the so-called interim climate data records. The architecture for climate monitoring contains the major activities describing the value adding chain for climate data down to the usage of the data in applications and subsequently for decision making. The CORE-CLIMAX process description focusses on the activity of creating and maintaining short/medium term and long term CDRs. Specifically, each step in generating a CDR was detailed using the IDEF0 (Icam DEFinition for Function Modeling, where 'ICAM' is an acronym for Integrated Computer Aided Manufacturing) standard.
Generic Validation Processes for ECV CDRs
To establish the traceability of a validation process, one is requested to document each step of the validation process. Starting from documenting how the reference dataset is generated, the generic process requires the assessment of the independence levels of the reference dataset. Furthermore, it is suggested to implement both the self-assessment and independent/external assessment of the CDR products and the validation process, and to assure the understanding of the data quality. For a complete validation process, one should also consider the consistency of the validated CDRs (e.g. compared with the physically inter-related variables). The last step is to sustain the established validation facilities and procedures. This final step means achieving an operational validation level, at which validation activities and data release are regularly implemented.
Reanalysis Feedback Mechanism to CDR updates
The advent of operational climate services increases the need for producers of CDRs to know about the needs of reanalysis, particularly regarding desirable CDR updates. The reanalysis needs can be effectively communicated to CDR producers via quality assessment procedures. It is through this communication (e.g. an iterative loop linking "Comparison & Assessment" and "Response by CDR provider") that (a) CDR weaknesses are reported, and (b) the CDR producer's response can be developed to a level at which CDR update plans can be formulated.
Reanalysis intercomparison
Reanalysis intercomparison activities are a key component of characterizing reanalysis uncertainties. As such, they yield information that assists users in deciding which reanalysis product might be most suitable for their particular application. The CORE-CLIMAX project developed a set of procedures for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs. To do so, five categories of comparisons are identified: 1) Descriptive product comparison; 2) Comparison with third party observation-based CDRs; 3) Intercomparison between different reanalyses; 4) thematic comparison (i.e. how well can the reanalysis products be applied to understand a particular problem); 5) Internal metrics comparison.
Recommendations
CORE-CLIMAX can be compared to ISO procedures where it comes to quality standards concerning CDR/ECV/reanalysis products, i.e. time series. As such the project constitutes an important building block for climate services.
With respect to the Copernicus Climate Change Service (C3S) the systematic application and further development of the CORE-CLIMAX system maturity matrix (SMM) and the spin-off Application Performance Metric (APM) were strongly endorsed by participants of the C3S Climate Data Store workshop in July 2015. However, currently no implementation plan, e.g. as part of the C3S Evaluation & Quality Control (EQC) pillar, exists for these functionalities. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation of C3S. Particularly, the SMM and APM can provide starting points for the EQC function of the C3S to assess the technical and scientific quality of the service including the value to users.
More training is required in order to capitalize on what has been achieved in CORE-CLIMAX project and more generally for the success of the C3S, which hints both at improving university education as well as other education activities (e.g. ESA training activities).
Currently many CDR’s are not ready yet for application, consistent high-resolution data is a challenge and more research is urgently needed. Linking ECV’s to Essential Water Variables is the next step from closing the gaps from CDRs to sectorial applications, the generation of different data sets should be related to high impact phenomena like severe droughts and flooding events.
At the closure review meeting at Research Executive Agency in Brussels, on 22 June 2015, the external reviewer commented that CORE-CLIMAX has produced results way beyond its original scope. It has set a new paradigm for data handling and data quality regarding the production of climate services. These issues are very sensitive and the best practice approach of the project has created community acceptance.
Project Context and Objectives:
CORE-CLIMAX coordinated the identification of available physical measurements, which can be reconciled with previously existing data records, to form long time series. It aimed to substantiate how Copernicus observations and products can contribute to climate change analyses, by establishing the extent to which Copernicus observations complement existing Climate Data Records (CDR).
With GCOS, Copernicus and ESA CCI projects, and EUMETSAT including its Satellite Application Facility (SAF) network, coordination took place with specific efforts to be undertaken by new Copernicus projects to further upgrade their product catalogues to include this climate relevant validation and information and lay the observational basis for service activities. CORE-CLIMAX identified the integration of ECVs into the reanalysis chain by proposing a feedback mechanism ensuring that the results of the re-analysis process get appropriately reflected into updates of the ECVs. Together with intercomparing different reanalyses, CORE-CLIMAX contributed to establish a European truly coupled gridded re-analysis which incorporates full exchanges and interactions between atmosphere, ocean, land, including the hydrological cycle.
Specific objectives:
1 Coordinate with Copernicus ongoing activities and contribute to the formulation of the Copernicus climate service theme (GCOS, Copernicus and climate change projects, ESA CCI projects, EUMETSAT including its SAF network and EUMETNET as part of the European Meteorological Infrastructure);
2 Propose a structured process for delivering ECVs through the stepped and quality controlled elaboration of CDR, the latter being derived from prioritisation of the most appropriate input data sets;
3 Propose a validation process aiming at qualifying the accuracy of the climate variables;
4 Propose a feedback mechanism ensuring that the results of the re-analysis process get appropriately reflected into updates of the CDR;
5 Propose a process to compare reanalyses
Project Results:
WP1 Project management
This workpackage concerned the project management and as such there are no Science and Technology results to report on.
WP2 European ECV capability and structured ECV process
The work in work package two was divided into two major tasks addressing a European assessment for the capacity to generate climate data records suitable for monitoring of climate variability and change and to document the process to current best knowledge on how climate data records are being generated.
2.1. Analysis of ECV capability and prioritisation for Copernicus climate service
Introduction
In support to the establishment of the Copernicus Climate Change Service and other international activities such as the establishment of the architecture for climate monitoring from space conducted by major space agencies one major objective of the CORE-CLIMAX project was to systematically assess the capacity of ongoing European activities in the area of generation and provision of climate data records. With respect to the Copernicus Climate Change Service also the role of in situ data and model-based reanalysis need to be considered. As the in-situ data sets are generally under national responsibility the CORE-CLIMAX assessment did not cover all in-situ data records generated by European providers. However, the inclusion of some of the best known in-situ data records in the assessment has demonstrated the value of the tools developed in CORE-CLIMAX and encourages more nationally managed data records to be included in a future assessment.
Capacity Assessment: Concept and Methods
For an assessment of the European capacity in the most objective way possible tools were developed that provided a basis for information preservation, expectations, and a metric for progress to completeness. The maturity matrix approach proposed by Bates and Privette (2012) offered a systematic mean to assess if the data record generation follows best practices in the areas science, information preservation and usage of the data. Some example uses of the matrix maturity are the assessments of data records developed in the NOAA Climate Data Record program and in the 2nd phase of SCOPE-CM to measure progress in the projects. For both these cases, maturity assessments were first done as self-assessments. External assessments could be done in a form of audit.
Basically, three different aspects of our capacity to generate data records need to be considered:
• Scientific, engineering and information preservation practices;
• Usage of products including feedback and update mechanisms;
• Quality of products with respect to applications.
Assessing if data record generation follows best practices provides an internal view on strengths and weaknesses of the processes to generate, preserve and improve climate data records for agencies and each individual data record provider. It also provides a general information to the community concerning the status of individual data records as well as collective information on the state of all existing records, highlighting areas for development and improvement. The assessment of quality of products is facilitating an external view on data records trying to answer the most important user question: Is the quality good enough for my application?
The CORE-CLIMAX project defined three major elements for its capacity assessment:
• Data record descriptions that contain technical specifications and also information on quality;
• A System Maturity Matrix (SMM) that evaluates if the production of a data record follows best practices for science, engineering, information preservation and facilitation of usage, and;
• An Application Performance Metric (APM) that attempts to evaluate the performance of an ECV CDR with respect to a specific application. To be able to apply the APM, user requirements for each application are needed to be compared to the actual technical specifications and validation results.
The three elements of the capacity assessment were designed to be independent of each other and represent means to support an assessment but do not provide the assessment results per se. The SMM is designed to be principally used without considering specific applications. With this the SMM does not depend on user requirements for specific applications and their change over time. In contrast the APM should facilitate a comparison of the real technical features of a data record and results of validation and other data quality assessment activities to user requirements for an application. It basically provides summary information on how close a specific data record is at fulfilling the requirements of a specific application. A prototype APM was developed during the capacity assessment workshop because the need of giving advice to data users which data record can be used for which application. This need is manifested for instance in the huge amount of information provided on validation of data records that is unlikely to be processed by institutions that want to use the data records. The APM is intended to support institutions in making choices among different existing data records without the need to assess the full documentation of all potential data records. However, it shall be noted that the APM is a new concept that was a spin-off result from the CORE-CLIMAX capacity assessment workshop. The development of an applicable tool was beyond the scope of the CORE-CLIMAX project. It is expected that the APM will be further adapted in the future to become fully functional. For details on data set description, as well as the SMM and APM tools please see Deliverable 2.25.
Capacity Assessment: Results
The CORE-CLIMAX capacity assessment experienced great community support (EUM SAF, ESA CCI, Copernicus Services, National weather services, EU projects) leading to a successful assessment. The assessment methodology outlined above was applied to 40+ data records including satellite, in situ and re-analysis data records. It can be noted that the self-assessments of the data records were performed with high honesty making the results reliable for further usage. During the review of the assessment report (Deliverable 2.25) some data providers updated the assigned maturity to lower values. Seeing other assessments obviously led to self-regulatory process to even more realistic results.
Assessment Workshop
The CORE-CLIMAX Essential Climate Variable (ECV) Capacity Assessment was initialised by a workshop that was held at EUMETSAT Head Quarters in Darmstadt on 21 – 23 January 2014. The workshop gathered about 40 participants representing all relevant European climate data record producers including EUMETSAT SAF network, ESA CCI projects, Copernicus Services, EU projects and EUMETSAT member states weather services as well as global stakeholders (EC, CGMS, CEOS, WMO). The workshop covered data records for Land, Ocean, and Atmosphere constructed from satellite and in situ data as well as reanalysis. The workshop participants were asked to do a self-assessment of their data records using the SMM prior to the workshop and there were 30 data sets (23 are based on satellite measurements, 6 are based on in situ measurements, and one is reanalysis based) with SMM results for analyses during the workshop.
The workshop participants endorsed the SMM as a useful tool for assessing completeness and identifying weaknesses in the developed systems which produce CDRs from satellites and in situ measurements and reanalyses. The presentations and breakout group discussions provided a common understanding of most of the elements in the SMM among different communities and programs and based on these understandings further improvements have been proposed for the SMM and its instruction manual. However, the workshop suggested that the process of introducing the concept of SMM in different communities needs to be open and inclusive. As the SSM is based on “best practices”, reviews and changes to the SMM are expected on longer time scales (~10 years), but it is expected to have a review and eventual change prior to an implementation to a major service such as the Copernicus Climate Change Service. It was recommended to use the SMM in the review process of existing CDR programs such as EUMETSAT-SAF network and ESA-CCI.
Assessment Completeness
The assessment tried to be as comprehensive as possible. Enough major European climate data providers participated in the assessment to allow conclusions about the applicability of the SMM for a capacity assessment. The assessment has been performed for satellite and in situ data records as well as weather prediction model-based reanalysis output. Among the satellite data providers are EUMETSAT including its Satellite Application Facility Network, the ESA Climate Change Initiative, Copernicus Atmosphere and Land Services.
It has to be emphasized that such an assessment had been done for the first time for in situ data records and the great response from the dataset developers is highly appreciated. Major European players such as Met Office Hadley Centre being responsible for the Hadley Centre Observational data sets (HadObs), Deutscher Wetterdienst having international responsibility for the Global Precipitation Climatology Centre (GPCC), the two leading the European Climate Assessment & Dataset (ECA&D) project, and the Alfred Wegener Institute hosting the Baseline Surface Radiation Network archive participated in the assessment. Their inputs to the fruitful discussions have very much contributed towards adapting the initially satellite oriented SMM to a matrix that is more generally applicable.
There is only one global reanalysis that is being produced in Europe (ERA Interim) and that has participated in the assessment. It is certainly possible to extend the assessment to regional reanalyses such as from the EU FP-7 UERRA project in the future. It is suggested to perform SMM assessments for such datasets in the framework of the Copernicus Climate Change Service.
Although the assessment was done in a self-assessment mode, an audit-type assessment for some of the randomly picked datasets revealed that most of the data providers have been very honest in assigning SMM scores to their products. This provides an overall large credibility of the self-assessment results presented here.
Value and Potential Usage
The value of the self-assessment has several aspects. Firstly, it provides a consistent view on strengths and weaknesses of the process of generating, preserving and improving CDRs to each individual CDR producer, agencies and the EC. In particular, the assessment fulfilled its promise to be useful for the data providers by providing guidance to further development of their data products. The assessment made some data providers for the first time thinking about issues such as software maintenance which are a cost sensitive long-term item for an operational service, in particular for data records based on satellite data. Data providers are encouraged to repeat the self-assessment periodically, e.g. annually, that enables progress monitoring of a guided development.
Secondly, the presented assessment provides detailed information on the status of of European activities to provide climate data records for GCOS ECVs. It may provide guidance for climate services where eventually to concentrate their investments for climate data record generation activities. For instance, data records showing maturity levels 3 and 4 (forming the initial operations capability) in many categories are certainly those that have good chance to be developed into full operations capability within the next 3-5 years. Others showing maturity levels 1 and 2 (being at research level) may be invested in but rather for doing more research and engineering development than transfer to operational services.
Thirdly, the assessment provides information to the general user community of CDRs, including science and services, on the status of individual records and the collective state of all records. It also provides this information for the first time across different observing systems, i.e. users can compare between satellite and in situ data sets on the different aspects covered by the maturity assessment. By this transparency and openness towards the user of CDRs has been significantly enhanced by the CORE-CLIMAX project.
Fourthly, the collective effort of the assessment to compile consistent descriptions of the participating data records has better facilitated Europe’s contributions to international activities on the evaluation of climate models using observations such as Obs4Mips that require standardised data record descriptions.
2.2. Preparation of a white book for structured process to derive ECV data records
This task covers a description of the structured process to derive ECV data records. The starting point for this description is the architecture for climate monitoring from space that outlines major processes to generate long term climate data records but also data records which are updated with higher timeliness as the so called interim climate data records.
Figure 1: Decomposition of the four pillar of the architecture for climate monitoring from space into individual activities.
Technically the structured description as delivered in D2.26 uses the IDEF0 standard (Icam DEFinition for Function Modeling, where 'ICAM' is an acronym for Integrated Computer Aided Manufacturing) to create flow diagrams that go deeper into details with each level of the considered process.
Figure 1 describes the top level adapted from the architecture for climate monitoring from space that contains the major activities describing the value adding chain for climate data down to the usage of the data in applications and subsequently for decision making. The CORE-CLIMAX process description concentrates on the elements A2 and A3 in Figure 1 that describe the production of data records. As an example Figure 2 shows the detailed description of the activities and outputs of the A3 box of Figure 1. The Deliverable 2.26 contains for each of the sub boxes shown in Figure 2 further descriptions.
Figure 2: Schematic representation of the process to generate a long climate data record.
In addition to the data record generation elements also a more generic description of the associated life cycle including all necessary review processes became part of Deliverable 2.26. Figure 3 shows the top level life cycle diagram for illustration. The life cycle has as major input user requirements as for instance provided by the Global Climate Observing System (GCOS). The requirements are analysed and converted into so called Product Requirements which represent a development target for the CDR. Such a step is necessary because user requirements can be apparently unreachable in particular if historic instruments are being used. By developing product requirements related to specific applications historic data are put into the right perspective.
The life cycle also demonstrates that each step of the generation of a climate data record is finalised by a formal review step that documents the status of the development and prevents any development and/or production of data on an unsound basis of methodology or data. The process for the final data record release review is finally sketched in Figure 4 that expands the process behind the L5 box in Figure 3.
Figure 3: Schematic representation of the life cycle for the development of Climate Data Records from initial user requirements to the final CDR.
Figure 4: Schematic representation of the Data Release Review that precedes the publication of climate data records.
WP3 Validation process
The generation of ECVs needs to put strong emphasis on the generation of fully described, error-characterized and consistent satellite-based ECV products. For example, generation of many ECVs (such as in the ESA CCI projects) requires ancillary information about the state of the atmosphere (e.g. cloud screening for SST, atmospheric correction for space-borne altimeters). As such, the consistency between the various ECV products (e.g. cloud flags in one ECV and non-flag in another one) extends to ensuring consistency in the approaches of CDR generation. The validation process as the integral components of the CDR processing chain, is a critical step to ensure consistency. In WP3, there are two main tasks:
• Task 3.1. Coordinating a harmonized approach for validating each ECV/CDR
• Task 3.2. Analysis of ECV/CDR validation network and strategy
In the following, the main science and technology results from these two tasks will be briefly introduced in terms of the three deliverables:
• Deliverable 3.1. Protocol for verifying, monitoring, calibrating and validating FCDRs and TCDRs
• Deliverable 3.2. Generic list for validation strategy for CDRs/ECVs
• Deliverable 3.3. Assessment report on consistency of the CDRs/ECVs
3.1. Protocol for verifying, monitoring, calibrating and validating FCDRs and TCDRs of the ECV CDRs
The need for minimal uncertainty in climate monitoring, together with the need to combine data from a variety of sources (space and in-situ), and emerging products with data assimilation, have placed “traceability” and its quantification at the top of the agenda for climate monitoring (Dowell et al 2013). Traceability is defined as the property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations/validations each contributing to the measurement uncertainty. It implies that a reference standard needs to be established. In CORE-CLIMAX, the System Maturity Matrix proposed in WP2 has included this in the uncertainty characterization sub-matrix. In addition, quality indicator has to be determined. The quality indicator (QI) is a means of providing a user of data or derived product with sufficient information to assess its suitability for a particular application (i.e. fitness for purpose). In CORE-CLIMAX, the Application Performance Matrix proposed in WP2 can be utilized as such indicator. This information should be based on a quantitative assessment of its traceability to an agreed reference or measurement standard, but can be presented as numeric or a text descriptor, providing the quantitative linkage is defined.
As the purpose of calibration and validation is to provide confidence in the quality of FCDRs/TCDRs and their uncertainties, it is important to provide sufficient information on how the Cal/Val activities are carried out, at each stage of the data process chain. It allows all end-users to evaluate a data product’s suitability for their particular application (e.g. ‘fitness for purpose’). To achieve this purpose, it is suggested that the documentation, at each stage of the data process chain (e.g. Cal/Val in the case discussed here), should follow guidelines or best practices, which aims to achieve transparency of approach (e.g. traceability) and improve efficiency. This has been covered by the SMM with the “Metadata”, “User Documentation” and “Uncertainty Characterization” sub-matrices.
As the existing GCOS principles, requirements of the Climate Data Records from Environmental Satellites: Interim Report of the National Research Council NRC, QA4EO, CEOS-WGCV and similar efforts are aiming to provide common reference standards for CDR monitoring and generation in a community-desired-endorsed manner. It is important for CORE-CLIMAX to adhere to these common reference standards. Therefore, the Cal/Val protocols for FCDRs/TCDRs can be proposed as follow:
1. Generation of FCDRs should adhere to community-desired-endorsed principles, requirements and standards, and be assessed with SMM and APM;
2. Validation of TCDRs should adhere to community-desired-endorsed principles, requirements and standards (e.g. CEOS working group on Cal/Val-WGCV protocols), and be assessed with SMM and APM;
3. The Cal/Val activity under planning should be documented, adhering to community-desired-endorsed principles, requirements and standards (e.g. QA4EO, GCOS-153). The documented Cal/Val plan should be evaluated with the SMM sub-matrices to understand how the Cal/Val plan is traceable;
4. When the Cal/Val activities related to FCDRs/TCDRs are beyond the assessment of the calibration and validation thematic areas proposed in this protocol, feedbacks should be given.
3.2. Generic Validation Strategy for ECV CDRs
The Committee on Earth Observing Satellites (CEOS) working group for Cal/Val (WGCV) defines validation as the process of assessing, by independent means, the quality of the data products derived from satellite observations. This can be called product validation. The product validation ensures that the quality of the products is properly assessed, through quantification of the uncertainties in both the data itself and the measurement system deployed for generating the data. It includes a quantitative understanding and characterization of the measurement system and its bias in time and space. In this context, validation can be considered a process that encompasses the entire system, from sensor to product.
Figure 5 shows two typical validation concepts: the scaling method and the direct comparison method. The scaling method is shown in the left panel of Figure 5. The scaling method uses an intermediate Very High Resolution (VHR) satellite data layer (or airborne campaign data) to compare the ground measurements with products at coarser spatial resolution. This permits reducing the uncertainties and the difficulties during the integration of several punctual ground measurements over a common area (or an Elementary Surface Area, ESU) to be used for the validation of the product at a pixel level. This is valid especially for products around 100m of resolution or more, for which it is very difficult to integrate several measurements to reach an ESU of that dimension also taking into consideration the landscape heterogeneity. This is the case for most of the terrestrial ECV CDRs (e.g. land use, LAI).
The consolidated and qualified campaign data (yellow) boxes in Figure 5 indicate the elements needed for validation, including the satellite data, the ancillary/auxiliary data and ground (reference) measurements. The left (green) boxes represent the FCDR processing and TCDR generation (the retrieved quantity or the retrieval) while the right (blue) ones are the processing of campaign data to produce VHR reference validation layers (“true quantity”). In the FCDR processing, raw satellite measurements are geolocated and atmospherically corrected first when relevant and then homogenized and inter-calibrated, to generate calibrated radiances, backscatter of active instruments or radio occultation bending angles. Afterwards, by means of data assimilation or modeling (e.g. radiative transfer model or specific retrieval algorithms), TCDR products are retrieved. In the right (blue) boxes, level 1 data are used to derive level 2 data products at very high resolution. Afterwards, the ground measurements are processed and “transferred” directly to the level 2 data product to represent the validation layers. Finally in the bottom (red) box, the TCDR products are validated by the use of the previously generated VHR reference validation layers.
Figure 5: Validation of ECV CDRs as an expansion of the validation function “D” in Figure 1 showing two typical direct validation concepts, the scaling method (left) and the direct comparison method (right).
It is noted that the generations of both FCDRs and TCDRs are recursive because when improved information becomes available (e.g. better algorithms or improved calibration/validation information) the observations are re-processed to generate improved CDRs. As understanding of sensor calibration issues and as the modeling of the radiative transfer from the Earth and atmosphere improves, products can be generated via reprocessing.
The main advantage of the scaling scheme is the fact that these intermediate layers are very close in terms of quality and resolution to the ground measurements (since they are obtained by VHR data), thus the uncertainties due to the ESU integration can be reduced because now the integration is applied to surfaces at the same spatial scale as the satellite pixels. One should realize that the optimal ESU size is determined by the level of within ESU variability that can be tolerated by the validation protocol and the effort available to conduct measures. The size of ESU within a reference region can vary with various factors (e.g. surface condition, instrument field of view and spatial sampling design etc.) [Fernandes et al. 2014]. Nevertheless, as shown in the blue column of Figure 5, the level 2 data retrieval, the transfer processing, and the aggregation introduce uncertainties that have to be monitored.
In the direct comparison method, the ground measurements are directly compared with the TCDR product retrieved for standard processing. The elaboration needed is the processing/integration of ground measurements to generate ESU comparable with the TCDR product. This method is more “direct” with respect to the other, but the ground measurement processing for the ESU generation can introduce additional uncertainties that again have to be estimated and monitored. Moreover, this processing is applicable if the campaign ground measurements have followed a specific protocol and the area to be covered by the ESU is comparable in size with the product pixel.
From the description of validation concept for ECV CDRs, there are at least three components needed for implementing a validation study (i.e. the purple-red highlighted TCDR box in Figure 5): the validation requirements, the generated TCDR itself, and the established reference data. In addition, one needs to select methods to implement validation. Based on how different European initiatives approached the validation of CDR and ECV products on these aspects, a generic validation process is proposed, together with a quality indicator. The proposed generic validation process for CDRs/ECVs may include:
1. The generation of independent reference datasets;
2. Assessing independence levels of reference datasets (section 3.3);
3. Self-assessment;
4. Independent assessment;
5. External review/evaluation of self-assessment validation practices;
6. Assessing independence levels of point 4 and 5 (section 3.3);
7. Consistency check for inter-related CDRs/ECVs;
8. Sustaining established processes and methods.
The generic validation process was matched with the quality indicator of “Validation” and “Formal Validation Report” of system maturity matrix as shown in Table 1.
Table 1. Matching the generic validation process with the “Validation” and “Formal Validation Report”
3.3. Consistency validation of the ECV CDRs
The climate data store (CDS) for Copernicus Climate Change Services (CCCS) envisions that it shall include essential climate variables (ECVs), uncertainty estimates, reanalysis, multi-model data (e.g. seasonal forecasts and up-to-date climate projections), and in-situ and satellite data. Furthermore, CDS should contain only ‘climate compliant’ data, as defined by Evaluation and Quality Control working group of CCCS. This implicates that one may find certain ECV variable from different sources, and there is a need to do consistency check to determine which dataset are fit for certain particular purpose. On the other side, when there is a certain ECV variable under evaluation, one would like to collect as many as possible independent datasets, under the constraints that they are measuring the same thing, to do (inter)comparison, in order to understand comprehensively the associated uncertainty.
For the in-situ datasets, the homogeneity adjustment plays a critical role in producing long-term climate data. There are direct and indirect methodologies to do homogeneity detection and adjustment [Peterson et al. 1998]. However, it is still difficult to avoid all the inhomogeneity caused by, for example, changes in instrumentation, station moves, changes in the local environment (e.g. building construction), or the introduction of different observing practices like a new formula for calculating mean daily temperature or different observation times. The existence of independent datasets and data assimilation technique provide unprecedented opportunities to spot “bugs” in the in-situ datasets. Figure 6 shows the use of reanalysis data to detect erroneous in-situ measurement.
Figure 6: Example for applying reanalysis for historical quality control: observed daily 10m wind speed at the station Seehausen, Germany (top left) match well with the frequency distribution from the interpolated ERA 20 C wind fields (top right) in recent years whereas 1980-1985 (lower left) the observed station values (lower right) are found to be suspicious.
For the satellite, reanalysis, and multi-model datasets, the same principle of consistency validation as discussed above (e.g. collect as many as possible available independent datasets) is applicable. On the other hand, this principle needs to be constrained with the reference data selection procedure. The major purpose of the consistency validation is to identify what is missing in regards to following the generic validation process (e.g. validation gaps). To achieve this purpose, it is suggested to collect comprehensively the relevant datasets/documents produced by data providers, end users and any other third parties. And, consistency validation is a key for identifying such gaps.
WP4 Reanalysis feedback mechanism to CDR updates
4.1 “Assessing CDR data quality by comparison with existing reanalysis”
The deliverable for this Task, together with Task 4.2 was the document D4.42 “Design of support infrastructure for CDR quality assessment in a reanalysis environment” which was successfully completed by the team and accepted by the Commission. The design was informed by a number of working visits that enabled consultation on proposals and practical testing of infrastructure prototypes. Some of these are highlighted here:
Joey Comeaux from NCAR visited ECMWF to support work on D4.42. The findings were summarized in a visit report. Key results were as follows:
(1) The unique identification of observations is of utmost importance; this identification should flow, untouched, from the data provider to the reanalysis center to allow for iterative update (enriching the original observation archive from assessment in reanalysis environment).
(2) Unintended differences in formats and values in the data when transferred from a data provider to a reanalysis center, even slight, are important to spot and fix early on, before a whole archive has been transferred. To check that, a dedicated method may be needed - probably it could be inspired by standard quality control assurance practices (systematic checks coupled with random checks).
(3) The integration of metadata information with observation data is desirable, before ingest by the reanalysis center. Consequently, such integration should be done by the observation provider rather than the reanalysis center.
Furthermore, the visit by ECMWF to KNMI allowed to approach the WG-GRUAN (Global Reference Upper-Air Network) which held a meeting at KNMI that same week. A visit report was written up. The GRUAN is expected to grow and potentially become a prime provider of near-real-time CDR for upper-air observations in years to come. The finding of interest for D4.42 was that product developers within WG-GRUAN are requesting quantitative feedback on their observation quality, as can be provided by the reanalysis process.
ECMWF visit D5.3-1 determined that the ozonesonde data holdings of the World Ozone and Ultraviolet Data Centre (WOUDC) remain the prime candidate for assessment of atmospheric ozone in-situ data within a reanalysis context. The procedures explored by Joey Comeaux in prior CORE-CLIMAX visit D4.2-1 provide a useful template/starting point. It remains to be seen whether any adaptation is needed when considering ozone rather than temperature data.
ECMWF visit D4.1-3 for which a report was issued, determined that the wind energy community collects a large number of in situ measurements, which could be prime candidates for validation of reanalyses at the local scales. However, due to the economic model under which the wind energy community is placed (concessions and permits), the value of the collected measurement data represents the first return of a concession/permit investment, and consequently these observational data are not exchanged between competitors, even for a fee. Consequently, in a climate service view, this community cannot be expected to contribute any observations at this point.
Visit D4.2-2 was planned with a provider of surface in situ observations, but in May 2013 we heard of recent developments owing to an imminent reprocessing of imagery data from the GOES series satellites. Data from GOES represented so far a glaring omission, such long time-series being not reprocessed, thus failing to realize the value of these observations for climate applications. We thus decided to invite a representative of the center who will do such reprocessing to discuss how to best assist in their efforts and understand how their work fits in a procedural approach to reprocess observations for reanalysis applications.
The resulting visit D4.2-2 to ECMWF by Steve Wanzong from CIMSS, from which a visit report was issued, determined that a satellite data provider who reprocesses imagery data into atmospheric motion vector (AMV) winds would have great interest into improving the feedback received from users, ideally by users making available their own quantitative comparisons of reprocessed data with their own application.
In the case of the reanalysis application, this means opening up the observation feedback archive and populating it with AMV products and corresponding reanalysis feedback as they are received.
ECMWF looked for a way to test the ideas to be presented in deliverable D4.42. To this end, ECMWF prepared a proposal for the Climate Monitoring Satellite Application Facility (CM-SAF) for a visiting scientist to compare with existing reanalyses the quality controlled SSM/T-2 data set acquired within the previous reporting period, from EUMETSAT, within Task 4.2. This proposal involved Paul Poli and Viju John as lead co-investigators, both from Core-Climax (note, without seeking funding for either one), and would fund a visiting scientist to compare SSM/T-2 brightness temperature data with existing reanalyses. This proposal was approved and represented an important test case of the procedure (deliverable D4.42). The activity is described further below and also resulted in the report “Pre-assimilation feedback on a Fundamental Climate Data Record of brightness temperatures from Special Sensor Microwave Imagers: A step towards MIPs4Obs?” which was approved for publication as ERA Report Series 19. It is co-authored by ECMWF, DWD (Satellite-based climate monitoring), and EUMETSAT. The document contains a feedback step from the FCDR producer, the CM-SAF in this case.
In a move toward comparison of reanalysis/ ground station data, so as to determine at which temporal-spatial scales the reanalysis data add useful information, DWD started analyzing frequency distributions and scales of variability in independent ground station data and reanalysis data.
In preparation for visiting the Global Runoff Data Centre (D4.2-4) the Global Runoff Data Base (GRDB) product provided by the Global Runoff Data Centre was investigated for the procedure intended for D4.42 (assessment by comparison - not assimilation). One issue that arose was that we were not able to establish a simple mechanism to identify retrospective updates of the database values, either by version numbering or by other means - the values obtained by different users depend on the date/time of the actual downloads. This may be acceptable for research-type exercises of limited extent requiring one-off downloads, but presents difficulties for coordinated inter-comparisons and for applications requiring periodic incremental downloads (e.g. near-real-time reanalyses). To explore these issues further, the Core-Climax System Maturity Matrix has been applied to the GRDB - done jointly with a colleague from the University of Uppsala.
A historical dataset has been evaluated following the procedure intended for D4.42 (assessment by comparison - not assimilation). It is a dataset of radiances collected by the IRIS instrument onboard the Nimbus-4 satellite in 1970-1971.
4.2: “Assessing CDR data quality by assimilation into reanalysis”
Task 4.2 complemented the activities of Task 4.1 in producing Deliverable D4.42. In support of this, EUMETSAT worked on 2 sets of input data for the planned analysis, drawing on activities performed outside the CORE-CLIMAX project:
• A quality controlled SSM/T-2 data;
• An analysis on homogenising Meteosat First Generation (MFG) infrared (IR) and water vapor (WV) channels based on High-resolution Infrared Sounder (HIRS) measurements
both served as input to Task 4.2. In support of this task, deliverable D4.42 EUMETSAT and ECMWF worked to transfer by FTP a quality controlled SSM/T-2 data set generated under the auspices of EU FP7 ERA-CLIM into the ECMWF File Storage system. This dataset covers approximately 35 satellite-years, DMSP F-11 (1993-2000), DMSP F-12 (1994-2002), DMSP F-14 (1997-2006), and DMSP F-15 (2000-2008). The dataset thus exchanged was not to be assimilated into the next reanalysis production but represented an excellent test case for a procedure whereby a dataset was assessed for its quality in an experimental reanalysis.
Deliverable D4.42 was drafted in several iterations, incorporating new insights from the visits in the order they were conducted. Particularly valuable was the experience with observational CDRs provided by EUMETSAT from the SSMI series of instruments (a Fundamental CDR, FCDR, DOI-referenced, CM-SAF product) and from AMV reprocessing activities (a Thematic CDR, TCDR, derived from European operational satellites). Initial results of the comparison of the FCDR with reanalyses were presented at the 6th EUMETSAT Working Group on Data Record Generation held in Darmstadt, Germany, on 12-13 June 2014. This feedback to the CDR team was highly appreciated and the document (in preparation at the time) formalized the process.
Of the remaining visits formulated for WP4, D4.1-2 D4.2-3 D4.2-6 D4.2-7 D4.3-2 and D4.3-4 were conducted by a Core-Climax Coordination Meeting Towards a Global Archive of Historical In Situ Snow Data, held at ECMWF on 17-19 November 2014. Among those invited were Eric Brun (Meteo-France), Matthew Menne (NOAA NCDC), Olga Bulygina (World Data Centre for Meteorology from the Russian Federation), Bob Dattore (NCAR), and Ross Brown (Environment Canada). The Chinese Meteorological Administration were also invited and investigated whether they could send someone to this meeting (on the understanding that a separate bi-lateral visit could be arranged if necessary). Having made originally provisions in the project plan for both visits and hosting events, we adjusted as anticipated in the DoW to whichever proved most practical for each. At the time, out of all the 11 exchanges conducted or committed, only 3 consisted in hosting, the other 8 had been visits. So hosting all the 8 remaining visits was a fair adjustment.
During initial consultation with the various parties, the invitation process revealed that some large institutions see the setting up of a new, uni-variate archive (here, of snow depth or snow fall, and snow water equivalent) as a threat to their large multi-variate archives. The fear is that large archives with aligned records would be gradually disintegrated into smaller entities, each covering only isolated aspects of the observational problem. This concern is fully legitimate as eventually reanalysis attempts to solve the representation of past time-series as a fully multi-variate problem. To win participation of the sceptics to this initiative we indicated that 1) thematic problems (snow here) need to be solved thematically (i.e. by bringing over snow experts around the discussion table), 2) it is unlikely that any of the existing archives covers all the snow data found in the others (this was also agreed by everyone), and 3) with today's technologies there are ways to avoid physically copying large data amounts and forfeiting data ownership, as long as the data are uniquely identified in each collection and a central catalogue can be set-up to allow finding/discovering data (which could then still be served by the relevant, currently existing archive).
Following the extensive preparations, including adjustments to the candidate participants contacted above, ECMWF hosted the “Core-Climax Coordination Meeting Towards a Global Archive of Historical In Situ Snow Data” organized in support of WP4, from 17 to 19 November 2014. This meeting was supported by Core-Climax visits D4.1-2 (Expert invited: Eric Brun, from Météo-France), D4.2-3 (Expert invited: Matthew Menne, from the U.S. National Oceanic and Atmospheric Administration National Climatic Data Center), D4.2-7 (Expert invited: Steve Worley from the U.S. National Center for Atmospheric Research, also representing the National Snow and Ice Data Center), D4.3-2 (Expert invited: Yu Yu from China Meteorological Administration), D4.3-4 (Expert invited: Ross Brown from Environment Canada), and D4.2-6 (Expert invited: Olga Bulygina, from the World Data Centre for Meteorology Federal Service for Hydrometeorology and Environmental Monitoring of Russia). One visit which had been scheduled was cancelled at the last minute (D4.2-6) with the Expert failing to get a visa in time for travel, in spite of our efforts to send all the paperwork more than 2 months before the meeting (ECMWF sent the invitation letter on 10 September 2014 for a meeting starting on 17 November 2014); the Expert sent presentations that were given at the meeting on her behalf by a colleague from the same institution, Alexander Sterin.
The meeting attracted a large audience, including from the project itself (FMI) but also from other interested entities such as the Global Precipitation Climatology Centre. The meeting ran smoothly over the course of 3 days. All presentations as well as the meeting report have been posted by ITC on the Core-Climax project website, at http://www.coreclimax.eu/?q=Snow. The report starts by a one-page executive summary of ‘key points’, for those interested in fast reading of the main outcomes.
4.3 “Provision of state-of-the-art reanalysis gridded data for use as ancillary information in CDR generation”
The deliverable for Task 4.3 was the document D4.41 “Procedure for feeding back improved ancillary data to assist CDR updates” which underwent a number of drafts. The procedures developed during the Task and were informed by a number of working visits that enabled consultation on proposals and practical testing of infrastructure prototypes.
ECMWF visited KNMI who uses global reanalysis data as input to regional reanalysis activities (process to derive CDR at regional resolution). A visit report document was issued. The key findings were that regional-scale CDR activities would benefit from the following information from global reanalyses:
• information about random and systematic uncertainties,
• ensemble of reanalysis realizations,
• information about atmospheric composition (~40-year timespan) and potentially atmosphere-ocean coupling
• regular presence of mechanisms to monitor/improve co-ordination
ECMWF visit D4.1-3 was also conducted, from which a visit report was issued. Addressing about 200 attendees Paul Poli delivered the opening presentation of the European Wind Energy Association (EWEA) Technical Workshop Resource Assessment 2013 in Dublin, and presented how the global reanalyses are conducted. In return, he collected, from panel and offline discussions, the following key information points, relevant to realize user needs and understand the value of global reanalyses for producers of kilometre-scale products from the wind energy community:
• The wind energy community are advanced users of observational and climate reanalysis data records.
• Members of this community generate kilometre-scale local products over 10-20 years from global reanalyses to cover both climate and short time-scales, downscaling with meso-scale and micro-scale models.
• They would appreciate improvements in the horizontal and temporal resolution of global reanalysis products (like the current MERRA reanalysis 2D fields: hourly).
• The wind community would realize cost savings if current reanalyses of the last decades could be delivered with timeliness shorter than the current 2 to 3 month delay.
• A global reanalysis at a resolution of 10~km or better would better serve the wind energy community.
• The wind community experience in using reanalysis and observational records is distributed in dozens of consultancy companies, with expertise in data mining, quality control of both observational and reanalysis time-series, and optimal combination or merging of observations with climate reanalysis data records.
Additional information was gathered from ECMWF visit D5.3-1 regarding how regional-scale CDR activities would benefit from the following development of global reanalyses. In the view of a climate service processing chain, their needs are:
• access to observational quality control information (e.g. first-guess checks, bias corrections), as being developed in ECMWF’s Observation Feedback Archive,
• mechanisms to exchange knowledge, especially about observations,
• ensembles of boundary conditions from global reanalysis datasets.
The product of Task 4.3 deliverable D4.41 was finalized by ECMWF, drawing on findings of visits D4.1-2 and D4.1-3 as well as direct exchanges with selected projects from the European Space Agency (ESA) Climate Change Initiative (CCI).
4.4: “Guidance of CDR updates”
The deliverable for Task 4.4 was the document D4.43 which underwent a number of drafts and was delivered on 22 December 2014. The guidance developed during the Task and was informed by a number of working visits that enabled consultation on proposals and refinement of the ideas. Some of these are highlighted here:
Visit D4.3-6 to ECMWF by Dave Santek from CIMSS was carried out, and a comprehensive report summarizing the visit findings was drafted. Representatives from EUMETSAT were also able to participate (under funding external to WP4) to enhance co-ordination with analogous European activities.
The visit determined that two important satellite instrument datasets have now matured:
• 30 years of imagery data from the AVHRR instruments have been reprocessed into atmospheric motion vector (AMV) wind products by CIMSS and EUMETSAT. (This covers 11 satellites, from NOAA-7 in 1982 to NOAA-18 in 2011, as well as MetOp-A in 2007-2012)
• EUMETSAT has reprocessed 20 years of Meteosat First Generation AMVs for the period 1982-2001 and reprocessing Meteosat Second Generation AMV covering 2004-date is ongoing
The visit also determined that several programmatic issues need to be tackled to improve the reprocessing of imagery data into atmospheric motion vector (AMV) winds and serve the needs of reanalyses:
• Co-ordinated effort to better understand the implications of differences between AMV reprocessing methods at different centers
• A unified global AMV processing code, supporting both real-time and historical processing for all satellites, would allow consistency in regional datasets, applying algorithms of choice whenever needed, and possibly realizing synergies and cost savings by consolidating expertise in developments. Co-ordinated scientific/technical development would be required.
• Reprocessing of additional long-term imagery datasets could be extremely valuable to cover recent satellite missions which do not include AMV processing as a main mission obligation (e.g. MODIS), and to extend backward the start of the “satellite era” in reanalyses (e.g. GOES-1 to -7, Nimbus)
Visit D4.3-6 was planned at NOAA Headquarters, with a view to discuss the issue of GOES reprocessing, which had never been conducted so far. However, in May 2013 we heard that CIMSS received internal funding to reprocess NOAA GOES imagery data. We thus coupled this visit with that of a scientific expert in D4.2-2 from the same institution to discuss all aspects of this reprocessing.
In support of deliverable D4.43 Visit D4.3-3 was carried out. It has enabled to better understand how to feed information about upper-air observation quality back to the data provider who assembles these observations from paper and microfilm records.
The report from visit D4.3-3 was drafted and is available from the Core-Climax co-ordinating team. The main conclusions are that the experience gained at a non-reanalysis producer center (U. Vienna) and at ECMWF using the in situ CDR (CHUAN V1.7) and its recent ERA-CLIM extension need consolidation before feeding back to the data producer. Information on the types and makes of historical radiosonde ascents found in a metadatabase (the so-called VAPOR database) will be fed back to the data producer so as to aid future users of the CHUAN CDR, e.g. to validate break-point detection in the CHUAN upper-air network, and could also potentially be used to support a self-adaptive bias correction scheme in the forthcoming new reanalysis at ECMWF. Generally this illustrates the importance of “detective work” on the metadata to improve the quality of a CDR.
Work on preparing large swaths of sample observation feedback from reanalysis was progressed to support the procedure delivered in D4.43.
ECMWF and EUMETSAT exchanged a document listing the satellite CDRs to be used in future reanalyses.
The ECMWF visit D4.3-5 to NASA Headquarters produced a report. The main conclusions are that Obs4MIPs cannot grow into a full-fledge super-observation archive because of its founding principles, which are to bring observations to the modellers in the shape they are used to, namely gridded products. This precludes extending Obs4MIPs to in situ sparse observations or observation-only data types such as brightness temperatures. A logical reading of this limitation would be to start thinking of a reverse approach to map systematically climate and reanalysis model estimates into observation space.
WP5 Intercomparing reanalysis results
In WP5 a worldwide survey about user needs in private and public sector both up- and downstream in regard to reanalyses and climate services was made. Practical means for intercomparing reanalyses, using feedback data and assessing black list were identified in wide international collaborations. WP5 produced 3 project reports, 4 posters, 6 abstracts and 6 submitted manuscripts. Two of the manuscripts are now published as peer-reviewed articles, two are under revision and two are rejected. WP5 ongoing work and results were presented in workshops, conferences and meetings worldwide. WP 5 is expected to have a high impact on future operational climate change services.
5.1 “Summarizing uncertainties related to reanalyses based on a survey on user requirements supported by peer reviewed literature”
Science: Data from the survey: 2578 respondents answered the check-box questions and several hundred also wrote free form responses. Publications: one report, one poster, two abstracts, one rejected manuscript (Gregow et al. 2013, Figure 7 shows an example), one published manuscript (Gregow et al. 2015a), one submitted manuscript (Gregow et al. 2015 b). Presentations: several oral presentations.
Task 5.1 focused on identifying uncertainties in reanalysis products and how these affect climate services/ scientific community. To be able to build a survey, an investigation on earlier surveys concerning reanalyses or climate services and/or research were made.
Figure 7: Correlation of ERA-Interim (1979-2012) wind gust product to sum of geostrophic and ageostrophic isallobaric wind (SWIND) from the paper Gregow et al. 2013 that was rejected. The figure is produced by Pauli Jokinen.
In Task 5.1 an online survey entitled “Reanalysis User and Application Survey” was made. The final version of the survey included only 11 sections to be answered. These sections resulted in around 360 multiple-choice questions (Fig. 8).
Figure 8: Description of the survey sections and emphasis of these. This figure is from the manuscript submitted by Gregow et al. to BAMS in June 2015. That manuscript focuses on the boxes shaded by light blue color.
The survey was distributed by WMO globally to regional offices, by help of NOAA and Paul Poli it was posted on the web page reanalysis.org. Additionally FMI, ECMWF and DWD distributed the survey nationally through their contacts, e.g. the Copernicus user forum, and for the known stakeholders. The links were posted on reanalysis.org ecmwf.int and ewea.org. The first analysis results of the questionnaire were distributed to the partners on 12th of February 2014.
An analysis of the survey and a literature review of reanalyses were compiled. When the analysis and the report D5.52 entitled “Reanalyses and user needs with respect to Climate Change Services” was in its final form by 31.3.2014 and was immediately sent to ECMWF by personal request from ECMWF director general Alan Thorpe to Hilppa Gregow on the Copernicus workshop 17.-18.2.2014.
Foreground of the survey material is shared among FMI, DWD and ECMWF. It is an excel file including all responses that can be analyzed in any statistical way. We have written a notification concerning Intellectual Property Rights in the excel file.
5.2: “Collecting and synthesizing information from current reanalyses”
Science: Publications: One report, two abstracts, two posters, Kaiser-Weiss et al. 2015 (accepted).
Task 5.2 focused on the process of building a database of diagnostics for reanalyses to allow quantitative comparison between reanalyses, as well as between reanalyses and CDR and other Copernicus observations. The aim was to yield information that assists users in deciding which reanalysis product might be most suitable for their particular application. With increased resolution of the reanalyses and applications, there is a growing need for evaluation with observations.
In Task 5.2 ECMWF and DWD synthesized the information from current reanalyses and investigated how the reanalyses can best be compared. The analysis of feedback statistics was identified as the appropriate method to identify the temporal-spatial scales for confident usage, as well as for comparison between different analyses. A methodological overview was carried out over formal processes for intercomparison of regional and global reanalysis. A pilot study was performed to demonstrate feasibility and potential value of feedback statistics. The latter yield the fit of reanalyses to assimilated observations, and are thus suitable as a means for comparing regional with global reanalysis. These findings were reported in D5.53 “Procedure for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs” This report presents a set of procedures for comparing reanalyses, and comparing reanalyses to assimilated observations and CDRs. Five categories of comparisons were identified. These were accompanied by two complexity ratings. The first rates the complexity of conducting the procedure (simple, moderate, difficult), and the second rates the complexity of interpreting the results (simple, moderate, difficult):
1. descriptive product comparison (simple to conduct, simple to interpret)
2. comparison with third party observation-based CDRs (moderate to conduct, moderate to interpret)
3. inter-comparison between different reanalyses (moderate to conduct, moderate/difficult to interpret)
4. thematic comparison (difficult to conduct, difficult to interpret)
5. internal metrics comparison (difficult to conduct, moderate to interpret)
5.3: “Summarizing Tasks 5.1 and Tasks 5.2 and highlighting the uncertainties and gaps that need addressing to deliver climate services”
Science: Publications: one report, one poster, four abstracts, one accepted manuscript (Jokinen et al. 2015), one submitted manuscript (Gregow et al. 2015 b). Presentations: several oral presentations.
In preparation of a deeper analysis of the survey a new manuscript (Gregow at al. 2015b) focusing on future climate services as supported by reanalyses was started (in June 2014). At that time also the groups Water, Energy, Agriculture and Forests were chosen to be examined more closely. The 2578 respondents were mostly users of global reanalyses and in particular ECMWF, NCEP and NASA reanalyses. An innovative procedure was devised to organize users’ requests into classes of priorities of similar significance (Table 2a&b). The groups (excluding energy) emphasized the importance of activities such as seasonal forecasting and verification, applied weather and climate research for impact assessment and/or statistical impact analyses for improving weather warnings and their criteria to be part of the future climate services (Table 2b).
Table 2a. Statistical significant differences for average scores for different statements about reanalysis data between two groups of respondents, namely respondents having vs. not having fresh water resources and management (WAT), agriculture and food production (AGR), forests (FOR), or energy (ENE) as a field or a subject of their work. For details, see the Gregow et al. (2015b, submitted to BAMS)
If a statement is agreed more strongly by the subgroup members than by the outsiders, then the following notation is used: ^ (p<0.05) ^^ (p>0.01) ^^^ (p<0.001). If it is disagreed more strongly, then v (p<0.05) vv (p>0.01) vvv (p<0.001).
Table 2b. Statistical significant differences for average scores for different suggestions for future climate service tasks or activities between two groups of respondents, namely respondents having vs. not having fresh water resources and management (WAT), agriculture and food production (AGR), forests (FOR), or energy (ENE) as a field or a subject of their work. For details, see the footnotes. (Copy from Gregow et al. 2015b submitted to BAMS).
If a statement is agreed more strongly by the subgroup members than by the outsiders, then the following notation is used: ^ (p<0.05) ^^ (p>0.01) ^^^ (p<0.001). If it is disagreed more strongly, then v (p<0.05) vv (p>0.01) vvv (p<0.001).
Next we show examples of use rate of data (Tables 3 and 4) that are part of the supplementary material in the submitted manuscript by Gregow et al. 2015b.
Table 3. Applications and methods used among sectors WAT (water), AGR (agriculture), FOR (forests), ENE (energy) and ALL (all 28 sectors). “R” indicates reanalysis data and “G” gridded in situ data. Color codes are analysed separately for ALL and each field of work for “R” and “G”. Green indicates the highest percentage (67-100%) of responses, yellow indicates the medium share (34-66%) and red the lowest share (1-33%) of responses within each main field of work. (from Gregow et al. 2015b submitted to BAMS)
Table 4. Essential climate variable (ECV) distribution concerning use of atmospheric and terrestrial reanalysis data (RD) and other data (OD) (in situ, radar, satellite). For the color codes see Table 3. (from Gregow et al. 2015b submitted to BAMS)
In addition to the manuscript that was prepared a synthesis report summarizing Tasks 5.1 and 5.2 and thus Deliverables D5.51-D5.53 was made. This task was coordinated with WP3 (task 3.2: Analysis of ECV/CDR validation network and strategy) where the consistency of gridded data was examined (FMI also involved). ECMWF visit D5.3-1 determined that the ozone sonde data holdings of the World Ozone and Ultraviolet Data Centre (WOUDC) remain the prime candidate for assessment of atmospheric ozone in-situ data within a reanalysis context.
Additional information was gathered from ECMWF visit D5.3-1 regarding how regional-scale CDR activities would benefit from the following development of global reanalyses. In the view of a climate service processing chain, their needs are:
• access to observational quality control information (e.g. first-guess checks, bias corrections), as being developed in ECMWF’s Observation Feedback Archive,
• mechanisms to exchange knowledge, especially about observations,
• ensembles of boundary conditions from global reanalysis datasets.
Deliverable D5.53 was influenced by W4 and this has been taken into account in the drafting of a white paper outlining the need for reanalysis intercomparison guidance and a task team at an international level. One indirect outcome was the creation of a WCRP Task Team for Intercomparison of ReAnalyses.
WP6 Dissemination, outreach and capacity building
This workpackage was about “Dissemination, outreach and capacity building” and as such there are no Science and Technology results to report on.
Potential Impact:
WP1 Project management
This workpackage concerned the project management and as such there are no “Impact and dissemination activities nor exploitation of results” to report on.
WP2 European ECV capability and structured ECV process
Work and results of Task 2.1 had specific impact at different levels of the European community engaged in the production or usage of climate data records. Impact was realised at national, European and global level (see below for specific examples). The impact mostly became possible by the way how the maturity assessment was conducted by the CORE-CLIMAX project which was highly inclusive for the participants and also had a big educative process behind that was explaining in detail the benefits of performing a maturity assessment.
Specific examples for currently known impacts at national level are:
• Deutscher Wetterdienst as one of the CORE-CLIMAX partners has used the maturity assessment to analyse its in situ data record provision. This had the consequence of introducing a version control for data sets and also to provide annual updates of the data records;
• The Met Office, United Kingdom, as participant in the CORE-CLIMAX assessment has realised the importance of software maintenance for the systems generating very prominent climate data records such as the Hadley Centre Sea Surface Temperature data set (HadSST). The consequence of that analysis was a recruitment of a software engineer to improve the software maintenance in the future;
• The Deutsche Klimarechenzentrum GmbH, Hamburg, Germany has used the CORE-CLMAX maturity model to start the definition of a similar maturity model for climate model data stewardship;
At European level:
• At the beginning of the CORE-CLIMAX project, ESA was pursuing similar activities within the ESA Climate Change Initiative (ESA CCI). The CORE-CLIMAX project was successful in insisting that ESA should join the activities in CORE-CLIMAX which ended in almost complete participation of ESA CCI data records in the CORE-CLIMAX assessment;
• The Framework 7 project QA4ECV has taken up the CORE-CLIMAX maturity matrix into its more general Quality Assurance framework. The maturity matrix will be applied towards the end of the project to assess all climate data records produced as part of QA4ECV;
• The Horizon 2020 project GAIA-CLIM is using the maturity matrix development within its concept to evaluate the adequacy of surface based reference measurements for the characterisation of satellite measurements and derived products. It is within the remits of this project that a further utilisation of the maturity matrix approach for in situ data will be further fostered;
• The Horizon 2020 project EUSTACE is using the maturity matrix approach to monitor the development of global surface temperature climate data records to be developed within the project. This will enable to clearly demonstrate progress due to the project work;
At global level:
• The WMO Sustained Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM, http://www.scope-cm.org/) initiative uses the maturity matrix to monitor progress for its ten global projects on climate data generation;
• In the context of the preparation of the next Climate Model Intercomparison Project (CMIP-6) the obs4mips activity, that prepares the provision of evaluation data sets for climate modelling groups, uses the CORE-CLIMAX Maturity Matrix as starting point for the development of a so called Model Evaluation Readiness Level Matrix to support the evaluation of the suitability of proposed climate data records for climate model evaluation;
• The CEOS/CGMS Working Group Climate has the potential to assess the content of the global inventory of GCOS Essential Climate Variable data records (currently 220 data records) using the CORE-CLIMAX methodology. As most of the data records produced in Europe were participating in the CORE-CLIMAX assessment the results will be added to the inventory by EUMETSAT (which takes responsibility of the inventory on behalf of CEOS and CGMS) after the project end.
Some potential impacts could not be realised due to resource issues. For instance the usage of the System Maturity Matrix and the distribution of the assessment results using the technology developed in the parallel project CHARMe could not be realised due to the short duration of the projects. Both projects were and the participants are still very interested to include CORE-CLIMAX assessment results into the commentary meta data feed of CHARMe. It is suggested that the results of both projects shall be at least utilised in the framework of the Copernicus Climate Change Service but it could be useful for other Copernicus services as well.
With respect to the Copernicus Climate Change Service the systematic application and further development of the CORE-CLIMAX system maturity matrix and the spin-off Application Performance Metric were strongly endorsed by participants of a Copernicus Climate Change Service workshop on the content of the Climate Data Store in July 2015. However, currently no implementation plan, e.g. as part of the Quality Control & Evaluation (QC&E) pillar, exists for this functionality. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation in the climate change service.
Compared to the impact of Task 2.1 the impact of Task 2.2 is not really measurable at the end of the project. This is due to the fact the associated deliverable D2.26 is only released at the end of the project. In addition, the relative big success of the maturity assessment and associated impact including all required communications around it has also led to a reduced community engagement for Task 2.2. This is mainly due to the relatively short duration of the project that does not fit long-term community engagement into the production of a document on the structured processes. However, the structured process diagrams are useful to other EU FP-7 and H2020 projects. For instance the QA4ECV project has started drafting so called traceability diagrams that sketch execution pathways for algorithms which represent further sublevels of the process diagrams provided by CORE-CLIMAX. In addition the Horizon 2020 project FIDUCEO aims at publication of so called best practice documents that can certainly make use of the structured process diagrams developed by CORE-CLIMAX.
WP3 Validation process
The most evidenced impact is indicated in the 37th CEOS-WGCV plenary minutes, which was quoted as: “Before developing a cross-cutting project on maturity matrices across CEOS, WGCV needs to look at the work of the CORE-CLIMAX group where they are already looking into maturity matrices approaches including extensions to the original Maturity Matrix, called the “System Maturity Matrix” and the Application Performance Matrix (APM). They are also looking at Uncertainty. What is the interest for WGCV in maturity matrices? What can WGCV learn from the CORE-CLIMAX study? Is a simple number based rating sufficient to meet WGCV’s requirement and those of the user, or is a different approach warranted?”
The CORE-CLIMAX efforts in developing a quality indicator for Cal/Val was recognized by CEOS-WGCV and provoked discussions among the community on how to incorporate this concept into the practical operation of CEOS-WGCV.
Dissemination has been done via participating in various of workshops, conferences and plenaries:
1) The GEPWs (GEO European Project Workshops, Barcelona & Athena, 2013 & 2014);
2) IGWCO (GEO Integrated Global Water Cycle Observations Community of Practice (CoP) Meetings, 2013 & 2014);
3) The 37th CEOS WGCV (working group on calibration and validation) Plenary (2014);
4) The GEO-X Plenary and the 2013 Ministerial Summit (2013);
5) The CORE-CLIMAX Collocation Meeting, Brussel (2013)
6) The CORE-CLIMAX ECV Capacity Assessment Workshop, EUMETSAT (2014)
7) The CORE-CLIMAX Capacity Building Workshop in Finish Meteorological Institute (FMI), Helsinki, Finland (2014)
8) The CORE-CLIMAX Capacity Building Workshop as the side event of the 14th EMS Annual Meeting & 10th European Conference on Applied Climatology (ECAC) (2014)
9) The CORE-CLIMAX Collocation Meeting, Brussel (2014)
10) The main Science and Technology result was published in an international peer-reviewed journal (International Journal of Applied Earth Observations and Geoinformation), 2015
WP4 reanalysis feedback mechanism to CDR update
As far as the impact concerns, WP4 Deliverables influenced the WP5 Deliverable D5.53 and this has been taken into account in the drafting of a white paper outlining the need for reanalysis intercomparison guidance and a task team at an international level. One indirect outcome was the creation of a WCRP Task Team for Intercomparison of ReAnalyses.
The ideas spelled out in the deliverable D4.42 “Design of support infrastructure for CDR quality assessment in a reanalysis environment” promoted the value of the observational feedback and collocation, and were instrumental in the virtual observatory concept now being worked in an H2020 project.
Dissemination activities, apart from the listed publications, workshops and conferences in Template A2 in this work-package consisted for a large part of working visits to several international institutions. These are listed in Template A2 under “visits”.
WP5 Intercomparing reanalysis results
WP5 has potentially still a high impact on development of Copernicus Climate Change Services (C3S). These are the main conclusions that were made during WP5 concerning C3S:
Data Store: Users of reanalysis products would benefit from improved data access. Explanations for nominal resolution and feature resolution should be available. Enhanced tools for data discovery, extraction and subsetting would reduce data transfer volumes. Ready-made graphical products would make the service more time efficient. Sectoral Information System: Up- and down-stream users have varying needs in the private and public sectors and thus a constant dialogue between the users would advance understanding of the changing needs. Faster release of the datasets would be a priority for providing better 24/7 climate services. Evaluation and Quality Control: The current level of uncertainty regarding biases in reanalysis datasets and resolution effects between the reanalyses in both the time and space dimensions, are identified as impediments to climate service and research. Five categories of comparison were identified which are fundamental in evaluating and quality-controlling the use of reanalyses. The categories are: descriptive product comparison; comparison with third party observation-based CDRs; intercomparison between different reanalyses; thematic comparison; internal metrics comparison. Each category benefits from the findings of the other categories. Outreach and Dissemination: Targeted training (e.g. e-learning) for different user subgroups would be appreciated. Examples of climate service case studies using alternative datasets as input could be useful in capacity building. A more comprehensive set of diagnostics would help the users to factor in the uncertainties inherent in the reanalysis data. Capacity-building for the five identified EQC intercomparison categories, with special emphasis on targeting the understanding of internal reanalysis metrics, is a priority for outreach.
Co-ordination: A need for co-ordination is clear, not least to mobilize the participants, to collate/disseminate findings, and to promote the sharing/use of common software tools. Efficiency is also enhanced when reanalysis intercomparisons are preceded by standalone evaluation of individual reanalyses under the responsibility of each producer. It could be argued that a minimum level of self-evaluation quality should be reached and documented prior to undertaking extensive intercomparison – this level could be characterized by achieving threshold levels (to be defined) in the Core-Climax System Maturity Matrix. More generally, the timeline of the intercomparison workflow will need a set of triggers (decision points) in order to establish when the various activities can be usefully undertaken.
Operational services Climate change services are essential for supporting decision making. In addition to inter-comparing and developing reanalysis products for boosting the delivery of climate services, interaction between science, business and policy making should be improved. For determining the cause for the new observed records, it would be crucial to have the observation, weather and climate services working closely using e.g. “mobility as a service” approach. By combining the chain from data assimilation to reanalysis production, product development, interactive tools, operational climate service, climate change research and communication, it will be possible to upgrade the climate services to an operational phase and thus advance the development Copernicus services as part of the Global Framework for Climate Services.
WP6 Dissemination, outreach and capacity building
This workpackage concerned the “Dissemination, outreach and capacity building” activities and had as objectives the coordination with (European and international) stakeholders and a dissemination and/or outreach activity.
Thematic coordinating with international activities took place through the organization of the following coordination forum workshops:
▪ Coordination forum Workshop 1 (D6-61): Kick-off Meeting “COordinating Earth observation data validation for RE-analysis for CLIMAte SercieS” (CORE-CLIMAX) at ITC premises, 17-18 January 2013, Enschede, The Netherlands
▪ Coordination forum Workshop 2 (D6-62): Co-location Meeting EC FP7, ESA and EUMETSAT ECV-related Projects at REA premises, 15-16 May 2013, Unit S2 – Space Research, COV2 19-SDR2, B-1049 Brussels, Belgium
▪ Coordination forum Workshop 3 (D6-63): CORE-CLIMAX Essential Climate Variable (ECV) Capacity Assessment Workshop at EUMETSAT headquarters, 21-23 January 2014, Darmstadt, Germany
▪ Co-location Meeting “Coordinating European efforts in generating climate data records for Copernicus Climate Change Services (C3S)”, at REA premises, 15-16 January 2015, Unit B1 – Space Research, COV2 17/149, B-1049 Brussels, Belgium
In addition thematic coordinating with international activities was also done by participation in international workshops and conferences, which are listed in Template A2.
Apart from the dissemination done by (scientific) publications, presentations, meetings, visits, posters, abstracts, an important component consisted of the organization of two Capacity Building workshops:
▪ 1st Capacity Building Workshop (D6-65), 19-20 March 2014 at FMI premises, Helsinki, Finland
▪ 2nd Capacity Building Workshop (D6-66), 10 October 2014 at ESM/ECAC conference, Prague, Czech Republic
Details hereof can be found on the project’s website and in Template A2.
Project as a whole: The CORE-CLIMAX paradigm
Following up on a previous collocation meeting of mid-2013, a second collocation meeting was organised by the CORE-CLIMAX project at the Research Executive Agency in Brussels on 15 and 16 January 2015. The aim of this meeting was to coordinate with the different H2020, FP7, ESA & EUMETSAT ECV-related projects the processes of generating, validating and updating (including reanalysis) ECVs and to understand the gaps between the ECVs and user needs in the newly established Copernicus Climate Change Services (C3S). Specific questions related to these processes were addressed: what needs to happen next, what would be needed for that to happen, which are issues of concern, and how can these potentially be solved?
European structured ECV capability
Concerning the “European structured ECV capability” it is important to look back on the success from the Joint Research Centre (JRC) workshop in 2009 to the establishment of the C3S today. For the establishment of the C3S the obstacles are the transition to operations and funding, e.g. issues with Long Term Data Preservation (LTDP) versus Climate Change Initiatives (CCI). Also, CDRs generated from scientific activities such as the CCI and the projects will need to be funded by the service or transitioned to other operational budget such as the Satellite Application Facilities (SAFs).
For the transfer of FP7/H2020 project results into C3S there is a need to establish and maintain a regular dialogue between C3S and the projects to enable best uptake as possible. Concerning the transfer of knowledge from projects to operational activities such as SAFs, best practice guides need to be brought into work, continuous efforts should be invested to achieve this. Such efforts should not and is not feasible to rely on one-off projects.
Validation process in ECVs
A better engagement of metrologists is needed to define best practices. It is recommended to link to the European metrology community as much as possible and to secure funding for independent pre-launch and long term post-launch Calibration/Validation (CAL/VAL) studies.
Innovative validation methods and infrastructures are needed to permit a cross-cutting quality monitoring (i.e. considering several variables at the same time and assessing the consistency between variables).
It is recommended to (1) develop the use of data assimilation for validation as the obtained analyses account for the synergies of the various upstream products and provide statistics that can be used to monitor the quality of the assimilated observations; (2) create or consolidate in situ reference data sets by measuring as many variables as possible at the same location, using several relevant sensors.
Finally, there is a need for better international collaboration. The European efforts in developing reference datasets need to be coordinated with the international community in as much as global products considered.
Reanalysis feedback mechanisms
The group recognized that reanalysis-based CDR assessment has proved valuable and endorsed the idea of widening its use.
It is recommended to have short feedback loops in the assessment process and to facilitate 3rd-party access to the quantitative datum-level feedback information.
The main needs are for (1) sustained infrastructure to compute/archive/analyse reanalysis feedback, (2) more personnel with the skills to conduct reanalysis-based CDR assessment, and (3) systems for the CDR providers to make visible to users the assessment findings and any ensuing CDR update plans.
Lack of capacity is a potential obstacle, which could be addressed by programs to build more capacity.
Inter-comparing reanalysis results
The main objective of inter-comparing re-analyses is to estimate the uncertainties of the various reanalysis products.
This is needed to help the users to decide whether re-analyses are generally fit for the user’s purposes, to help users to decide which product to use, and last but not least to build trust in the reanalysis products.
Typically, users firstly require an overview of available products. Possibly they are looking for any alternatives to familiar products. In any cases, they need help to pick the right product. Maybe they just think of the easiest to download or the highest resolution and would benefit from being pointed to other criteria. In the future, specially targeted inter-comparisons could provide such criteria.
It is important to note that usually users do not have the resources to do inter-comparisons, or to estimate uncertainties for their particular applications on their own. They might lack the know-how or the resources to do so. Thus, there is a gap to be bridged for a full use of C3S re-analyses data sets, where re-analyses inter-comparisons could help.
A concern is that various users have quite different applications and backgrounds. In other words, we need various bridges and have to cover a range of inter-comparison methods and independent reference data. Strengthening the community efforts could help to address a wide range of questions related to uncertainty and could also increase user involvement. Inter-comparison of re-analyses would as such feed into C3S guidance and user support.
Beyond CORE-CLIMAX – final Recommendations
CORE-CLIMAX can be compared to ISO procedures where it comes to quality standards concerning CDR/ECV/reanalysis products, i.e. time series. As such the project constitutes an important building block for climate services.
With respect to the C3S the systematic application and further development of the CORE-CLIMAX system maturity matrix and the spin-off Application Performance Metric were strongly endorsed by participants of the C3S Climate Data Store workshop in July 2015. However, currently no implementation plan, e.g. as part of the Evaluation & Quality Control (QC&E) pillar, exists for these functionalities. The CORE-CLIMAX project highly suggests involving the knowledge gained by the CORE-CLIMAX consortium in future implementation of C3S. Particularly, the SMM and APM can provide starting points for the EQC function of the C3S to assess the technical and scientific quality of the service including the value to users.
More training is required in order to capitalize on what has been achieved in CORE-CLIMAX project and more generally for the success of the C3S, which hints both at improving university education as well as other education activities (e.g. ESA training activities).
Currently many CDR’s are not ready yet for application, consistent high-resolution data is a challenge and more research is urgently needed. Linking ECV’s to Essential Water Variables is the next step from closing the gaps from CDRs to sectorial applications, the generation of different data sets should be related to high impact phenomena like severe droughts and flooding events.
At the closure review meeting at Research Executive Agency in Brussels, on 22 June 2015, the external reviewer commented that CORE-CLIMAX has produced results way beyond its original scope. It has set a new paradigm for data handling and data quality regarding the production of climate services. These issues are very sensitive and the best practice approach of the project has created community acceptance.
List of Websites:
Public website address: www.coreclimax.eu
Documents connexes
- final1-figure8.pdf
- final1-table3.pdf
- final1-figure4.pdf
- final1-figure1.pdf
- final1-table2a.pdf
- final1-coreclimax-final-report-sections-41-and-42-text-figures-tables.pdf
- final1-figure6.pdf
- final1-figure3.pdf
- final1-figure7.pdf
- final1-templatea2.pdf
- final1-table4.pdf
- final1-figure2.pdf
- final1-table2b.pdf
- final1-table1.pdf
- final1-figure5.pdf
