Periodic Reporting for period 3 - FIDUCEO (Fidelity and Uncertainty in Climate data records from Earth Observations)
Periodo di rendicontazione: 2017-09-01 al 2019-08-31
In many Earth observation programmes internationally, considerable investment is rightly made to inform global society about environmental changes relevant to climate. Given the high societal stakes, the confidence attributable to information in climate data records (technically, “uncertainty”) needs to be transparently established and communicated. Such rigour and transparency in climate data records (CDRs) needs to be based on fundamental climate data records (FCDRs) that embed similar principles.
FIDUCEO has utilised Europe’s expertise this domain to establish new standards and methodologies for accuracy and rigour in the generation of FCDRs and CDRs, with detailed, traceable uncertainty information.
Concept development has been kept stringent and practical, because FIDUCEO has applied developments to four important >20 yr-long satellite sensor series, creating four new FCDRs of widespread utility. Four CDRs addressing essential climate variables have thereby been developed, including new uncertainty quantification.
The main technical objectives were:
(i) to create FCDRs for Meteosat, AVHRR, HIRS, and the microwave humidity sounder series with physics-based harmonisation of radiance calibration, traceable propagation of estimates of associated uncertainty components and a standard, convenient format for wider use (“easyFCDR” and “easyCDR” formats); and
(ii) to demonstrate CDRs for upper-tropospheric humidity, sea and lake surface temperature, surface albedo and atmospheric aerosol with rigorous uncertainty information deriving from the underlying FCDRs.
This choice of technical objectives (i) produced data of high scientific relevance, and (ii) drove the methodological developments to be ambitious, rigorous and practical.
FIDUCEO has utilised Europe’s expertise this domain to establish new standards and methodologies for accuracy and rigour in the generation of FCDRs and CDRs, with detailed, traceable uncertainty information.
Concept development has been kept stringent and practical, because FIDUCEO has applied developments to four important >20 yr-long satellite sensor series, creating four new FCDRs of widespread utility. Four CDRs addressing essential climate variables have thereby been developed, including new uncertainty quantification.
The main technical objectives were:
(i) to create FCDRs for Meteosat, AVHRR, HIRS, and the microwave humidity sounder series with physics-based harmonisation of radiance calibration, traceable propagation of estimates of associated uncertainty components and a standard, convenient format for wider use (“easyFCDR” and “easyCDR” formats); and
(ii) to demonstrate CDRs for upper-tropospheric humidity, sea and lake surface temperature, surface albedo and atmospheric aerosol with rigorous uncertainty information deriving from the underlying FCDRs.
This choice of technical objectives (i) produced data of high scientific relevance, and (ii) drove the methodological developments to be ambitious, rigorous and practical.
Work carried out and key findings:
• Conceptualising, formalising, demonstrating and publishing a complete set of methodologies for metrology of “level 1” Earth observations (used in FIDUCEO for FCDRs, but generally applicable). Specifically, the concepts and methodologies developed are: satellite-relevant metrology vocabulary; measurement-function centred analysis of uncertainty; traceable, formalised documentation of error covariance information; mathematics for deriving user-oriented, summary uncertainty information; and a standard product format enabling exploitation by level-1 users (easyFCDR).
• Demonstrating this set of metrology methodologies for the four target FCDRs, all of which are successfully “uncertainty-quantified” and freely available in the new standard formats.
• Disseminating the methodologies through the FIDUCEO web site, including attractive, well-read blogs, online “cookbook” video tutorials and training materials, two open FIDUCEO workshops (with international attendance), training at two additional events at the request of the European Space Agency, targeted presentation at the 2017 EUMETSAT conference, presentation at many other conferences, input into the CEOS & CGMS networks of space agencies via the joint Working Group on Climate, and publication of an open access overview in a key metrology journal (which was downloaded >1400 times within 5 months of publication) as well as further detailed scientific publications.
• Long term FCDRs across many sensors need to be carefully and sensitively aligned. In FIDUCEO, this process of “harmonisation” is tackled be re-calibrating sensor radiances consistently, used advanced mathematical and computational techniques to derive new calibration parameters. The science of harmonisation differs across sensors because of their individual characteristics and scientific opportunities. For example, harmonisation of the Meteosat visible FCDRs involved innovative retrieval of the process of changing responsivity of the instruments as they degraded in time, whereas for HIRS the main challenge is using coincident observations to “daisy chain” the calibration along the series. The metrological principles underpinning harmonisation are in common for all the sensors. Harmonisation was a harder challenge than foreseen: while the Meteosat harmonisation has succeeded fully as hoped, various restrictions on the achieved harmonisation apply for the other series.
• Harmonisation tools have been developed and are available for re-use and further development. These include satellite-satellite match-up systems (to find the coincident observations needed), advanced solvers to find the harmonisation solution that best reconciles the calibration of a series of sensors, and tools to work with the inputs to and outputs from the solvers. A further freely available tool propagates uncertainty in easyFCDRs to a user-specified retrieval of a geophysical variable of interest.
•Four demonstration CDRs were derived from the FCDRs, addressing atmospheric humidity, aerosol, albedo and surface water temperatures (ST). All of these apply the metrological principle of propagation of uncertainty from the easyFCDR to the CDR, and give users advanced uncertainty information. Additionally, the ST CDR was experimentally produced in ensemble form, using innovative methods to ensure that the spread across the 10 members represents ST uncertainty at all scales and correctly estimates sources of uncertainty that could not otherwise be captured.
• Conceptualising, formalising, demonstrating and publishing a complete set of methodologies for metrology of “level 1” Earth observations (used in FIDUCEO for FCDRs, but generally applicable). Specifically, the concepts and methodologies developed are: satellite-relevant metrology vocabulary; measurement-function centred analysis of uncertainty; traceable, formalised documentation of error covariance information; mathematics for deriving user-oriented, summary uncertainty information; and a standard product format enabling exploitation by level-1 users (easyFCDR).
• Demonstrating this set of metrology methodologies for the four target FCDRs, all of which are successfully “uncertainty-quantified” and freely available in the new standard formats.
• Disseminating the methodologies through the FIDUCEO web site, including attractive, well-read blogs, online “cookbook” video tutorials and training materials, two open FIDUCEO workshops (with international attendance), training at two additional events at the request of the European Space Agency, targeted presentation at the 2017 EUMETSAT conference, presentation at many other conferences, input into the CEOS & CGMS networks of space agencies via the joint Working Group on Climate, and publication of an open access overview in a key metrology journal (which was downloaded >1400 times within 5 months of publication) as well as further detailed scientific publications.
• Long term FCDRs across many sensors need to be carefully and sensitively aligned. In FIDUCEO, this process of “harmonisation” is tackled be re-calibrating sensor radiances consistently, used advanced mathematical and computational techniques to derive new calibration parameters. The science of harmonisation differs across sensors because of their individual characteristics and scientific opportunities. For example, harmonisation of the Meteosat visible FCDRs involved innovative retrieval of the process of changing responsivity of the instruments as they degraded in time, whereas for HIRS the main challenge is using coincident observations to “daisy chain” the calibration along the series. The metrological principles underpinning harmonisation are in common for all the sensors. Harmonisation was a harder challenge than foreseen: while the Meteosat harmonisation has succeeded fully as hoped, various restrictions on the achieved harmonisation apply for the other series.
• Harmonisation tools have been developed and are available for re-use and further development. These include satellite-satellite match-up systems (to find the coincident observations needed), advanced solvers to find the harmonisation solution that best reconciles the calibration of a series of sensors, and tools to work with the inputs to and outputs from the solvers. A further freely available tool propagates uncertainty in easyFCDRs to a user-specified retrieval of a geophysical variable of interest.
•Four demonstration CDRs were derived from the FCDRs, addressing atmospheric humidity, aerosol, albedo and surface water temperatures (ST). All of these apply the metrological principle of propagation of uncertainty from the easyFCDR to the CDR, and give users advanced uncertainty information. Additionally, the ST CDR was experimentally produced in ensemble form, using innovative methods to ensure that the spread across the 10 members represents ST uncertainty at all scales and correctly estimates sources of uncertainty that could not otherwise be captured.
The FIDUCEO project has:
Integrated European scientific expertise on metrology, calibration, and retrieval science in a coherent set of traceable methodologies for the production and provision of uncertainty-quantified satellite observations (including FCDRs).
These methods are now fit and tested for wider application, including within the ongoing developments for Copernicus High Priority Candidate Missions.
Developed and promoted methods for characterising and propagating uncertainty that are already informing practice for CDRs in the ESA Climate Change Initiative, the Copernicus Climate Change Service and other European activities to generate climate information from space assets, thereby adding value to those data streams.
Generated uncertainty-quantified FCDR for historical missions key to climate studies, and derived exemplary innovative CDRs. These contribute to societal benefit areas of GEO/GEOSS.
The provision of FCDRs will enable ongoing improved CDRs for oceanic, land surface and atmospheric parameters. With this FIDUCEO directly contributes to the GEO Climate Task CL-01 Climate Information for Adaptation that foresees the delivery of improved climate data records related to GCOS ECVs.
Retrieval systems potentially applied to the FCDR outputs and also the CDRs produced within FIDUCEO can have indirect impact on the weather, water, and agriculture societal benefit areas.
FIDUCEO activities on FCDRs and CDRs are also associated to several projects of the WMO Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM) initiative where GEO is one of the stake holders.
The findings from the analysis of historic and current observations may also support the GEO Infrastructure Task IN-01 Earth Observing Systems by showcasing the best use of observations and making suggestions for future observing systems.
The longest-term legacy of FIDUCEO will be its metrological approaches to Earth observation, which are re-usable and adaptable embodiments of best-practice approaches to harmonisation and uncertainty estimation for FCDR/CDRs for the wider community.
Integrated European scientific expertise on metrology, calibration, and retrieval science in a coherent set of traceable methodologies for the production and provision of uncertainty-quantified satellite observations (including FCDRs).
These methods are now fit and tested for wider application, including within the ongoing developments for Copernicus High Priority Candidate Missions.
Developed and promoted methods for characterising and propagating uncertainty that are already informing practice for CDRs in the ESA Climate Change Initiative, the Copernicus Climate Change Service and other European activities to generate climate information from space assets, thereby adding value to those data streams.
Generated uncertainty-quantified FCDR for historical missions key to climate studies, and derived exemplary innovative CDRs. These contribute to societal benefit areas of GEO/GEOSS.
The provision of FCDRs will enable ongoing improved CDRs for oceanic, land surface and atmospheric parameters. With this FIDUCEO directly contributes to the GEO Climate Task CL-01 Climate Information for Adaptation that foresees the delivery of improved climate data records related to GCOS ECVs.
Retrieval systems potentially applied to the FCDR outputs and also the CDRs produced within FIDUCEO can have indirect impact on the weather, water, and agriculture societal benefit areas.
FIDUCEO activities on FCDRs and CDRs are also associated to several projects of the WMO Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM) initiative where GEO is one of the stake holders.
The findings from the analysis of historic and current observations may also support the GEO Infrastructure Task IN-01 Earth Observing Systems by showcasing the best use of observations and making suggestions for future observing systems.
The longest-term legacy of FIDUCEO will be its metrological approaches to Earth observation, which are re-usable and adaptable embodiments of best-practice approaches to harmonisation and uncertainty estimation for FCDR/CDRs for the wider community.