CORDIS provides links to public deliverables and publications of HORIZON projects.
Links to deliverables and publications from FP7 projects, as well as links to some specific result types such as dataset and software, are dynamically retrieved from OpenAIRE .
Deliverables
The purpose for the development of the D-PAC processor was the necessity to provide a flexible tool for the off-line processing of MIPAS level 2 data. The flexibility of the processor allows a large variety of parameters and options to be adapted to the current and future needs of the user community. This includes the retrieval of additional trace gases as well as the retrieval of partial profiles in case of cloud contamination or weak signals, including sequential or joint retrieval of species profiles. The D-PAC processor is attached to an in-house database allowing simple access to an archive system containing ENVISAT data. The generation of products can be done interactively or fully automated under computer control. Internally, the processor relies on the KOPRA forward model (a highly precise line-by-line radiative transfer code written by IMK, Karlsruhe), and an inversion package supporting a variety of techniques and numerical alternatives. Further, a number of analysis tools are available that allow a detailed assessment of input data and results. Among the most important parameters that can be selected are climatological data, spectroscopic data, atmospheric modelling details, instrumental parameters, details for inversion control / regularization / discretization, use of numerical libraries, and additional diagnostics. Horizontal gradients can be handled, too. During the AMIL2DA activities it turned out that the basic concept of the processor did provide a stable and reliable platform and a robust test environment. However, the comparison of results among the partners led us to the correction of several weak points in our software, notably the stability of the line-of-sight corrections, an initial selection of spectral shift parameters, a flexible handling of partial covariance matrices and of continuum parameters. As a result of the intercomparison activities, we can profit from a reliable software package for the retrieval of various trace gas profiles. Another aspect deserving more and more importance is the in-house comparison with other atmospheric instruments where MIPAS results are cross-checked for validation purposes. Thus, the use of the processor software is no longer limited to pure MIPAS-based studies but paves the way for detailed inter-instrument comparisons and large-scale scientific studies.
A standardised error analysis has been produced for the retrieval schemes of the different groups involved in the AMIL2DA project. This consists of two main components:
1: Averaging kernel matrices describing the relationship between the real atmosphere and the retrieved profiles generated by each processor
2: Systematic error analysis of each group's retrievals, based on the spectral ranges used, masks, assumptions on contaminant gases and instrumental parameters.
The second component utilises the micro-window selection algorithm developed for MIPAS. This starts with a representation of all systematic error terms in measurement space, either as an error in the measurement (eg calibration uncertainty) or as an error in the forward model (e.g. uncertainty in a contaminant species). Assumptions are also made about the correlation, independence of these errors in the spectral and tangent altitude domains. The errors are then propagated through the retrieval process to convert these into systematic error components of the retrieved profiles.
A further, posterior error analysis technique has also been developed during the duration of this project: Residual and Error Correlation (REC) Analysis. In this, the residual spectral signatures, i.e. the difference between the measurements and the final iteration of the forward model, are statistically correlated with the error spectra mentioned above. This allows a rapid identification and quantification of actual systematic errors in the retrieved products, compared with the expected sizes. This has been used extensively in identifying problems with ESA's L1 processing which will affect all AMIL2DA groups using the L1B spectra.
Purpose of the work package is to characterise instrumental errors of MIPAS/ENVISAT, especially in case of unexpected results from the level 2 processing teams. The instrument characterisation and error propagation into the level 1 product is only marginally covered by ESA. The present work is linked to other activities co-ordinated by ESA.
The work covered detector non-linearity characterisation and correction, the effect of micro-vibrations, phase errors, ice contamination, radiometric calibration, and noise.
A number of unexpected instrumental effects occurred, which was addressed. Among these are the ESA in-flight detector non-linearity characterisation, which turned out to be insufficient, the radiometric impact of micro-vibrations, which lead to a modification of level 1 processing, and the contamination of detector cold optics with ice.
The results were provided to ESA, the AVCT, and the AMIL2DA team. The results definitely improved the quality of MIPAS data and thus the scientific output of the entire mission.
A new tool was developed for analysis of MIPAS data, which can be also applied to other satellite and non-satellite remote sensing FT instruments. New methods were developed within this tool for radiometric characterisation.
The characterisation of the instrument and improvement of level 1 product will continue in frame of the ESA Quality Working Group within phase E.
Finally, we expect to have a fully characterised Level 1 product with specified error bars, which allows proper error propagation into level 2.
Inversion of geophysical parameters (e.g., pressure-Temperature, ozone, water vapour, methane and NO2) under non-LTE conditions have been carried out by the first time. This has been made possible with the development of highly sophisticated inversion tools and non-LTE atmospheric modelling. The results show that these are important above about the stratopause.
In addition, the analysis of the day/night spectra measured by MIPAS has revealed the first detection of atmospheric non-LTE features in the methane infrared bands, in water vapour and in other weak bands of NO and CO. Also, the expected large NLTE effect in NO2 has been proven to me much weaker than anticipated. These can be partially mitigated in a high spectral resolution instrument, as MIPAS but have to be taken into account in the inversion of these species in the mesosphere from wide-band radiometers.
The RAL Level-2 data analysis methodology to retrieve atmospheric constituent profiles from MIPAS limb emission spectra has been refined, with the objective of scientific analysis beyond the ESA operational processor, particularly with respect to ozone and water vapour in the upper troposphere and lower stratosphere. The methodology has been implemented as computer code (so-called processor). The processor consists of a radiative transfer forward model (FM2D), and a retrieval code (RET2D), which inverts the measurements to obtain atmospheric profiles.The forward model FM2D was developed. This includes the adaptation of the RAL in-house forward model capable of handling horizontal variations in the line of sight for operation in the infrared, and inter-comparison with the MIPAS Reference Forward Model.
The 2-dimensional retrieval code (RET2D) was developed and implemented for use at infrared wavelengths by adaptation of the RAL mm-wave retrieval scheme. Auxiliary software tools were also developed, e.g. for plotting and inspecting diagnostics, handling MIPAS data and constructing appropriate atmospheric fields, from ECMWF and MIPAS climatology data.
After forward model cross-validation and retrieval blind tests some upgrading was performed. No major problems were found, but tests indicated that the set-up of the forward model was not optimal for some cases. In particular, the number of atmospheric levels in the atmospheric model and the number of monochromatic pencil beams used in the calculation were shown to be important. The set-ups were adjusted accordingly.
Early comparisons on blind test spectra indicated an artefact in minimisation routines, which was addressed and corrected. The driver files for the forward model were upgraded to allow explicit specification of view, scan number, spectral micro-window. This allows simple specification of forward model parameters for multiple runs including spectral ranges and satellite position. The retrieval model was enhanced to make use of the new forward model options.
For real data retrievals, external informationon temperature, pressure and pointing, such as ECMWF analyses combined with standard Mipas pressure/temperature products is employed and so no temperature/pressure/pointing retrieval tests were carried out for the RAL processor.
After upgrading of the processor and optimized setting of the processing parameters, good retrievals of ozone and water vapour were obtained.
The basic implementation of the RAL processor and its interface with MIPAS and auxiliary data operate correctly. However, a number of further refinements were shown to be desirable in terms of efficiency and the handling of data artifacts. To address this, the forward model was enhanced to allow generation and re-use of absorption coefficient files. This has produced a significant decrease in calculation time for real, iterative retrieval calculations.
The code can now also be run in a mode, which scales the retrieved products to allow for widely differing values of weighting functions. This is particularly relevant for retrieval of instrument parameters in conjunction with mixing ratio profiles where values may differ by many orders of magnitude.With the improvements indicated the RAL processor is able to produce scientifically valuable results. Remaining problems probably indicate artefacts in the current version of the MIPAS data rather than issues with processor. The final processor is considered appropriate for advanced MIPAS data analysis. The FM2D and RET2D algorithms are considered reliable state-of-the-art software. This work has brought RAL in a position to scientifically analyze MIPAS data using higher sophistication than the operational ESA processor.
The main achievement for Oxford under the AMIL2DA project has been the development and validation of the OPTIMO retrieval code. This differs from the ESA operational algorithm in two fundamental respects:
1. It is based on optimal estimation rather than a global fit. This gives greater flexibility to input data availability and avoids the need for generating occupation matrices in the pre-processing and also provides a convenient mechanism for regularisation
2. The internal forward model is based on the RFM (Reference Forward Model), an Oxford code originally developed as a reference model for testing the ESA operational forward model. This has the advantage of allowing different options varying from emulating the ESA forwardmodel to incorporating additional precision (e.g. line-by-line calculation) or physics (e.g. non-LTE emission)
In addition a pre-processor was written to anodise ESA L1B spectra, and extract micro-windows and generate instrument line shape parameters for the forward model.
The OPTIMO code and associated software and diagnostics have been gradually developed from a single profile retrieval to an orbital retrieval and an attempt is now underway to retrieve several consecutive days worth of data for an external project.
The main problems encountered have been in oscillating profiles and cloud effects. The oscillations can be reduced by increasing the regularisation in the a priori covariance but the source has been identified as an inconsistency in the L1 gain calibration and feedback has been provided to ESA. To reduce cloud contamination effects the current method is to exploit the optimal estimation approach by setting the a priori continuum extinction high for spectra where cloud effects are detected.
As a result of the experience gained with OPTIMO a further code, MORSE, is under development, based on the same principles but with greater flexibility, usability and handling of horizontal gradients.
Two retrieval algorithms for MIPAS Level 2 data analysis are being maintained/developed in the frame of this project. The first one (ORM_R) is the scientific version of the algorithm implemented in the ESA's Payload Data Segment and therefore is a reference for the results supplied by ESA's Level 2 near real time processor. The second algorithm (ORM_I) is an upgrade of the first one and includes the possibility of retrieving several instrument - related parameters such as:
a) altitude- and frequency- dependent offset,
b) frequency calibration,
c) intensity calibration,
d) instrument line-shape broadening.
These additional features of the algorithm will allow detecting potential deficiencies in the instrument characterization and/or in the Level 1b processor.Both retrieval algorithms are based on the global-fit analysis of the individual limb-scan sequences and use the Levenberg - Marquardt approach.The retrievals of the blind test exercise have been performed using the ORM_R code and the following conclusions have been drawn:
1) in general the ORM_R provides stable results also in presence of systematic errors in the forward model.
2) The error estimates supplied by the Oxford tool (MWMAKE) and the VCM of the retrieval are in general consistent with the discrepancies between retrieved and reference (true) profiles with a good confidence level.The only discrepancies not fully consistent with the error predictions were found in the cases in which a) the "shape error" (neglected in MWMAKE) is significant and b) the tangent pressure error exceeds its expected value (3%).
3) Neglecting pressure-shift and self-broadening does not impact the accuracy of the retrieved profiles in the considered test cases.
Therefore there is no clear indication suggesting that these two effects cannot be neglected in MIPAS retrievals.
4) The loss of accuracy due to neglecting NLTE and Line-Mixing can be adequately controlled using appropriate selection micro windows for the retrieval. Therefore, again, there is no clear indication suggesting that simulation of NLTE and Line-Mixing is compulsory for the retrieval of MIPAS key species. The ORM_I code was used to test the retrieval processor on some sets of real MIPAS data, for the characterisation of the above error sources. The results turned-out to be quite useful and allowed to introduce in MIPAS Level 2 processor a quadratic frequency-shift correction.
Furthermore the tests confirmed that the investigated instrument error sources are affecting the retrieved profiles within the limits forecasted by the error analysis carried-out by the Oxford University team.
Fitting the residual instrument offset could be avoided in MIPAS ground processor as the retrieved offset, so far, is always well below the measurement noise level. The ORM_I was upgraded to include the possibility of retrieving from MIPAS spectra further chemical species different from the key ones retrieved in the ESA's ground segment. This new functionality was exploited to retrieve CFC-11 and ClONO2 profiles from selected limb-scans relating to MIPAS orbit #2081. The results of these retrievals have been compared with the corresponding results obtained by the IMK and the OU processors. The observed discrepancies are consistent with the estimated profiles error budgets.
The IMK level-2 data analysis methodology has been developed for analysis of MIPAS level-2 data beyond the official ESA data product. It includes a radiative transfer forward model (KOPRA), a retrieval control program (RCP), several auxiliary software tools and databases as well as a default processor setup.
KOPRA is a line-by-line radiative transfer code which supports simulation of all major radiative effects of atmospheric infrared radiation, including line-coupling, non-local thermodynamic equilibrium, scattering.
RCP is an iterative constrained nonlinear least squares program, which supports both optimal estimation and Tikhonov-type smoothing constraints.
The auxiliary software tools include computer programs for selection of optimized micro-windows and dedicated diagnostic tools.
The default processor setup parameters have been tuned at hand of test retrieval calculations.
During AMIL2DA it turned out that the careful choice of processing parameters is essential for accurate retrievals of atmospheric state parameters. These parameters have been revised during the several steps of validation, which were performed under AMIL2DA.
As result of the radiative transfer forward model inter-comparison it was found that:
- The general concept of the model is sound and that reliable radiative transfer calculations are possible with this model;
- accuracy parameters for all kind of numerical integration play a key role for accurate spectral calculations.
- The inclusion of far wings of lines, which is very costly, is justified in order to appropriately describe the background radiance level.
From the blind test retrieval experiment and application of the retrieval processor to real MIPAS data including validation based on data from other Envisat and non-Envisat data it was learned that:
- IMK retrievals of atmospheric state parametersfrom MIPAS data are generally robust and reliable.
- Micro-window selection can and should be tuned to give highest accuracy in the altitude region of largest scientific interest.
- Sometimes joint retrievals of species (N2O and CH4) is not only more accurate but also more efficient than suboptimal sequential retrieval.
- The IMK MIPAS data processor is very sensitive to errors in spectral calibration. In order to avoid propagation of frequency calibration errors, the IMK processor now retrieves a frequency calibration correction on a routine basis before retrievals of any other instrument or geophysical parameters are performed.
- Retrievals are more robust if a smoothing constraint is not only applied in the altitude domain but also in the wave-number domain.
The ability to analyze species beyond the official routine data product of the European Space Agency, which covers vertical profiles of temperature, pressure and mixing ratios of ozone water vapour, nitric acid, nitrogen dioxide, laughing gas and methane, has been proven. The IMK data processor now is used for scientific exploitation of MIPAS results.
The retrievals of the scientific version of the algorithm implemented in the ESA's Payload Data Segment (ORM_R) from MIPAS-ENVISAT measurements have been compared with the retrievals provided by other five processors available to the partners of the AMIL2DA study (IMK, Oxford, DLR-a, D-PAC and RAL). The comparison has been done using MIPAS measurements relating to scans 3, 12, 20, 36 and 68 of orbit 2081 acquired on July 24th, 2002. The selected scans contain no or low cloud contamination and include the atmospheric scenarios of polar summer (scan 3), polar winter (36), mid-latitude day/night (12/68) and equatorial (20).In general, good agreement between the results of ORM_R and of the other processors is achieved. Especially for conditions where unexpected large systematic error sources are not present, e.g., in mid-latitude conditions, the discrepancies between the ORM_R results and the other processors involved in the AMIL2DA study are within the predicted errors.
A typical feature of the profiles retrieved by ORM_R is the smoothness. Quite often profiles retrieved by the ORM_R are as smooth as the profiles retrieved by processors that use a regularization scheme, while ORM_R did not use it in these retrievals. An explanation of this peculiarity lies in the convergence criterion used by the ORM_R, i.e. a threshold in the actual difference between the obtained chi2 and the chi2 predicted with a first order expansion of the forward model. This criterion still allows large changes of the retrieved parameters at the last iteration, and therefore lets the Levenberg-Marquardt parameter to act as a regularization constraint. Whenever the convergence is reached in very few (less than 3) iterations the resulting profiles are smoother than profiles retrieved with a weak regularization and several iterations.
The largest profiles oscillations are observed in the polar winter case (scan 36). In this case it is very likely that systematic errors, especially horizontal temperature gradients, are responsible for the large discrepancies observed among the results of the processors involved in the AMIL2DA study.
The results of the activities regarding the blind test retrievals and the Intercomparison of AMIL2DA results for MIPAS-ENVISAT data confirmed that the ORM_R provides, within its applicability limits (e.g. only for MIPAS key species), results of accuracy comparable with the accuracy of the other processors available to the partners of the AMIL2DA study.
The suite of programs `MIRART' `Modular InfraRed Atmospheric Radiative Transfer' and `PyReS' `PrototYpe REtrieval System' has been developed with special emphasis on efficient and reliable numerical algorithms and a modular approach appropriate for a large variety of applications in atmospheric remote sensing. MIRART is a flexible high-resolution line-by-line forward model and can be used for simulation and analysis of data of Fourier Transform spectrometers, Fabry-Perot interferometers, and heterodyne receivers with various viewing geometries (up-looking, down-looking, and limb-viewing; convolution with appropriate field-of-view functions optional). Weighting functions (Jacobians) are implemented by means of algorithmic derivatives. In addition to the forward model inter-comparison performed within the AMIL2DA project the code has been extensively tested in an inter-comparison of microwave radiative transfer models organized by University of Bremen.
PyReS comprise a variety of iterative solvers for the solution of ill-posed inverse problems arising in atmospheric vertical and limb sounding. The main motivation was to provide a test bed for analyzing the performance and capabilities of state-of-the-art numerical optimization methods and to have a powerful tool available for analysis of a variety of atmospheric spectra. The basic solution approach relies on Tikhonov regularization, i.e. least squares approach with quadratic constraints, with several algorithms implemented to determine the proper regularization ('degree of smoothing') during the iterative solution process.
In the course of the AMIL2DA project it has been essentially confirmed that MIRART and PyReS are applicable to analysis of spaceborne limb emission infrared spectroscopic observations as made by MIPAS on ENVISAT. Adjustments and upgrades to the software turned out to be minor, e.g., implementation of path refraction in the forward model or implementation of the MIPAS specific field-of-view. The recent implementation of an iterative regularized least squares solver with additional bound constraints (non-negativity of solution vector) significantly improved the quality of some of the constituent profiles (nb. NO2 and N2O) retrieved from MIPAS spectra.
Improved spectral line parameters have been provided for a number of species of atmospheric interest: Ozone, Nitric acid. These new line parameters have been included in the spectroscopic database dedicated to the MIPAS experiment. They allow one to retrieve from the atmospheric spectra recorded by MIPAS more accurate concentration profiles. It is worth noticing that the improved line parameters are also of interest for remote sensing techniques working at the same spectral range as MIPAS.
The principle of atmospheric sounding by measuring infrared emissions to derive geophysical parameters such as temperature, pressure, and trace species abundances is based on the assumption that the atmospheric compounds emit according to the Planck function at the local kinetic temperature (LTE). It is known, however, that many of the energy levels of the atmospheric constituents emitting in the infrared have `excitation' (usually called vibrational) temperatures which differ from the local kinetic temperature, i.e., they are in non-LTE. The non-LTE emissions affect significantly the retrieval of atmospheric species. Thus, the retrieval of accurate geophysical parameters from remote sounding measurements requires a precise knowledge of non-LTE populations.
In this project we have computed the most updated and accurate to date climatology of non-LTE vibrational populations of the main atmospheric emitters in the infrared. This have been done by means of non-LTE models which are based on the solution of the statistical and radiative transfer equations describing the physical/chemical processes responsible for populating the excited vibrational states. These processes have to be included in such models and information about them is normally taken from the analysis of the previous measurements of atmospheric emissions and on laboratory experiments. In the second part of the project we have applied this knowledge to the retrieval of geophysical parameters from MIPAS/Envisat, and have improved our knowledge of them from the analysis of MIPAS/Envisat data.
Searching for OpenAIRE data...
There was an error trying to search data from OpenAIRE
No results available