Periodic Reporting for period 1 - SENTINEL (Design and Implementation of an Advanced Nonlinear and Non-Gaussian Data Assimilation Algorithm for Bounded Variables in Numerical Weather Prediciton Models.)
Reporting period: 2024-06-01 to 2026-05-31
Summary of the context and overall objectives of the project
Current DA algorithms present serious flaws that hinder the accurate state estimation of the atmosphere over the sea. Different studies revealed several issues that limit extreme weather forecast skills, which arise from the mathematical formulation underlying such DA techniques. Two of the main limitations of current DA techniques
are (i) the underlying assumption that model and observational uncertainties are normally distributed and (ii) the use of simplified linearized versions of the DA equations. In general, these assumptions are well preserved for model variables and observations related to temperatures, winds, or pressure fields, among others. However, the estimation of cloud properties and precipitation rates from remote-sensing instruments (i.e. Doppler radars, lidars, or satellites), do not follow these simple assumptions. For example, aerosols, water vapour, clouds, precipitation, sea-ice or even plankton concentrations, which are semi-positive definite variables (i.e. can take positive or zero values), are characterized to have uncertainty probability distributions that are skewed and better approximated by Gamma and Inverse-Gamma probability density functions (PDFs) than Gaussian PDFs. The use of current DA algorithms on these situations provokes atmospheric state representation flaws and therefore, the forecasts initiated from these estimations will also be deficient. In such cases, atmospheric state-cloud process relationships are clearly nonlinear, resulting in most observations from instruments such as satellites are not properly exploited and therefore, wasted. Under these circumstances, current DA techniques are sub-optimal, limiting or even deteriorating their resulting analysis. Currently, this problem is one of the most important challenges that the international community is facing, and progress in this direction will significantly contribute to improve numerical weather and climate predictions.
This project proposes to go beyond the state of the art in Data Assimilation and Numerical Weather Prediction of coastal hazardous weather events in two phases. The first phase aims at developing and implementing a novel DA method for non-Gaussian and bounded variables, such as clouds and precipitation, based on the original GIGG-EnKF, that will perform better than current DA algorithms, even those that do not assume Gaussianity, such as Particle Filters. This new DA method will be referred as The Nonlinear Bounded Variable Ensemble Transform Filter (NBVT) and it will better account for nonlinearities than standard EnKF methods. Another appealing feature of NBVT is that will avoid having non-physical values in the analysis fields. This new theory would allow us to improve the initial conditions estimation over regions with lack of in-situ observations, such as over maritime areas. During this phase, we will introduce the NBVT method to the DA community implementing the code in the DART software from NCAR. DART is well-known in the DA community and a great number of researchers worldwide use DART to carry out their research in DA. For this reason, implementing NBVT into DART will allow us to share our code with most researchers in DA. The second phase is a practical implementation of the NBVT to forecast real hazardous weather events. I will assess the potential of the NBVT to improve initial conditions over maritime regions and then quantitatively assess its impact on the predictability of coastal hazardous weather events. In this project, I will focus my attention on one of the most destructive weather events in the Mediteranean region, which are known as Mediterranean Hurricanes (medicanes). They are typically initiated over the sea, resulting in initial conditions that are poorly estimated. Consequently, forecasting the intensity and trajectory of medicanes still remains very challenging and thus, they are excellent candidates to test the NBVT and compared with standard DA methods. This project will allow us to better diagnose the physical mechanisms involved in these medicanes and improve our understanding of how they initiate and develop. The novelty of this project comes from the theoretical development of an innovative and advanced DA algorithm that will be used to accurately estimate initial conditions in poorly observed regions and it will be implemented for the first time on high-impact real weather events. It is also expected that the novel DA technique developed in this project will significantly contribute to the international community, enhancing current DA algorithms operationally used worldwide at National Weather Centers (NWCs) to ultimately improve weather forecasts at all the scales of interest and to improve climate projections.
The overarching objective of this proposal is to advance in the theoretical development and implementation of a novel DA technique which will improve the state estimation of the atmosphere and the ocean through the assimilation of cloud-based, precipitation and concentration observations which are not accurately handled by current DA schemes. This novel technique will significantly improve global, regional, and climate forecasts, which have shown to be sensitive to the initial conditions of the atmosphere and ocean20,21,22. The final goal of this project is to quantitatively assess the impact of this proposed technique in real cases through the assimilation of non-Gaussian
and bounded observations. This will be achieved through the following three specific objectives:
Objective 1: Further develop the theoretical basis underlying the GIGG-EnKF DA algorithm. Obtain the new equations of the NBVT DA algorithm that deals with nonlinearities and avoid non-physical values in the
analysis estimates.
Objective 2: Implement the NBVT obtained from the new theory in DART. Testing of the new code over simplified and controlled numerical experiments known as Observation System Simulation Experiments (OSSEs).
Objective 3: Perform the first implementation of the novel NBVT on medicanes. Quantifying the impact of assimilating bounded observations from satellite instruments to improve predictability of different medicanes.
Contribute to better understanding of the physical mechanisms associated with the initiation and development of such medicanes.
are (i) the underlying assumption that model and observational uncertainties are normally distributed and (ii) the use of simplified linearized versions of the DA equations. In general, these assumptions are well preserved for model variables and observations related to temperatures, winds, or pressure fields, among others. However, the estimation of cloud properties and precipitation rates from remote-sensing instruments (i.e. Doppler radars, lidars, or satellites), do not follow these simple assumptions. For example, aerosols, water vapour, clouds, precipitation, sea-ice or even plankton concentrations, which are semi-positive definite variables (i.e. can take positive or zero values), are characterized to have uncertainty probability distributions that are skewed and better approximated by Gamma and Inverse-Gamma probability density functions (PDFs) than Gaussian PDFs. The use of current DA algorithms on these situations provokes atmospheric state representation flaws and therefore, the forecasts initiated from these estimations will also be deficient. In such cases, atmospheric state-cloud process relationships are clearly nonlinear, resulting in most observations from instruments such as satellites are not properly exploited and therefore, wasted. Under these circumstances, current DA techniques are sub-optimal, limiting or even deteriorating their resulting analysis. Currently, this problem is one of the most important challenges that the international community is facing, and progress in this direction will significantly contribute to improve numerical weather and climate predictions.
This project proposes to go beyond the state of the art in Data Assimilation and Numerical Weather Prediction of coastal hazardous weather events in two phases. The first phase aims at developing and implementing a novel DA method for non-Gaussian and bounded variables, such as clouds and precipitation, based on the original GIGG-EnKF, that will perform better than current DA algorithms, even those that do not assume Gaussianity, such as Particle Filters. This new DA method will be referred as The Nonlinear Bounded Variable Ensemble Transform Filter (NBVT) and it will better account for nonlinearities than standard EnKF methods. Another appealing feature of NBVT is that will avoid having non-physical values in the analysis fields. This new theory would allow us to improve the initial conditions estimation over regions with lack of in-situ observations, such as over maritime areas. During this phase, we will introduce the NBVT method to the DA community implementing the code in the DART software from NCAR. DART is well-known in the DA community and a great number of researchers worldwide use DART to carry out their research in DA. For this reason, implementing NBVT into DART will allow us to share our code with most researchers in DA. The second phase is a practical implementation of the NBVT to forecast real hazardous weather events. I will assess the potential of the NBVT to improve initial conditions over maritime regions and then quantitatively assess its impact on the predictability of coastal hazardous weather events. In this project, I will focus my attention on one of the most destructive weather events in the Mediteranean region, which are known as Mediterranean Hurricanes (medicanes). They are typically initiated over the sea, resulting in initial conditions that are poorly estimated. Consequently, forecasting the intensity and trajectory of medicanes still remains very challenging and thus, they are excellent candidates to test the NBVT and compared with standard DA methods. This project will allow us to better diagnose the physical mechanisms involved in these medicanes and improve our understanding of how they initiate and develop. The novelty of this project comes from the theoretical development of an innovative and advanced DA algorithm that will be used to accurately estimate initial conditions in poorly observed regions and it will be implemented for the first time on high-impact real weather events. It is also expected that the novel DA technique developed in this project will significantly contribute to the international community, enhancing current DA algorithms operationally used worldwide at National Weather Centers (NWCs) to ultimately improve weather forecasts at all the scales of interest and to improve climate projections.
The overarching objective of this proposal is to advance in the theoretical development and implementation of a novel DA technique which will improve the state estimation of the atmosphere and the ocean through the assimilation of cloud-based, precipitation and concentration observations which are not accurately handled by current DA schemes. This novel technique will significantly improve global, regional, and climate forecasts, which have shown to be sensitive to the initial conditions of the atmosphere and ocean20,21,22. The final goal of this project is to quantitatively assess the impact of this proposed technique in real cases through the assimilation of non-Gaussian
and bounded observations. This will be achieved through the following three specific objectives:
Objective 1: Further develop the theoretical basis underlying the GIGG-EnKF DA algorithm. Obtain the new equations of the NBVT DA algorithm that deals with nonlinearities and avoid non-physical values in the
analysis estimates.
Objective 2: Implement the NBVT obtained from the new theory in DART. Testing of the new code over simplified and controlled numerical experiments known as Observation System Simulation Experiments (OSSEs).
Objective 3: Perform the first implementation of the novel NBVT on medicanes. Quantifying the impact of assimilating bounded observations from satellite instruments to improve predictability of different medicanes.
Contribute to better understanding of the physical mechanisms associated with the initiation and development of such medicanes.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
During the 6 months this project run, I was able to:
1. Derive a new set of DA equations analogous to the one used for the widely used Perturbed Observations Ensemble Transform Kalman Filter (ETKF) for the Gaussian case that was obtained from applying Bayes’ Theorem using Gaussian PDFs for both forecast and observation uncertainties. However, instead of assuming Gaussian PDFs for forecast and observation uncertainties, we assumed that the ensemble forecast was sampled from a Gamma PDF and that the observations were obtained from an Inverse-Gamma PDF. High-impact weather and climate variables such as those pertaining to aerosols, rainfall, water-vapour mixing ratio, cloud-water/ice concentrations and sea ice are better approximated by Gamma and Inverse-Gamma probability density functions than Gaussian PDFs. This approach retains the accuracy of the standard ETKF in the Gaussian case, while lending it a high degree of accuracy in the non-Gaussian and highly skewed cases. If our new NBVT filter assimilates all observations with the Gaussian assumption, the posterior PDF is identical to that of the ETKF (perturbed version). However, when Gamma or Inverse-Gamma PDF pairs are assumed to best describe forecast and observation uncertainty, the posterior mean, standard deviation and skewness are all significantly different to that which would be obtained by the perturbed observations ETKF.
What makes the new NBVT DA scheme more appealing, is the fact that the equations resemble the ones from the perturbed ETKF, which is used in operational centers. In this sense, it facilitates enormously its implementation to National Weather Centers.
2. Avoid non-physical values in the analysis: When trying to assimilate bounded variables, one common problem is getting non-physical values of the variables in the analysis. This is typically produced during the linear-regression step into the DA process, where the analysis increments obtained at the grid model point where we are assimilating the observations is used through the ensemble covariances to extend to the rest of model variables in the numerical domain. This linear-regression step does not guarantee that regressed analysis values far from the analysis grid model point will fall in the bounded limits. To mitigate this issue, we have replaced the linear-regression step by a new blended regression step, which basically combines the linear-regression method with a logarithmic-based regression step. More specifically, the linear-regression method is used for the values that are far from the bounds, meanwhile the logarithmic-regression step is used for values that are close to the boundaries. This new approximation helps us to avoid obtaining out of bounds values of the variables updated by the DA scheme for non-Gaussian and skewed variables.
3. Development of a Local form of the equations: In real-life applications, National Weather Centers only can use a reduce ensemble size to perform their high-resolution numerical weather simulations because of limiting computational resources. Unfortunately, the use of small to moderate ensemble size has the disadvantage of producing spurious forecast error covariances, deteriorating the analysis results. In order to reduce these spurious forecast error covariances, we have developed a Local form of the NBVT equations that allows to mitigate this problem based on the separation distance between the analysis grid model point and the rest of model grid points within the numerical domain. Typically, for Gaussian DA schemes, the way of mitigating this problem is increasing the observation error variance as a function of the separation distance between the analysis grid model point and the observation location. More the observation is far from the analysis grid model point, the more the observation error variance is increased. This method works very well for Gaussian DA schemes. However, we notice that this methodology could not be applied in our new NBVT DA scheme because is written in terms of relative observation error variance instead of the typical observation error variance, used in Gaussian DA. For this reason, we developed a new and equivalent localization technique called the “Ensemble Squeeze Localization”, which is based on reducing the variance of the ensemble in the same proportion the observation error was increased in the Local form of the ETKF.
Another feature that is crucial when developing a new DA scheme to be used in real-life situations is its scalability when the model and the observations become more and more complex using high-resolution data (i.e. 106-8 model variables, 108 observations). In those situations, National Weather Centers need to run DA algorithms that could process all the observations very fast in order to get analysis fields very soon to be able to predict what is going to happen and warn population, if necessary. If the numerical models and DA schemes do not allow a quick computation, they are not useful in practice.
1. Derive a new set of DA equations analogous to the one used for the widely used Perturbed Observations Ensemble Transform Kalman Filter (ETKF) for the Gaussian case that was obtained from applying Bayes’ Theorem using Gaussian PDFs for both forecast and observation uncertainties. However, instead of assuming Gaussian PDFs for forecast and observation uncertainties, we assumed that the ensemble forecast was sampled from a Gamma PDF and that the observations were obtained from an Inverse-Gamma PDF. High-impact weather and climate variables such as those pertaining to aerosols, rainfall, water-vapour mixing ratio, cloud-water/ice concentrations and sea ice are better approximated by Gamma and Inverse-Gamma probability density functions than Gaussian PDFs. This approach retains the accuracy of the standard ETKF in the Gaussian case, while lending it a high degree of accuracy in the non-Gaussian and highly skewed cases. If our new NBVT filter assimilates all observations with the Gaussian assumption, the posterior PDF is identical to that of the ETKF (perturbed version). However, when Gamma or Inverse-Gamma PDF pairs are assumed to best describe forecast and observation uncertainty, the posterior mean, standard deviation and skewness are all significantly different to that which would be obtained by the perturbed observations ETKF.
What makes the new NBVT DA scheme more appealing, is the fact that the equations resemble the ones from the perturbed ETKF, which is used in operational centers. In this sense, it facilitates enormously its implementation to National Weather Centers.
2. Avoid non-physical values in the analysis: When trying to assimilate bounded variables, one common problem is getting non-physical values of the variables in the analysis. This is typically produced during the linear-regression step into the DA process, where the analysis increments obtained at the grid model point where we are assimilating the observations is used through the ensemble covariances to extend to the rest of model variables in the numerical domain. This linear-regression step does not guarantee that regressed analysis values far from the analysis grid model point will fall in the bounded limits. To mitigate this issue, we have replaced the linear-regression step by a new blended regression step, which basically combines the linear-regression method with a logarithmic-based regression step. More specifically, the linear-regression method is used for the values that are far from the bounds, meanwhile the logarithmic-regression step is used for values that are close to the boundaries. This new approximation helps us to avoid obtaining out of bounds values of the variables updated by the DA scheme for non-Gaussian and skewed variables.
3. Development of a Local form of the equations: In real-life applications, National Weather Centers only can use a reduce ensemble size to perform their high-resolution numerical weather simulations because of limiting computational resources. Unfortunately, the use of small to moderate ensemble size has the disadvantage of producing spurious forecast error covariances, deteriorating the analysis results. In order to reduce these spurious forecast error covariances, we have developed a Local form of the NBVT equations that allows to mitigate this problem based on the separation distance between the analysis grid model point and the rest of model grid points within the numerical domain. Typically, for Gaussian DA schemes, the way of mitigating this problem is increasing the observation error variance as a function of the separation distance between the analysis grid model point and the observation location. More the observation is far from the analysis grid model point, the more the observation error variance is increased. This method works very well for Gaussian DA schemes. However, we notice that this methodology could not be applied in our new NBVT DA scheme because is written in terms of relative observation error variance instead of the typical observation error variance, used in Gaussian DA. For this reason, we developed a new and equivalent localization technique called the “Ensemble Squeeze Localization”, which is based on reducing the variance of the ensemble in the same proportion the observation error was increased in the Local form of the ETKF.
Another feature that is crucial when developing a new DA scheme to be used in real-life situations is its scalability when the model and the observations become more and more complex using high-resolution data (i.e. 106-8 model variables, 108 observations). In those situations, National Weather Centers need to run DA algorithms that could process all the observations very fast in order to get analysis fields very soon to be able to predict what is going to happen and warn population, if necessary. If the numerical models and DA schemes do not allow a quick computation, they are not useful in practice.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
The main result of this WP is the development of the new NBVT DA scheme to assimilate bounded variables. In contrast to current Gaussian DA schemes, our NBVT can effectively assimilate non-gaussian and bounded variables, such as the ones linked to high-impact weather. Moreover, we have developed a local form of the NBVT algorithm that enable to be implemented at National Operational Weather Centers with minor changes in the code respect existing ones. Additionally, the NBVT can be parallelized in High Performance Computing facilities and used operationally in real-world situations.