The work can be divided in nonlinear data assimilation, causal discovery, and applications of these methodologies to real-world problems. More has been done than can be reported here due to space limitations.
Quick progress was made on the data-assimilation side, where we explored four main variants: tempered particle filters, equivalent-weight particle filters, synchronization, data-assimilation on the Wasserstein space, and particle flow filters. We made strong progress on all these methods, and determined which method is most useful for which kind of problem. We also worked on exciting new methods to estimate errors in model equations, e.g. resulting in the only comprehensive method that can be applied in operational weather prediction, and we worked on method to increase the efficiency of solvers for large linear systems via so-called randomized preconditioners. More details can be found in the publications, and in papers that will appear, or be submitted soon.
However, the major breakthrough has been the development of a so-called Particle Flow Filter halfway the project. This is the first fully nonlinear data-assimilation (Bayesian Inference) method that can be applied to systems with arbitrary dimensions, and this achievement has solved a major problem that has plagued the data-assimilation community, including weather, ocean, and climate forecasters, for decades. We are still working on several papers, applying the method to a global atmospheric model, to high-resolution storm forecasting, and to ocean reanalysis.
After two years of hard work, sometimes encountering dead ends, we have managed to put causality on a completely new footing by including all possible interactions between the drivers. Interestingly, any nonlinear multivariate interaction had been ignored in causality estimation, while most physical, biological etc. systems are strongly governed by this category of interactions. Furthermore, we implemented causality in a Bayesian framework, allowing for proper uncertainty quantification for scientific reasoning.
In parallel, we worked on causal discovery in numerical models, exploring the model equations. It was hard to find a good postdoc to lead this work, but in the last year this was solved. We have developed a new method for information propagation in complex systems based on model equations, the first of its kind. This has turned out a much harder problem than anticipated, partly because the existing literature is less useful than previously thought, and either is misleading, incomplete, or cannot be generalized to the higher dimensional systems of interest. We essentially had to start from scratch, but have been successful and a paper is in preparation.
For the application of these methodologies to the ocean we collected data from satellites, ships, buoys, floats, and moorings, and combined them with ocean model NEMO, using the PDAF data-assimilation software system. Inflexibility of the ocean model made us change to MOM6, and we are working hard on finishing the 30-year reanalysis and the causal discovery. We performed nonlinear data assimilation on an evolving Hurricane (Patricia from 2015) and applied our causal discovery framework to understand the rapid intensification process in these huge storms. We found that ocean surface processes, vertical alignment processes related to strong convective rain systems, and favourable outflow conditions in the upper troposphere combine to make this process happen. We used our newly developed Particle Flow filter on cloud property retrievals of marine shallow cumulus and are applying the causal framework to better understand cloud microphysics.
Overall, the main results so far can be summarized as follows:
1) The development of the first fully nonlinear data-assimilation method for high-dimensional systems that converges to the true solution without any approximation.
2) The development of the first nonlinear causal discovery framework that includes nonlinear interactions between drivers
3) The development of the first causal discovery framework for numerical models that explores the model equations and can handle high-dimensional systems
4) Application of the causal discovery methodology of 2) to a realistic Hurricane simulation using the nonlinear data-assimilation of 1).
These main results have been published or about to be published. The nonlinear data-assimilation methodology has been implemented in community data-assimilation system DART, and in the near operational data-assimilation system JEDI. This will allow for maximum exposure of the methodology to both the academic and the operational communities.
