The project started in September 2015 with a 2 years out-going phase at the University of Minnesota.
The main tasks for the initial part of the out-going phase was to take part in the development and testing of the existing MHD WOMBAT code. During this phase the researcher tackled truly novel numerical problems resulting in a strong benefit for his trainig and future career and for the value of the project. Results on the WOMBAT code were presented Mendygral et al (2017). In this paper we were able to show perfect strong scaling to 68 threads on Intel KNL many-core processors and industry leading weak scaling to 260.000 cores on the BlueWaters machine.
During the out-going phase the researcher finalised also two publications connected with the Marie Curie project (Donnert et al 2016, 2017).
A strategic decision that was taken during the out-going phase was to implement the high order WENO5 grid method into WOMBAT. Implementing a 5th order scheme to solve MHD equations instead of a TVD 2nd order scheme increases significantly the novelty of our numerical scheme and its capabilities to model complex magnetised flows as those expected in the cosmological framework. First steps in this direction were already taken at the end of the out-going phase, however the WENO5 implementation of WOMBAT was in fact the main task during the incoming phase.
The incoming phase started on September 1st 2017 at the IRA-NAF.
During the incoming phase the researcher successfully implemented and fully tested a 5th weighted essentially non-oscillatory scheme (WENO5) for Eulerian magneto-hydrodynamics in WOMBAT. The scheme doubles effective resolution with respect to state-of-the-art 2nd order schemes (PPM,TVD) and resolves instabilities even better and is much less susceptible to advection errors, which are a common problem in cosmological simulation with Eulerian codes (see FIGURE). First results on testing and comparison with other algorithms have been discussed in Donnert et al (2018), a longer paper is in advanced stage of preparation (Donnert et al to be submitted). Recently WOMBAT was scaled by the OpenMPI collaboration to 512.000 cores on the Trinity supercomputer at Los Alamos National Laboratory, USA. To our understanding our code now holds the record for the largest MHD simulation ever run.
Having in hands a unprecedented highly scalable and high-precision MHD code, the last step of the work is to implement WENO5-WOMBAT in a full cosmological framework. The cosmological implementation of the WOMBAT code started at the end of the out-going phase and continued during the incoming phase. We successfully developed a 3D gravity implementation for Dark Matter in the code, final tests are ongoing and the results will be published soon (Nolting, Donnert et al, in prep).
In these years the results of the project have been presented in several international conferences and workshops. The researcher established a website of the project at http://wombatcode.org , where he presents an introduction for the public, a blog for researchers and performance engineers, the public version of the code and all publications. The researcher organized a 1 week group workshop at the IRA-INAF in July 2018, where the whole collaboration met to discuss the future of the project.
In conclusion during the 3 years of the project we have developed a innovative platform that is capable to overcome the limitations of the available MHD cosmological codes and algorithms thus allowing us to model the magnetic fields in galaxy clusters with unprecedented detail and fidelity. Large (PB-scale) simulations with our code will shortly provide the first steps to the eagerly awaited predictions for the next generation of cluster observations.
