Final Report Summary - EXAGAL (Hydrodynamical Simulations of Galaxy Formation at the Peta- and Exascale)
Galaxies are comprised of up to several hundred billion stars and display a variety of shapes and sizes. Their formation involves a complicated blend of astrophysics, including gravitational, hydrodynamical and radiative processes, as well as dynamics in the enigmatic dark sector of the Universe, which is composed of dark matter and dark energy. Because the governing equations are too complicated to be solved analytically, numerical simulations have become a primary tool in theoretical studies of cosmic structure formation. This ERC project has aimed to leverage large parallel supercomputers to model representative pieces of the Universe with novel numerical techniques, aiming at predicting how galaxies formed and how they cluster in space. Such calculations connect the comparatively simple initial state left behind by the Big Bang some 13.5 billion years ago with the complex, evolved state of the Universe today.
A first highlight of the project's work has been the "Illustris Simulation", reported in Nature in May 2014, the at the time largest hydrodynamical simulation of galaxy formation. For it we have employed our novel moving-mesh methodology for hydrodynamical flows, as implemented in our AREPO code. This method for cosmological hydrodynamics uses a finite-volume approach on a three-dimensional, fully dynamic Voronoi tessellation. The moving mesh is particularly well suited to the high dynamic range in space and time posed by the galaxy formation problem, and delivers better accuracy then computing techniques. It has been a central aim of the ERC project to demonstrate these advantages in demanding scientific applications, and to further refine the technical infrastructure of the code as well as its physics models. Besides gravity in the dark matter and cosmic baryons, the processes modelled in our typical simulations also include radiative cooling, star formation and stellar evolution, energy feedback by supernova explosions and growing black holes, as well as metal enrichment by stellar and galactic winds. In the course of the project, we have added magneto-hydrodynamics, anisotropic transport of cosmic rays and heat energy, as well as novel treatments for feedback by supermassive black holes.
The Illustris calculation is one of our first flagship results. It was carried out in a periodic box comoving with the cosmic expansion, covering a region of about 350 million lightyears across. It has made rich predictions for a diverse set of physical properties, including the X-ray emission of the intracluster gas, metal lines in the intergalactic medium, or the Sunyaev-Zeldovich effect. Most importantly, Illustris for the first time yielded a realistic mix of elliptical and spiral galaxies, thereby overcoming a decade old impasse in the field.
Moreover, we found that the simulation can explain the enrichment of heavy elements in neutral hydrogen gas found in so-called damped Lyman-alpha absorbers. Also, carefully prepared mock observations of the simulated universe show that the calculated galaxies are distributed in space as observed with telescopes, so that deep mock images of galaxies created from Illustris show a stunning similarity to real observations such as the Hubble Ultradeep Field. Quantitatively, this is reflected in a good match of the simulation's predictions for the abundance of galaxies at different epochs as a function of their stellar mass. In order to achieve this success, energetic feedback processes from supernova and supermassive black holes are invoked in the physics model calculated by the simulation. These two powerful actors suppress star formation in small and large dark matter halos, respectively, producing a characteristic mass scale at which galaxy formation is most efficient. This reconciles observations of the galaxy abundance with theoretical expectations for the abundance of dark matter halos.
Complementing the Illustris simulation, we have also simulated individual galaxies of the size of the Milky Way at very high resolution in the AURIGA project, including for the first time magneto-hydrodynamics as well as black hole accretion in calculations of this type. The resulting disk galaxies are consistent with many basic properties of the Milky Way, and predict the formation of a magnetic field of several microgauss strength that is largely azimuthally aligned in the disk. The simulations show that a small-scale dynamo operating efficiently already at high redshift is amplifying magnetic fields to the observed strength, until the dynamo saturates close to equipartition one the magnetic pressure becomes similar to the thermal pressure.
As final capstone of the ERC project, we carried out The Next Generation Illustris simulations, IllustrisTNG. These calculations build up on the success of the Illustris-project, but improve it in critical ways. Most importantly, IllustrisTNG uses a substantially improved physical and numerical model, and pushes the calculations to much larger sizes. In particular, the new simulations account for a novel kinetic wind model to represent black hole feedback physics, they incorporate magnetic field amplification from high redshift to the present, and they employ new techniques for tracking the production of heavy elements in supernova explosions of different type.
The ambitious simulation program carried out in IllustrisTNG consists in total of three complementary cosmological simulations: TNG300 (a large volume simulation with a periodic boxsize of 300 Mpc), TNG100 (intermediate volume, boxsize 110 Mpc), and a third, still more difficult calculation, TNG50 (small volume, boxsize 50 Mpc). To assess resolution convergence, the TNG300 simulation (as well as the other flagship runs) has been supplemented by other lower resolution simulations, spaced by factors of 8 in mass resolution. And to isolate the effects of the galaxy formation model onto the underlying dark matter distribution, each full baryonic physics run has also been repeated with the same initial condition phases, but including only dark matter and gravity.
To give an idea about the scope of the simulations, the TNG300 simulation was carried out using 24000 processor cores on the Hazel Hen supercomputer in Stuttgart and contained 31.25 billion resolution elements, half in the form of dark matter particles and half as Voronoi gas cells. In addition, more than 15 billion Monte Carlo tracer particles for tracking gas flows were included. The simulation required about 100 TB of RAM to execute, corresponding to nearly all of the RAM of the assigned compute nodes. The cumulative data volume of the restart files written and read again in order to carry out the calculation over the coarse of several months was in excess of 10 Petabytes, corresponding to a time-averaged I/O rate of about 1 GB/sec. The data volume kept permanently for scientific analysis is of the oder of 250 TeraByte for this simulation alone.
Already some of the first results of the IllustrisTNG project demonstrated a spectacular breadth of new theoretical predictions. For example, for the first time these hydrodynamical simulations of galaxy formation have reached sufficient volume to allow precision predictions for clustering on cosmologically relevant scales. In fact, the IllustrisTNG simulations have determined the non-linear correlation functions and power spectra of baryons, dark matter, galaxies and haloes over an exceptionally large range of scales. This showed that baryonic effects increase the clustering of dark matter on small scales, and damp the total matter power spectrum on scales up to 3 Mpc. The two-point correlation function of the simulated galaxies has been found to agree well with observational data from the Sloan Digital Sky Survey, both as a function of stellar mass and when split according to galaxy color.
The TNG simulations also confirmed that structure formation amplifies tiny initial magnetic seed fields to the values observed in low-redshift galaxies. The magnetic field topology is closely connected to galaxy morphology such that irregular fields are hosted by early-type galaxies, while large-scale, ordered fields are present in disc galaxies. This for example allows predictions of the diffuse radio emission of galaxy clusters, and observational forecasts for forthcoming radio telescopes such as the Square Kilometer Array (SKA).
A first highlight of the project's work has been the "Illustris Simulation", reported in Nature in May 2014, the at the time largest hydrodynamical simulation of galaxy formation. For it we have employed our novel moving-mesh methodology for hydrodynamical flows, as implemented in our AREPO code. This method for cosmological hydrodynamics uses a finite-volume approach on a three-dimensional, fully dynamic Voronoi tessellation. The moving mesh is particularly well suited to the high dynamic range in space and time posed by the galaxy formation problem, and delivers better accuracy then computing techniques. It has been a central aim of the ERC project to demonstrate these advantages in demanding scientific applications, and to further refine the technical infrastructure of the code as well as its physics models. Besides gravity in the dark matter and cosmic baryons, the processes modelled in our typical simulations also include radiative cooling, star formation and stellar evolution, energy feedback by supernova explosions and growing black holes, as well as metal enrichment by stellar and galactic winds. In the course of the project, we have added magneto-hydrodynamics, anisotropic transport of cosmic rays and heat energy, as well as novel treatments for feedback by supermassive black holes.
The Illustris calculation is one of our first flagship results. It was carried out in a periodic box comoving with the cosmic expansion, covering a region of about 350 million lightyears across. It has made rich predictions for a diverse set of physical properties, including the X-ray emission of the intracluster gas, metal lines in the intergalactic medium, or the Sunyaev-Zeldovich effect. Most importantly, Illustris for the first time yielded a realistic mix of elliptical and spiral galaxies, thereby overcoming a decade old impasse in the field.
Moreover, we found that the simulation can explain the enrichment of heavy elements in neutral hydrogen gas found in so-called damped Lyman-alpha absorbers. Also, carefully prepared mock observations of the simulated universe show that the calculated galaxies are distributed in space as observed with telescopes, so that deep mock images of galaxies created from Illustris show a stunning similarity to real observations such as the Hubble Ultradeep Field. Quantitatively, this is reflected in a good match of the simulation's predictions for the abundance of galaxies at different epochs as a function of their stellar mass. In order to achieve this success, energetic feedback processes from supernova and supermassive black holes are invoked in the physics model calculated by the simulation. These two powerful actors suppress star formation in small and large dark matter halos, respectively, producing a characteristic mass scale at which galaxy formation is most efficient. This reconciles observations of the galaxy abundance with theoretical expectations for the abundance of dark matter halos.
Complementing the Illustris simulation, we have also simulated individual galaxies of the size of the Milky Way at very high resolution in the AURIGA project, including for the first time magneto-hydrodynamics as well as black hole accretion in calculations of this type. The resulting disk galaxies are consistent with many basic properties of the Milky Way, and predict the formation of a magnetic field of several microgauss strength that is largely azimuthally aligned in the disk. The simulations show that a small-scale dynamo operating efficiently already at high redshift is amplifying magnetic fields to the observed strength, until the dynamo saturates close to equipartition one the magnetic pressure becomes similar to the thermal pressure.
As final capstone of the ERC project, we carried out The Next Generation Illustris simulations, IllustrisTNG. These calculations build up on the success of the Illustris-project, but improve it in critical ways. Most importantly, IllustrisTNG uses a substantially improved physical and numerical model, and pushes the calculations to much larger sizes. In particular, the new simulations account for a novel kinetic wind model to represent black hole feedback physics, they incorporate magnetic field amplification from high redshift to the present, and they employ new techniques for tracking the production of heavy elements in supernova explosions of different type.
The ambitious simulation program carried out in IllustrisTNG consists in total of three complementary cosmological simulations: TNG300 (a large volume simulation with a periodic boxsize of 300 Mpc), TNG100 (intermediate volume, boxsize 110 Mpc), and a third, still more difficult calculation, TNG50 (small volume, boxsize 50 Mpc). To assess resolution convergence, the TNG300 simulation (as well as the other flagship runs) has been supplemented by other lower resolution simulations, spaced by factors of 8 in mass resolution. And to isolate the effects of the galaxy formation model onto the underlying dark matter distribution, each full baryonic physics run has also been repeated with the same initial condition phases, but including only dark matter and gravity.
To give an idea about the scope of the simulations, the TNG300 simulation was carried out using 24000 processor cores on the Hazel Hen supercomputer in Stuttgart and contained 31.25 billion resolution elements, half in the form of dark matter particles and half as Voronoi gas cells. In addition, more than 15 billion Monte Carlo tracer particles for tracking gas flows were included. The simulation required about 100 TB of RAM to execute, corresponding to nearly all of the RAM of the assigned compute nodes. The cumulative data volume of the restart files written and read again in order to carry out the calculation over the coarse of several months was in excess of 10 Petabytes, corresponding to a time-averaged I/O rate of about 1 GB/sec. The data volume kept permanently for scientific analysis is of the oder of 250 TeraByte for this simulation alone.
Already some of the first results of the IllustrisTNG project demonstrated a spectacular breadth of new theoretical predictions. For example, for the first time these hydrodynamical simulations of galaxy formation have reached sufficient volume to allow precision predictions for clustering on cosmologically relevant scales. In fact, the IllustrisTNG simulations have determined the non-linear correlation functions and power spectra of baryons, dark matter, galaxies and haloes over an exceptionally large range of scales. This showed that baryonic effects increase the clustering of dark matter on small scales, and damp the total matter power spectrum on scales up to 3 Mpc. The two-point correlation function of the simulated galaxies has been found to agree well with observational data from the Sloan Digital Sky Survey, both as a function of stellar mass and when split according to galaxy color.
The TNG simulations also confirmed that structure formation amplifies tiny initial magnetic seed fields to the values observed in low-redshift galaxies. The magnetic field topology is closely connected to galaxy morphology such that irregular fields are hosted by early-type galaxies, while large-scale, ordered fields are present in disc galaxies. This for example allows predictions of the diffuse radio emission of galaxy clusters, and observational forecasts for forthcoming radio telescopes such as the Square Kilometer Array (SKA).