## Periodic Reporting for period 2 - ITHACA (An Information Theoretic Approach to Improving the Reliability of Weather and Climate Simulations)

Reporting period: 2019-04-01 to 2020-09-30

Climate change is one of the defining problems of our age. We know that global temperatures have increased, and will continue to increase, due to our carbon emissions. However, beyond this, there are profound uncertainties. Climate change is a problem for humanity because increasing carbon dioxide in the atmosphere interacts with the hydrological cycle, changing the amount of water vapour, cloud and precipitation in the atmosphere. The feedbacks with these hydrological processes can amplify, or indeed potentially damp, the basic climate change signal. This means that humanity is facing a spectrum of possible futures, from catastrophe to a rather muted “luke-warm” signal.

The only unambiguous way to reduce these uncertainties is through numerical models of climate which encode the laws of physics. The PI of ITHACA, together with one of the directors of the Max Planck Institute for Meteorology, wrote an important paper:

https://www.pnas.org/content/116/49/24390

in the Proceedings of the National Academy of Science outlining why current generation climate models are not good enough for the tools required of them: to inform on climate adaptation, climate mitigation, climate attribution, climate geoengineering, early warning and disaster risk reduction. As a specific example, the current generation of climate model simulate coherent long-lived self-organised circulation fields so poorly, we currently have no real basis for knowing whether to take the notion of a tipping point seriously or not.

The answer is to develop much higher resolution models than we currently have, where key physical processes are represented by the proper laws of physics rather than by crude empirically based parametrisations.

In principle this is possible given sufficient supercomputer power. However, even with exascale supercomputing, which will become available in a few years, there still won’t be enough computer power needed to get model grids down to 0(1km) in the horizontal. And yet this is what is needed if we are to simulate deep convection, orographic gravity wave drag and ocean mesoscale eddies with the laws of physics rather than with crude semi-empirical parametrisation.

Hence, in order to enable such high-resolution models to become feasible with the dawn of exascale computing, we need to look carefully at the way current models are formulated and see if computer resources are being used effectively and efficiently. For example, in a contemporary computer model, the actual sustained speed of the code as it executes on a supercomputer is only a few percent of the possible speed of the supercomputer. It is as if 95% is being wasted!

In ITHACA we are approaching one aspect of this question, motivated by work done in the PI’s previous ERC Advanced Grant PESM (Towards a Prototype Probabilistic Earth System Model), which explored the use of stochastic parametrisation (rather than the conventional deterministic parametrisation) in Earth-System models. The issue is numerical precision. Numerical models of climate are based on partial differential equations which are in turn based on so-called real numbers. Conventionally in climate models, these real numbers are represented by 64-bit floating point representations. Since there are many millions of real number variables in a climate model, there are correspondingly many billions of bits. Not only are computer resources used in performing calculations on these bits, even more vital energy resources are used transporting these bits from one part of the computer to another.

The key question posed in ITHACA is this. What is the information content in these billions of bits? How many bits contain useful information and how many bits are effectively just noise. Can we somehow formulate our models retaining only the bits that contain useful information?

The only unambiguous way to reduce these uncertainties is through numerical models of climate which encode the laws of physics. The PI of ITHACA, together with one of the directors of the Max Planck Institute for Meteorology, wrote an important paper:

https://www.pnas.org/content/116/49/24390

in the Proceedings of the National Academy of Science outlining why current generation climate models are not good enough for the tools required of them: to inform on climate adaptation, climate mitigation, climate attribution, climate geoengineering, early warning and disaster risk reduction. As a specific example, the current generation of climate model simulate coherent long-lived self-organised circulation fields so poorly, we currently have no real basis for knowing whether to take the notion of a tipping point seriously or not.

The answer is to develop much higher resolution models than we currently have, where key physical processes are represented by the proper laws of physics rather than by crude empirically based parametrisations.

In principle this is possible given sufficient supercomputer power. However, even with exascale supercomputing, which will become available in a few years, there still won’t be enough computer power needed to get model grids down to 0(1km) in the horizontal. And yet this is what is needed if we are to simulate deep convection, orographic gravity wave drag and ocean mesoscale eddies with the laws of physics rather than with crude semi-empirical parametrisation.

Hence, in order to enable such high-resolution models to become feasible with the dawn of exascale computing, we need to look carefully at the way current models are formulated and see if computer resources are being used effectively and efficiently. For example, in a contemporary computer model, the actual sustained speed of the code as it executes on a supercomputer is only a few percent of the possible speed of the supercomputer. It is as if 95% is being wasted!

In ITHACA we are approaching one aspect of this question, motivated by work done in the PI’s previous ERC Advanced Grant PESM (Towards a Prototype Probabilistic Earth System Model), which explored the use of stochastic parametrisation (rather than the conventional deterministic parametrisation) in Earth-System models. The issue is numerical precision. Numerical models of climate are based on partial differential equations which are in turn based on so-called real numbers. Conventionally in climate models, these real numbers are represented by 64-bit floating point representations. Since there are many millions of real number variables in a climate model, there are correspondingly many billions of bits. Not only are computer resources used in performing calculations on these bits, even more vital energy resources are used transporting these bits from one part of the computer to another.

The key question posed in ITHACA is this. What is the information content in these billions of bits? How many bits contain useful information and how many bits are effectively just noise. Can we somehow formulate our models retaining only the bits that contain useful information?

We are already well advanced in our collaboration with the European Centre for Medium Range Weather Forecasts (ECMWF), and as a result of this collaboration the numerical precision of real-number variables in their operational model will be reduced from 64 to 32 bits without loss of forecast accuracy and with a 40% reduction in run time.

We have now published a paper based on work performed entirely within ITHACA showing that a somewhat simplified version of the ECMWF operational forecast model can be successfully run with 16-bit precision for floating point numbers.

We have a paper in review which discusses the ability to utilise reduced precision in data assimilation systems. These systems enable initial conditions to be produced from sets of observations and are typically based on some minimisation algorithms. These can be sensitive to numerical precision. However, we have shown that providing certain algorithmic safeguards are performed – e.g. ensuring the strict orthogonality of quasi-independent perturbations – then our reduced-precision programme also works for data assimilation too.

We ran a major international workshop in 2019 on the use of AI for weather and climate modelling. This is particularly relevant for reduced precision studies, because many neural net and related pieces of code have been designed to run on 16-bit GPU processors. We have found an extremely novel application of these techniques for solving part of the time-stepping schemes for grid-point models. A paper on this is about to be submitted for publication. The proceedings of the AI workshop will be published as a special edition of Philosophical Transactions of the Royal Society.

By the end of September 2020, considerable progress had been made.

The most substantial result was made in collaboration with scientists at ECMWF, testing a hypothesis made by the PI in the ITHACA grant application. Here it was speculated that the computational savings made by reducing the numerical precision of an operational weather forecast model (from 64 to 32 bits) could be reinvested to increase the vertical resolution of the model, and that this would lead to a more accurate forecast model overall, but with no overall increase in computational cost. With co-authors from ECMWF, the PI is currently working on a paper for publication documenting these results. This is a major result of this project.

We have now published a paper based on work performed entirely within ITHACA showing that a somewhat simplified version of the ECMWF operational forecast model can be successfully run with 16-bit precision for floating point numbers.

We have a paper in review which discusses the ability to utilise reduced precision in data assimilation systems. These systems enable initial conditions to be produced from sets of observations and are typically based on some minimisation algorithms. These can be sensitive to numerical precision. However, we have shown that providing certain algorithmic safeguards are performed – e.g. ensuring the strict orthogonality of quasi-independent perturbations – then our reduced-precision programme also works for data assimilation too.

We ran a major international workshop in 2019 on the use of AI for weather and climate modelling. This is particularly relevant for reduced precision studies, because many neural net and related pieces of code have been designed to run on 16-bit GPU processors. We have found an extremely novel application of these techniques for solving part of the time-stepping schemes for grid-point models. A paper on this is about to be submitted for publication. The proceedings of the AI workshop will be published as a special edition of Philosophical Transactions of the Royal Society.

By the end of September 2020, considerable progress had been made.

The most substantial result was made in collaboration with scientists at ECMWF, testing a hypothesis made by the PI in the ITHACA grant application. Here it was speculated that the computational savings made by reducing the numerical precision of an operational weather forecast model (from 64 to 32 bits) could be reinvested to increase the vertical resolution of the model, and that this would lead to a more accurate forecast model overall, but with no overall increase in computational cost. With co-authors from ECMWF, the PI is currently working on a paper for publication documenting these results. This is a major result of this project.

We are very much leading the work worldwide in low-precision numerics for high-resolution models. Our aim for the end of this project is for this work to be incorporated into some of the leading weather and climate prediction models in the world. We expect that a next-generation weather and climate prediction model will judiciously combine single and half-precision numerics and with much of the parametrisation and Earth-System complexity represented by 16-bit AI. In this way, ITHACA will make a major contribution to the development of very high-resolution global Earth-System models, the need for which has been articulated above.

Although many of the results from ITHACA will be demonstrated using emulators of low-precision software, for the results to have full value it is important that they can be demonstrated in hardware. With the explosion of interest in AI, mixed-precision arithmetic is becoming available on leading supercomputers. The PI is investigating possible routes for utilising exascale hardware to be able to demonstrate ITHACA results in hardware by the end of the project.

Although many of the results from ITHACA will be demonstrated using emulators of low-precision software, for the results to have full value it is important that they can be demonstrated in hardware. With the explosion of interest in AI, mixed-precision arithmetic is becoming available on leading supercomputers. The PI is investigating possible routes for utilising exascale hardware to be able to demonstrate ITHACA results in hardware by the end of the project.