MAchinE Learning for Scalable meTeoROlogy and cliMate

Información del proyecto

MAELSTROM

Identificador del acuerdo de subvención: 955513

DOI

10.3030/955513

Proyecto cerrado

Fecha de la firma de la CE 2 Diciembre 2020

Fecha de inicio 1 Abril 2021

Fecha de finalización 31 Marzo 2024

Financiado con arreglo a

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Coste total

€ 4 312 412,50

Aportación de la UE

€ 2 156 206,25

Coordinado por

EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
United Kingdom

CORDIS proporciona enlaces a los documentos públicos y las publicaciones de los proyectos de los programas marco HORIZONTE.

Los enlaces a los documentos y las publicaciones de los proyectos del Séptimo Programa Marco, así como los enlaces a algunos tipos de resultados específicos, como conjuntos de datos y «software», se obtienen dinámicamente de OpenAIRE .

Resultado final

Other (10)

First version of workflow tools published that allows to perform quarterly benchmarks of ML solutions
Hackathon I
Dissemination workshop I
Benchmarking and test infrastructure
Dissemination workshop II
Hackathon II
First version of customised ML solutions for the datasets of D1.2
Final version of workflow tools published
Revised version of customised ML solutions for the updated datasets of D1.2 that are based on the workflow tools of WP2 and optimised based on the feedback from WP3 regarding performance and energy consumption
Performance model for hardware configurations

Documents, reports (15)

Final report on hardware performance benchmarking for ML solutions with the full implementation of the workflow tools of D2.2
Report on tests with a tangent linear and adjoint version of ML emulators with 4DVar
Report on software performance benchmarking for ML solutions from deliverable D1.4
Report on the survey of the workflow, the MAELSTROM protocol and ML requirements

Report on the survey of the workflow the MAELSTROM protocol and ML requirements

Report on hardware performance benchmarking for ML solutions from D1.3 on a number of hardware configurations
Report on hardware performance benchmarking for simplistic ML solutions for benchmark data sets in D1.2 on existing hardware solutions

Report on hardware performance benchmarking for simplistic ML solutions for benchmark data sets in D12 on existing hardware solutions

Report on a survey of MAELSTROM applications and ML tools and architectures
Plan for Dissemination and Communication
Initial list of hardware related requirements for ML solutions in W&C

Initial list of hardware related requirements for ML solutions in WC

Report on software performance benchmarking for ML solutions from deliverable D1.3

Report on software performance benchmarking for ML solutions from deliverable D13

Plan for Gender Balance
Report on the application of ML solutions within the W&C workflow
Report on solution design and architecture blueprint
Roadmap analysis of technologies relevant for ML solutions in W&C

Roadmap analysis of technologies relevant for ML solutions in WC

Report on improved data processing tools, and the weather data loading pipeline designed for large-scale deep learning training for the benchmark datasets from Deliverable D1.1

Open Research Data Pilot (1)

Data Management Plan

Data sets, microdata, etc (1)

First version of datasets and cost functions to develop ML solutions for A1-A6

Websites, patent fillings, videos etc. (1)

Web Portal including Software Collaboration Platform

Publicaciones

Conference proceedings (19)

Almost 5 years of deep learning in Earth system modelling – where do we come from and where do we go

Autores: Dueben, Peter
Publicado en: Edición 1, 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7108876

Stochastic downscaling of meteorological fields with deep neural networks

Autores: Langguth, Michael; Gong, Bing; Ji, Yan; Mozaffari, Amirpasha; Schultz, Martin
Publicado en: Living Planet Symposium 2022, LPS2022, Bonn, Germany, 2022-05-23 - 2022-05-27, Edición 1, 2022
Editor: Living Planet Symposium 2022

Clairvoyant prefetching for distributed machine learning I/O

Autores: Nikoli Dryden; Roman Böhringer; Tal Ben-Nun; Torsten Hoefler
Publicado en: SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Edición 1, 2021
Editor: Association for Computing Machinery New York, NY, United States
DOI: 10.48550/arxiv.2101.08734

efficiently training large-scale neural networks with bidirectional pipelines

Autores: Shigang Li; Torsten Hoefler
Publicado en: SC, 2022, ISBN 9781450384421
Editor: Association for Computing Machinery New York, NY, United States
DOI: 10.1145/3458817.3476145

Near-optimal sparse allreduce for distributed deep learning

Autores: Li, Shigang; Hoefler, Torsten
Publicado en: Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '22), 2022
Editor: ACM
DOI: 10.1145/3503221.3508399

Neural Parameter Allocation Search

Autores: Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, Kate Saenko
Publicado en: ICLR 2022, 2022
Editor: ICLR
DOI: 10.48550/arxiv.2006.10598

Overview of State of the Art Use of ML/AI for Earth System Science

Autores: Dueben
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7081282

PROGRAML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations

Autores: Cummins, Chris; Fisches, Zacharias V.; Ben-Nun, Tal; Hoefler, Torsten; O'Boyle, Michael F P; Leather, Hugh
Publicado en: Cummins , C , Fisches , Z V , Ben-Nun , T , Hoefler , T , O'Boyle , M F P & Leather , H 2021 , PROGRAML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations . in Proceedings of the 38th International Conference on Machine Learning . Proceedings of Machine Learning Research , vol. 139 , pp. 2244-2253 , Thirty-eighth International Conference on Machine Learning , 1, Edición 1, 2021
Editor: 38th International Conference on Machine Learning

High-Performance and Programmable Attentional Graph Neural Networks with Global Tensor Formulations

Autores: Maciej Besta; Pawel Renc; Robert Gerstenberger; Paolo Sylos Labini; Alexandros Ziogas; Tiancheng Chen; Lukas Gianinazzi; Florian Scheidl; Kalman Szenes; Armon Carigiet; Patrick Iff; Grzegorz Kwasniewski; Raghavendra Kanakagiri; Chio Ge; Sammy Jaeger; Jarosław Wąs; Flavio Vella; Torsten Hoefler
Publicado en: SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Edición 1, 2023, ISBN 979-8-4007-0109-2
Editor: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1145/3581784.3607067

Machine Learning for Weather and Predictions

Autores: Dueben
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7081333

Machine Learning for Weather and Climate Prediction

Autores: Dueben, Peter
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.6792121

A Data-Centric Optimization Framework for Machine Learning

Autores: Oliver Rausch; Tal Ben-Nun; Nikoli Dryden; Andrei Ivanov; Shigang Li; Torsten Hoefler
Publicado en: ICS '22: Proceedings of the 36th ACM International Conference on Supercomputing, 2022, ISBN 9781450392815
Editor: Association for Computing Machinery New York, NY, United States
DOI: 10.48550/arxiv.2110.10802

Machine Learning in Weather and Climate Modelling

Autores: Dueben
Publicado en: 2021
Editor: Zenodo
DOI: 10.5281/zenodo.7081199

Challenges and Limitations of Machine Learning for Atmospheric Sciences

Autores: Dueben
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7081632

Productive Performance Engineering for Weather and Climate Modeling with Python

Autores: T. Ben-Nun, L. Groner, F. Deconinck, T. Wicky, E. Davis, J. Dahm, O. Elbert, R. George, J. McGibbon, L. Trümper, E. Wu, O. Fuhrer, T. Schulthess, T. Hoefler
Publicado en: SC'22, 2022
Editor: SC'22

Spatial Mixture-of-Experts

Autores: N. Dryden, T. Hoefler
Publicado en: NeurIPS'22, 2022
Editor: Neural Information Processing Systems 35

Productive Performance Engineering for Weather and Climate Modeling with Python

Autores: Ben-Nun, Tal; Groner, Linus; Deconinck, Florian; Wicky, Tobias; Davis, Eddie; Dahm, Johann; Elbert, Oliver D.; George, Rhea; McGibbon, Jeremy; Trümper, Lukas; Wu, Elynn; Fuhrer, Oliver; Schulthess, Thomas; Hoefler, Torsten
Publicado en: SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Edición 41, 2022, ISSN 2331-8422
Editor: ArXiv.org
DOI: 10.1109/sc41404.2022.00078

Machine Learning at ECMWF

Autores: Peter Dueben
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7100588

The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores

Autores: Maciej Besta; Robert Gerstenberger; Marc Fischer; Michal Podstawski; Nils Blach; Berke Egeli; Georgy Mitenkov; Wojciech Chlapek; Marek Michalewicz; Hubert Niewiadomski; Juergen Mueller; Torsten Hoefler
Publicado en: SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Edición 1, 2023, ISBN 979-8-4007-0109-2
Editor: Proceedings of the International Conference for High Performance Computing
DOI: 10.1145/3581784.3607068

Non-peer reviewed articles (1)

PROGRAML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations

Autores: Cummins, Chris; Fisches, Zacharias V.; Ben-Nun, Tal; Hoefler, Torsten; O'Boyle, Michael F P; Leather, Hugh
Publicado en: Cummins , C , Fisches , Z V , Ben-Nun , T , Hoefler , T , O'Boyle , M F P & Leather , H 2021 , PROGRAML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations . in Proceedings of the 38th International Conference on Machine Learning . Proceedings of Machine Learning Research , vol. 139 , pp. 2244-2253 , International Conference on Machine Learning 2021 , 1/07/21 ., Edición vol. 139, PMLR, pp. 2244-2253,, 2021, ISSN 2640-3498
Editor: PMLR

Peer reviewed articles (26)

Machine learning emulation of gravity wave drag in numerical weather forecasting

Autores: Matthew Chantry; Sam Hatfield; Peter Dueben; Inna Polichtchouk; Tim Palmer
Publicado en: Journal of advances in modeling earth systems, Edición 3, 2021, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2021ms002477

AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning

Autores: Lessig, Christian; Luise, Ilaria; Gong, Bing; Langguth, Michael; Stadtler, Scarlet; Schultz, Martin
Publicado en: arXiv, Edición 37, 2023, ISSN 2331-8422
Editor: USA
DOI: 10.48550/arxiv.2308.13280

CLGAN: a generative adversarial network (GAN)-based video prediction model for precipitation nowcasting

Autores: Y. Ji; Y. Ji; B. Gong; M. Langguth; A. Mozaffari; X. Zhi
Publicado en: Geoscientific Model Development, Vol 16, Pp 2737-2752 (2023), Edición 1, 2023, ISSN 1991-959X
Editor: Copernicus Gesellschaft mbH
DOI: 10.5194/gmd-16-2737-2023

HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformers

Autores: Besta, Maciej; Catarino, Afonso Claudino; Gianinazzi, Lukas; Blach, Nils; Nyczyk, Piotr; Niewiadomski, Hubert; Hoefler, Torsten
Publicado en: arXiv, Edición 34, 2023, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2311.18526

Deep Learning to Estimate Model Biases in an Operational NWP Assimilation System

Autores: Patrick Laloyaux1, Thorsten Kurth2, Peter Dominik Dueben1, and David Hall
Publicado en: Journal of Advances in Modeling Earth Systems, Edición 19422466, 2022, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2022ms003016

Improving Medium-Range Ensemble Weather Forecasts with Hierarchical Ensemble Transformers

Autores: Zied Ben Bouallègue; Jonathan A. Weyn; Mariana C. A. Clare; Jesper Dramsch; Peter Dueben; Matthew Chantry
Publicado en: Artificial Intelligence for the Earth Systems, Edición 34, 2023, ISSN 2769-7525
Editor: Artificial Intelligence for the Earth Systems
DOI: 10.1175/aies-d-23-0027.1

Deep learning for quality control of surface physiographic fields using satellite Earth observations

Autores: Tom Kimpson; Margarita Choulga; Matthew Chantry; Gianpaolo Balsamo; Souhail Boussetta; Peter Dueben; Tim Palmer
Publicado en: eISSN: 1607-7938, Edición 34, 2023, ISSN 1027-5606
Editor: European Geophysical Society
DOI: 10.5194/hess-27-4661-2023

Statistical Modeling of 2-m Temperature and 10-m Wind Speed Forecast Errors

Autores: Ben-Bouallegue, Zied; Cooper, Fenwick; Chantry, Matthew; Düben, Peter; Bechtold, Peter; Sandu, Irina
Publicado en: Monthly Weather Review, Edición 1, 2023, ISSN 1520-0493
Editor: Monthly Weather Review
DOI: 10.1175/mwr-d-22-0107.1

Machine Learning Emulation of 3D Cloud Radiative Effects

Autores: David Meyer; Robin J. Hogan; Peter Dueben; Shannon Mason
Publicado en: Machine Learning Emulation of 3D Cloud Radiative Effects, Edición 2, 2022, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2021ms002550

Neural Graph Databases

Autores: Besta, Maciej; Iff, Patrick; Scheidl, Florian; Osawa, Kazuki; Dryden, Nikoli; Podstawski, Michal; Chen, Tiancheng; Hoefler, Torsten
Publicado en: arXiv, Edición 1, 2022, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2209.09732

PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices

Autores: Osawa, Kazuki; Li, Shigang; Hoefler, Torsten
Publicado en: arXiv, Edición 30, 2022, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2211.14133

Compressing multidimensional weather and climate data into neural networks

Autores: Huang, Langwen; Hoefler, Torsten
Publicado en: arXiv, Edición 4, 2023, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2210.12538

Compressing atmospheric data into its real information content.

Autores: Milan Klöwer, Miha Razinger, Juan J. Dominguez, Peter D. Düben & Tim N. Palmer
Publicado en: Nature Computational Science, 2021, Página(s) Nat Comput Sci 1, 713–724 (2021), ISSN 2662-8457
Editor: Nature Computational Science
DOI: 10.1038/s43588-021-00156-2

Machine Learning Emulation of Urban Land Surface Processes

Autores: David Meyer1,2, Sue Grimmond1, Peter Dueben3, Robin Hogan1,3, and Maarten van Reeuwijk2
Publicado en: Journal of Advances in Modeling Earth Systems, Edición 19422466, 2021, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2021ms002744

Temperature forecasting by deep learning methods

Autores: Gong, Bing; Langguth, Michael; Ji, Yan; Mozaffari, Amirpasha; Stadtler, Scarlet; Mache, Karim; Schultz, Martin G.
Publicado en: Geoscientific Model Development, Vol 15, Pp 8931-8956 (2022), Edición 1, 2022, ISSN 1991-959X
Editor: Copernicus Gesellschaft mbH
DOI: 10.5194/gmd-2021-430

Cached Operator Reordering: A Unified View for Fast GNN Training

Autores: Bazinska, Julia; Ivanov, Andrei; Ben-Nun, Tal; Dryden, Nikoli; Besta, Maciej; Shen, Siyuan; Hoefler, Torsten
Publicado en: arXiv, Edición 35, 2023, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2308.12093

Spatial Mixture-of-Experts

Autores: Dryden, Nikoli; Hoefler, Torsten
Publicado en: Advances in Neural Information Processing Systems 35, Edición 1, 2022, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2211.13491

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Autores: Frantar, Elias; Ashkboos, Saleh; Hoefler, Torsten; Alistarh, Dan
Publicado en: arXiv, Edición 1, 2022, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2210.17323

Challenges and Benchmark Datasets for Machine Learning in the Atmospheric Sciences: Definition, Status, and Outlook

Autores: Peter D. Dueben1, Martin G. Schultz2, Matthew Chantry1, David John Gagne II3, David Matthew Hall4, and Amy McGovern5
Publicado en: Artificial Intelligence for the Earth Systems, Edición 27697525, 2022, ISSN 2769-7525
Editor: American Meteorological Society
DOI: 10.1175/aies-d-21-0002.1

STen: Productive and Efficient Sparsity in PyTorch

Autores: Ivanov, Andrei; Dryden, Nikoli; Ben-Nun, Tal; Ashkboos, Saleh; Hoefler, Torsten
Publicado en: arXiv, Edición 40, 2023, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2304.07613

A Generative Deep Learning Approach to Stochastic Downscaling of Precipitation Forecasts

Autores: Harris, Lucy; McRae, Andrew T. T.; Chantry, Matthew; Dueben, Peter D.; Palmer, Tim N.
Publicado en: Crossref, Edición 33, 2022, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2022ms003120

A High-Performance Design, Implementation, Deployment, and Evaluation of The Slim Fly Network

Autores: Blach, Nils; Besta, Maciej; De Sensi, Daniele; Domke, Jens; Harake, Hussein; Li, Shigang; Iff, Patrick; Konieczny, Marek; Lakhotia, Kartik; Kubicek, Ales; Ferrari, Marcel; Petrini, Fabrizio; Hoefler, Torsten
Publicado en: arXiv, Edición 32, 2023, ISSN 2331-8422
Editor: arXiv
DOI: 10.48550/arxiv.2310.03742

Building Tangent-Linear and Adjoint Models for Data Assimilation With Neural Networks

Autores: Sam Hatfield; Matthew Chantry; Peter Dueben; Philippe Lopez; Alan J. Geer; Tim Palmer
Publicado en: journal of advances in modeling earth systems, Edición 2, 2021, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2021ms002521

A comparison of data-driven approaches to build low-dimensional ocean models

Autores: Niraj Agarwal; Dmitri Kondrashov; Dmitri Kondrashov; Peter Dueben; E. A. Ryzhov; Pavel Berloff; Pavel Berloff
Publicado en: journal of advances in modeling earth systmes, Edición 3, 2021, ISSN 1942-2466
Editor: American Geophysical Union
DOI: 10.1029/2021ms002537

Further analysis of cGAN: A system for Generative Deep Learning Post-processing of Precipitation

Autores: Cooper, Fenwick C.; McRae, Andrew T. T.; Chantry, Matthew; Antonio, Bobby; Palmer, Tim N.
Publicado en: Further analysis of cGAN: A system for Generative Deep Learning Post-processing of Precipitation, Edición 1, 2023, ISSN 1520-0493
Editor: arXiv
DOI: 10.48550/arxiv.2309.15689

Bridging observations, theory and numerical simulation of the ocean using machine learning

Autores: Maike Sonnewald; Maike Sonnewald; Maike Sonnewald; Redouane Lguensat; Daniel C. Jones; Peter Dueben; Julien Brajard; Venkatramani Balaji; Venkatramani Balaji
Publicado en: Environmental Research Letters, Edición 3, 2021, ISSN 1748-9326
Editor: Institute of Physics Publishing
DOI: 10.1088/1748-9326/ac0eb0

Other (15)

Machine Learning at ECMWF

Autores: Dueben
Publicado en: 2022
Editor: Zenodo
DOI: 10.5281/zenodo.7081735

Machine learning for weather and climate predictions

Autores: Dueben, Peter
Publicado en: Edición 1, 2021
Editor: Zenodo
DOI: 10.5281/zenodo.5152016

How to reduce numerical precision in weather and climate simulations

Autores: Dueben, Peter
Publicado en: Edición 4, 2021
Editor: Zenodo
DOI: 10.5281/zenodo.5151995

ENS-10: A Dataset For Post-Processing EnsembleWeather Forecasts

Autores: Ashkboos, Saleh and Huang, Langwen and Dryden, Nikoli and Ben-Nun, Tal and Dueben, Peter and Gianinazzi, Lukas and Kummer, Luca and Hoefler, Torsten
Publicado en: 2022
Editor: arXiv
DOI: 10.48550/arxiv.2206.14786

Machine learning, high-performance computing and numerical weather prediction

Autores: Dueben, Peter D.
Publicado en: Edición 33, 2021
Editor: Zenodo
DOI: 10.5281/zenodo.5533706

Challenges when preparing machine learning tools for use in operational weather predictions

Autores: Dueben, Peter D.
Publicado en: Edición 5, 2021
Editor: Zenodo
DOI: 10.5281/zenodo.5533639

MAELSTROM: First benchmarks at ISC22

Autores: Nassyr, Stepan
Publicado en: Edición 1, 2022
Editor: JSC Accelerating Devices Lab
DOI: 10.34732/xdvblg-tufftf

Machine learning for weather and climate prediction

Autores: Dueben, Peter Dominik
Publicado en: Edición 5, 2023
Editor: Zenodo
DOI: 10.5281/zenodo.8025367

Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization

Autores: Pacchiardi, Lorenzo and Adewoyin, Rilwan and Dueben, Peter and Dutta, Ritabrata
Publicado en: 2021
Editor: arXiv
DOI: 10.48550/arxiv.2112.08217

The next decade of machine learning at ECMWF

Autores: Dueben, Peter
Publicado en: Edición 4, 2021
Editor: Zenodo
DOI: 10.5281/zenodo.5152043

Km-scale weather models, machine-learned weather models, and km-scale machine-learned weather models

Autores: Dueben, Peter Dominik
Publicado en: Edición 5, 2023
Editor: Zenodo
DOI: 10.5281/zenodo.8025410

Machine learning and predictability of weather and climate

Autores: Dueben, Peter Dominik
Publicado en: Edición 1, 2022
Editor: Zenodo
DOI: 10.5281/zenodo.8025284

Machine Learning for Weather and Climate Modelling

Autores: Dueben, Peter Dominik
Publicado en: Edición 32, 2022
Editor: Zenodo
DOI: 10.5281/zenodo.8025265

Downscaling and global high-resolution modelling

Autores: Dueben, Peter Dominik
Publicado en: Edición 5, 2022
Editor: Zenodo
DOI: 10.5281/zenodo.8025314

Machine Learning at the European Centre for Medium-Range Weather Forecasts -- Some notes on the progress 2018-2022

Autores: Dueben, Peter Dominik
Publicado en: Edición 30, 2022
Editor: Zenodo
DOI: 10.5281/zenodo.8025270