Periodic Reporting for period 2 - AIDA (Artificial Intelligence Data Analysis) Reporting period: 2019-09-01 to 2022-02-28 Summary of the context and overall objectives of the project AIDA uses some of the most advanced data analysis techniques to analyse the data available from space missions and from space simulations. AIDA identifies three major trends in our field of research and in the wider society. First, there has been a tremendous accumulation of data from space studies. Every new space mission accumulates new data. But every new mission also increase the amount of data, its quality and resolution compared with previous missions, leading to an ever increasing quantity of data. Similarly, the computer simulations of space science are becoming ever more complex and their results are more detailed and rich, also leading to a rapidly increasing mass of data available. All this data is becoming progressively more openly accessible. The data from space missions is now regularly made available to the public and even the results of simulations start to be shared in public databases. General databases of space data are also being created. AIDA intends to develop tools to make this data more readily accessible to the public, creating common standards for storing data when such standards do not yet exist while using community standards whenever possible. This advancement will be achieved by federating existing tools and creating new ones into AIDApy, the main outcome of the AIDA project: a software tool to gather and analyse space-relevant data from observations and simulations. Second, the vast amount of space data accumulated by past missions and past simulation studies and the new data generated every day by new and ongoing missions and by new simulations is becoming so large that a single human brain cannot grasp its entire complexity. Luckily, a trend in sweeping society to address this very same problem arising in every aspect of human activities: machine learning and artificial intelligence (AI). AIDA wants to bring the latest developments in AI to the space arena (AI in AIDA is indeed Artificial Intelligence for Data Analysis, AIDA). The competences and passions needed to be a space scientist are quite different from those of an expert in artificial intelligence. AIDA brings together expertise in computer science, AI and space science to create a new tool AIDApy that will make the use of AI simpler for the space scientist and even the general public. Third, there is an ongoing trend in all big data analysis and in AI in particular to move towards a free open source language for computers called python. The space community is traditionally linked primarily with MATLAB and IDL. These are commercial tools of highly refined tools that come at a very high cost, although non profit organisations and students can obtain better deals. To truly democratize science, including the so-called trend in citizen science, to promote its spread in developing countries and to simplify access to the data and its analysis to all, python provides a free alternative. Python potentially has even more features than those commercial alternatives and many science communities are currently making the transitions to it. AIDA aims at fostering this same transition to python also for the space community, collecting in AIDApy (where py stands for python) tools previously based on non-free languages. And developing new tools based on the most advanced tools in the state of the art. As this brief summary clarifies, the aim of AIDA is as much scientific as it is directed at transforming the way society can access the results of space missions and space simulations, making the data accessible to all and giving to all, including non expert citizens, access to the most modern Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far AIDA has two main outcomes: AIDApy, a community-based new tool for analysing space data based on python and on the use of machine learning tools and AIDAdb, a database of high level data from simulations and observation that supports the use if AIDApy. AIDApy and AIDAdb are tools that allow users to obtain space data and analyse it using the python language, freely available to all. The development of both tools progresses on schedule and both have been released to the public. AIDApy includes packages to retrieve data from remote sources, such as space missions and space simulations, packages to make statistical analysis of space data, packages to synthetize virtual satellite observations within computer simulations, packages to deploy AI techniques to the analysis of space data. For the latter task, AIDA embarked into an important task: analysing all the progress made in recent months by such tech giants as google, intel and facebook in developing tools for machine learning (ML) and artificial intelligence (AI). We compared all the major AI and ML software for a suite of general tasks and for specific tasks relative to the analysis of space data and to the forecasting of space weather impacts on the Earth. The results of this study led to the selection of a specific tool. In a quantitative very detailed analysis presented as one of our deliverables AIDA explains the reasons for this choice. We then proceeded to develop each component of AIDApy, producing the first components to:Link AIDApy with some of the most important space databases (e.g. OMNIweb) and space missions (MMS mission of NASA) to retrieve data from remote sources;Interfaces with computer codes for space simulations such as OpenGGCM for modelling the EArth environment and iPic3D for modelling energy releases in space;Tools to deploy advanced statistical tools to space data;Tools to synthetize observations from computer simulations that have the same format and can be directly compared with space missions;AI techniques to analyse space data and to make forecast of space weather events. Help improve space weather predictions with methods to include data in space weather physics-based simulations and use AI to make space weather predictions AIDAdb is a database collecting the results of specific cases of application of AIDApy. Users of AIDApy external to the AIDA consortium have access to all this data and all these tools. But to make full use of the resources they need examples of how to use them. AIDAdb provides all the data and all the correct solutions to these test cases. Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) The main objective of AIDA to achieve this goal was the creation of AIDApy, a software tool written in the free language python that gives the user access to data of space science and to the methods needed to analyse it, chief among them the tools of AI now revolutionising every aspect of life. In support of AIDApy we also created the AIDAdb database of examples of use of AIDApy. In it, new users can find specific data sets not already available in the open domain and the results of the analysis of space events using AIDApy. The goal is to provide the users and even common citizens with examples to learn how to access and study space data. To reach this end, two schools were organised with teaching material and exercises all designed in the free language python that anybody can freely use and obtain freely from the WWW.AIDA-SPACE.EU web site. Left: machine learning of solar wind data with self-organising maps. Right: solar wind turbulence Related documents AIDA_logo.png AIDAposter_October2020.jpg flyer_AIDA_page_2.jpg flyer_AIDA_page_1.jpg
