With reference to the above identified objectives, the following work was done:
• Development of a scalable ML framework and integration of several architectures (CPU, GPUs, FPGAs).
The Experimented Researcher (ER), together with the Turin group, has deployed and tested the ML framework, which is available on opportunistic resources of the Turin INFN computing centre. The framework is used by students of the course Big data and Machine Learning, held by the ER for doctoral students, and by researchers of the University of Torino for a variety of applications (analysis of MRI data, fast silicon sensors, log files). The computing applications are run on virtual clusters deployed on top of the physical infrastructure. Task scheduling is managed by an orchestration layer (Kubernetes), leveraging Docker containers to define and isolate the runtime environment. The virtual clusters developed to execute ML workflows are accessed through a web interface based on JupyterHub. When a user authenticates on the Hub, a notebook server is created as a containerized application. A set of libraries and helper functions is provided to execute a parallelized ML task by automatically deploying a Spark driver and several Spark execution nodes as Docker containers. This solution automates the delivery of the software stack required by a typical ML workflow, and enables scalability by allowing the execution of ML tasks, including training, over commodity (i.e. CPUs) or high-performance (i.e. GPUs) resources distributed over different hosts across a network.
• Promotion of ML techniques within the LHC community and beyond, by creating links with the local scientific community, and within international collaborations in the field of High Energy Physics (HEP).
In this context, the ER is coordinating a cross-experiment activity, named Operational Intelligence (OI), aimed at developing and collecting tools for the application of artificial intelligence (AI) to computing operations for large scientific collaborations. The ER started collaborations with several groups interested in ML applications for earct sciences and detector developments at the University of Turin. She holds a course for Ph.d students on Big Data Science and Machine Learning.
• Application to the use case of the optimisation of CMS computing operations.
The ER is coordinating all monitoring activities for the CMS computing system: managing the infrastructure that collects all logging information, providing access and support to the various groups involved. Such infrastructure is vital to provide the various teams all necessary information to steer computing operations. An automatic alert system is in place to notify the relevant teams of subsystem failures. An intelligent system providing combination, silencing, and grouping of alerts coming from different sources is currently under development.
A list of dissemination and communication activities is given below:
• The following conferences were attended by the ER:
◦ CHEP (Computing in High Energy Physics) 2018 in Sofia in July 2018;
◦ CHEP 2019 in Victoria (Australia) in November 2019;
◦ 3rd Rucio Community Workshop in FNAL in March 2020.
• The following publications were prepared:
◦ The CMS monitoring infrastructure and applications, submitted to Computing and Software for Big Science;
◦ Operational Intelligence for Distributed Computing Systems for Exascale Science, accepted for publication on EPJ Web Conf;
◦ Big data solutions for CMS computing monitoring and analytics, accepted for publication on EPJ Web Conf;
◦ Delivering a machine learning course on HPC resources, submitted for publication on EPJ Web Conf.
• Outreach activities:
◦ Speaker at the Premiere of the movie “Almost Nothing”,
◦ Shifter at the INFN stand at the Turin Book Exposition (Salone del Libro),
◦ Partecipation to “Pint of Science” nights in Turin,
