Periodic Reporting for period 1 - RED-SEA (Network Solution for Exascale Architectures)
Berichtszeitraum: 2021-04-01 bis 2022-09-30
Network interconnects play an enabling role in HPC systems – and this will be even truer for the coming Exascale systems that will rely on higher node counts and increased use of parallelism and communication. Moreover, next-generation HPC and data-driven systems will be powered by heterogeneous computing devices, including low-power Arm and RISC-V processors, high-end CPUs, vector acceleration units and GPUs suitable for massive single-instruction multiple-data (SIMD) workloads, as well as FPGA and ASIC designs tailored for extremely power-efficient custom codes.
The compute units will be surrounded by distributed, heterogeneous (often deep) memory hierarchies, including high-bandwidth memories and fast devices offering microsecond-level access time. At the same time, modern data-parallel processing units such as GPUs and vector accelerators can crunch data at amazing rates (tens of TFLOPS). In this landscape, the network may well become the next big bottleneck, similar to memory in single node systems.
RED-SEA will build upon the European interconnect BXI (BullSequana eXascale Interconnect), together with standard and mature technology (Ethernet) and previous EU-funded initiatives to provide a competitive and efficient network solution for the exascale era and beyond. This involves developing the key IPs and the software environment that will deliver:
- scalability, while maintaining an acceptable total cost of ownership and power efficiency;
- virtualization and security, to allow various applications to efficiently and safely share an HPC system;
- Quality-of-service and congestion management to make it possible to share the platform among users and applications with different demands;
- reliability at scale, because fault tolerance is a key concern in a system with a very large number of components;
- support of high-bandwidth low-latency HPC Ethernet, as HPC systems increasingly need to interact securely with the outside world, including public clouds, edge servers or third party HPC systems;
- support of heterogeneous programming model and runtimes to facilitate the convergence of HPC and HPDA;
- support for low-power processors and accelerators.
The compute units will be surrounded by distributed, heterogeneous (often deep) memory hierarchies, including high-bandwidth memories and fast devices offering microsecond-level access time. At the same time, modern data-parallel processing units such as GPUs and vector accelerators can crunch data at amazing rates (tens of TFLOPS). In this landscape, the network may well become the next big bottleneck, similar to memory in single node systems.
RED-SEA will build upon the European interconnect BXI (BullSequana eXascale Interconnect), together with standard and mature technology (Ethernet) and previous EU-funded initiatives to provide a competitive and efficient network solution for the exascale era and beyond. This involves developing the key IPs and the software environment that will deliver:
- scalability, while maintaining an acceptable total cost of ownership and power efficiency;
- virtualization and security, to allow various applications to efficiently and safely share an HPC system;
- Quality-of-service and congestion management to make it possible to share the platform among users and applications with different demands;
- reliability at scale, because fault tolerance is a key concern in a system with a very large number of components;
- support of high-bandwidth low-latency HPC Ethernet, as HPC systems increasingly need to interact securely with the outside world, including public clouds, edge servers or third party HPC systems;
- support of heterogeneous programming model and runtimes to facilitate the convergence of HPC and HPDA;
- support for low-power processors and accelerators.
During the period, the progress has been made in each work-package. The work performed is in-line with the workplan.
The network requirements and architecture have been defined as well as an initial list of benchmarks to evaluate the network thanks to the co-design approach.
LinkTest is the primary benchmark tool used to evaluate latency and bandwidth in the various message size ranges, it is ready for BXI integration through the Portals 4 API.
The application partners have started porting, extension, optimisation and tunning of their applications on the RED-SEA testbeds and simulators:
- NEST/INFN: an extended analysis of the NEST application was made to provide recommendations for the network architecture. Separate timers were added for the computation and communication stages
- LAMMPS & SOW/EXACT: extensive benchmarks were run on Dibona and TGCC clusters. Two parallel versions of the SOM algorithm were developed
6 hardware testbeds have been setup with different targets: (i) the analysis of the BXI architecture, (ii) the definition of the application requirement for the network. (iii) to develop and debug applications, benchmarks, and environments (optimization of MPI libraries), and (iv) development, validation, and evaluation of performance of the IPs proposed in technical work packages. All testbeds are running and stable.
Moreover, the simulation platforms have also been defined and enhanced. Main work performed is described below:
- SAURON: migrated to OMNET++ framework to version 6.0 as well as the integration with INET 4.3 framework
- COSSIM simulator: an easy-to-use Virtual Machine has been created which includes the COSSIM simulator with the latest version of GEM5 so as to have a full Linux-based system simulation
Regarding the work performed on High Performance Ethernet, Internet Protocol over BXI2 kernel module has been optimised with the improvement up to x4 times. Moreover, 3 Architectures specifications have been defined on Ethernet Gateway IP, Low-Latency Ethernet MAC & PCS as well as transport layer. The implementation of the specifications has been started, First RTL version of Low-Latency Ethernet MAC & PCS has been completed.
On the efficient Network Resource management side, the congestion characterization has been finished and first results of congestion management have been obtained.
Preliminary results of BXI2 Link Layer design for FPGA have been produced and they meet the expectations. The new RDMA HW Engine has been made, the small size of transfers has been optimised due to pipelined design.
Finally, the design of a sPIN (Stream Processing In-Network) has been completed; a first prototype of MPC with multi-rail feature has been developed; the extension of ParaStation MPI supporting BXI networks has been worked out.
The network requirements and architecture have been defined as well as an initial list of benchmarks to evaluate the network thanks to the co-design approach.
LinkTest is the primary benchmark tool used to evaluate latency and bandwidth in the various message size ranges, it is ready for BXI integration through the Portals 4 API.
The application partners have started porting, extension, optimisation and tunning of their applications on the RED-SEA testbeds and simulators:
- NEST/INFN: an extended analysis of the NEST application was made to provide recommendations for the network architecture. Separate timers were added for the computation and communication stages
- LAMMPS & SOW/EXACT: extensive benchmarks were run on Dibona and TGCC clusters. Two parallel versions of the SOM algorithm were developed
6 hardware testbeds have been setup with different targets: (i) the analysis of the BXI architecture, (ii) the definition of the application requirement for the network. (iii) to develop and debug applications, benchmarks, and environments (optimization of MPI libraries), and (iv) development, validation, and evaluation of performance of the IPs proposed in technical work packages. All testbeds are running and stable.
Moreover, the simulation platforms have also been defined and enhanced. Main work performed is described below:
- SAURON: migrated to OMNET++ framework to version 6.0 as well as the integration with INET 4.3 framework
- COSSIM simulator: an easy-to-use Virtual Machine has been created which includes the COSSIM simulator with the latest version of GEM5 so as to have a full Linux-based system simulation
Regarding the work performed on High Performance Ethernet, Internet Protocol over BXI2 kernel module has been optimised with the improvement up to x4 times. Moreover, 3 Architectures specifications have been defined on Ethernet Gateway IP, Low-Latency Ethernet MAC & PCS as well as transport layer. The implementation of the specifications has been started, First RTL version of Low-Latency Ethernet MAC & PCS has been completed.
On the efficient Network Resource management side, the congestion characterization has been finished and first results of congestion management have been obtained.
Preliminary results of BXI2 Link Layer design for FPGA have been produced and they meet the expectations. The new RDMA HW Engine has been made, the small size of transfers has been optimised due to pipelined design.
Finally, the design of a sPIN (Stream Processing In-Network) has been completed; a first prototype of MPC with multi-rail feature has been developed; the extension of ParaStation MPI supporting BXI networks has been worked out.
In the next period, we are focusing on the achievement of the 10 objectives and their associated 11 KPIs. We can highlight some of KPIs among RED-SEA key developments:
- Scalability enhancements: BXI already supports up to 64k nodes. With the outcome of the READ-SEA project, BXI will be able to further scale to larger sizes and with minimal congestion
- Ethernet bridging: with the seamless BXI integration with Ethernet to better integrate the exascale system within the datacentre and the global federation of HPC and data resources
Major project outcomes enabling the BXI3 development is one of the potential impacts the project expect to make. Our work on enriching EU ecosystem of interconnect technology will foster the community adoption around the generated hardware & software ecosystems.
The results made at the end of the project is expecting to have major impact and to contribution to the future EU Interconnect.
- Scalability enhancements: BXI already supports up to 64k nodes. With the outcome of the READ-SEA project, BXI will be able to further scale to larger sizes and with minimal congestion
- Ethernet bridging: with the seamless BXI integration with Ethernet to better integrate the exascale system within the datacentre and the global federation of HPC and data resources
Major project outcomes enabling the BXI3 development is one of the potential impacts the project expect to make. Our work on enriching EU ecosystem of interconnect technology will foster the community adoption around the generated hardware & software ecosystems.
The results made at the end of the project is expecting to have major impact and to contribution to the future EU Interconnect.