Periodic Reporting for period 2 - ExaNoDe (European Exascale Processor Memory Node Design)
Período documentado: 2017-04-01 hasta 2019-06-30
The primary objective of ExaNoDe was to deliver a compute element, integrating core technologies capable of relevance for exascale computing. Innovative nanotechnology and manufacturing were required as were the necessary system and runtime software to facilitate applications use. ExaNoDe built on multiple prior European initiatives for scalable computing utilizing low-power processors and advanced nanotechnologies and on the FP7 project EUROSERVER (UNIMEM system architecture).
Key technology objectives linked to the achievement of that objective were:
a) Building blocks for heterogeneous computing devices targetting HPC systems.
b) The use and validation of 3D silicon-on-silicon level fabrication technology and advanced process geometry.
c) The integration of devices in-package within a daughter-board compatible with the ExaNeSt H2020 prototype.
d) A system based on the use of arm cores equipped with the OS and runtime support for resource sharing.
e) A performance projection of the potential impact of ExaNoDe technologies on future HPC processors.
The role and scope of the building block design in a System-on-Chip was to facilitate the silicon development and validation in the HPC context of the 3D-integration technologies. The provision of a system and runtime software stack exploiting the UNIMEM system architecture had the objective to support multi-node programming of applications.
Key technology objectives linked to the achievement of that objective were:
a) Building blocks for heterogeneous computing devices targetting HPC systems.
b) The use and validation of 3D silicon-on-silicon level fabrication technology and advanced process geometry.
c) The integration of devices in-package within a daughter-board compatible with the ExaNeSt H2020 prototype.
d) A system based on the use of arm cores equipped with the OS and runtime support for resource sharing.
e) A performance projection of the potential impact of ExaNoDe technologies on future HPC processors.
The role and scope of the building block design in a System-on-Chip was to facilitate the silicon development and validation in the HPC context of the 3D-integration technologies. The provision of a system and runtime software stack exploiting the UNIMEM system architecture had the objective to support multi-node programming of applications.
The initial project plan to develop compute capability combined with 3D-integration technology was to procure chiplets (System-on-Chip to be stacked on a silicon interposer) that met ExaNoDe’s requirements. At project commencement it became clear that the chiplet market maturity was insufficient to allow chiplet procurement meeting our needs, while an in-project development was infeasible for budget reasons. Nevertheless, recent developments confirm the business relevance, with adoption of computing modules including chiplets illustrated by recent announcements from AMD .
The consortium thus used an FPGA compute unit with embedded ARM-v8 processors. The FPGA configurable logic was used to support UNIMEM. The 3D integration was maintained as an architectural vision and prototyped in a 3D-Integrated-Circuit (3D-IC) via the chiplet design, manufacturing and assembly onto a silicon interposer.
The two ExaNoDe prototypes allowed for focused developments of software on FPGA-based compute nodes and 3D integration technology development and manufacturing. Both prototypes realised in daughter boards compatible with the ExaNeSt project prototype. The two ExaNoDe prototypes use different variants of the Multi-Chip-Module: variant 1 (MCM-1) without 3-D integration, but with the necessary software stack to include the various programming models/run-times and execution of mini-applications; variant 2 (MCM-2) with 3-D integration including two chiplets stacked on one silicon interposer with the necessary functionality in the FPGA to facilitate the 3D-IC evaluation.
ExaNoDe’s main achievements cover all aspects of heterogeneous integration including:
• Architecture and design:
o Innovative, high-speed and low-power interconnect for chiplets via a silicon interposer.
o A Convolutional Neural Network (CNN) accelerator hardware IP within the chiplet.
o A chiplet System-on-Chip (SoC) in a 28FDSOI technology node.
• Advanced integration:
o 3D integration of chiplets on an active silicon interposer with approximately 50,000 high density (20 µm pitch) connections.
o Advanced package integration with two FPGA bare dies including ARMv8 cores, one interposer and 43 decoupling capacitors in a 68.5 mm ×55 mm Multi-chip-Module (MCM).
o Integration of two MCMs on a 260 mm x 120 mm daughter board.
• System software:
o Development of a complete SW stack including UNIMEM-based system software and middleware; Runtime libraries optimized for the UNIMEM architecture (OmpSs, MPI, OpenStream, GPI); Checkpointing technology for virtualisation; A set of mini-applications for benchmarking purposes.
• Benchmarking and evaluation:
o Porting and evaluation of some representative mini-applications on the daughter-board implementing the MCM-1 module with FPGA.
o A performance projection of ExaNoDe technologies into a strawman architecture representative of upcoming HPC processors.
The project also generated a number of “lessons learned” that are of significant value for the project participants and potentially for the European Commission’s HPC programme. It should be highlighted that planning for innovative, leading-edge hardware developments poses a challenge for R&D projects, and despite continuous risk monitoring and the use of mitigation plans, the original project time-lines could not be met. Key aspects were:
• Advanced packaging technologies generated significant challenges for assembly and integration (not all could be met within the extended project period).
• Providing the related full software stack implies a significant effort in debugging and performance tuning (which exceeded original project plans).
For the core hardware developments, especially advanced integration, highlighted aspects are:
• The project validated a process for 3-D assembly (microbumps of 10 µm diameter with a 20 µm pitch) more advanced than current industrial standards.
• Since single component failure can impact overall system functionality, mitigation approaches should be defined in the design stage.
• Known Good Die (KGD) tests performed on the active interposer could have improved pre-assembly selection.
• Additional board-level simulation should be performed to avoid errors being detected at the time of bring-up.
The consortium thus used an FPGA compute unit with embedded ARM-v8 processors. The FPGA configurable logic was used to support UNIMEM. The 3D integration was maintained as an architectural vision and prototyped in a 3D-Integrated-Circuit (3D-IC) via the chiplet design, manufacturing and assembly onto a silicon interposer.
The two ExaNoDe prototypes allowed for focused developments of software on FPGA-based compute nodes and 3D integration technology development and manufacturing. Both prototypes realised in daughter boards compatible with the ExaNeSt project prototype. The two ExaNoDe prototypes use different variants of the Multi-Chip-Module: variant 1 (MCM-1) without 3-D integration, but with the necessary software stack to include the various programming models/run-times and execution of mini-applications; variant 2 (MCM-2) with 3-D integration including two chiplets stacked on one silicon interposer with the necessary functionality in the FPGA to facilitate the 3D-IC evaluation.
ExaNoDe’s main achievements cover all aspects of heterogeneous integration including:
• Architecture and design:
o Innovative, high-speed and low-power interconnect for chiplets via a silicon interposer.
o A Convolutional Neural Network (CNN) accelerator hardware IP within the chiplet.
o A chiplet System-on-Chip (SoC) in a 28FDSOI technology node.
• Advanced integration:
o 3D integration of chiplets on an active silicon interposer with approximately 50,000 high density (20 µm pitch) connections.
o Advanced package integration with two FPGA bare dies including ARMv8 cores, one interposer and 43 decoupling capacitors in a 68.5 mm ×55 mm Multi-chip-Module (MCM).
o Integration of two MCMs on a 260 mm x 120 mm daughter board.
• System software:
o Development of a complete SW stack including UNIMEM-based system software and middleware; Runtime libraries optimized for the UNIMEM architecture (OmpSs, MPI, OpenStream, GPI); Checkpointing technology for virtualisation; A set of mini-applications for benchmarking purposes.
• Benchmarking and evaluation:
o Porting and evaluation of some representative mini-applications on the daughter-board implementing the MCM-1 module with FPGA.
o A performance projection of ExaNoDe technologies into a strawman architecture representative of upcoming HPC processors.
The project also generated a number of “lessons learned” that are of significant value for the project participants and potentially for the European Commission’s HPC programme. It should be highlighted that planning for innovative, leading-edge hardware developments poses a challenge for R&D projects, and despite continuous risk monitoring and the use of mitigation plans, the original project time-lines could not be met. Key aspects were:
• Advanced packaging technologies generated significant challenges for assembly and integration (not all could be met within the extended project period).
• Providing the related full software stack implies a significant effort in debugging and performance tuning (which exceeded original project plans).
For the core hardware developments, especially advanced integration, highlighted aspects are:
• The project validated a process for 3-D assembly (microbumps of 10 µm diameter with a 20 µm pitch) more advanced than current industrial standards.
• Since single component failure can impact overall system functionality, mitigation approaches should be defined in the design stage.
• Known Good Die (KGD) tests performed on the active interposer could have improved pre-assembly selection.
• Additional board-level simulation should be performed to avoid errors being detected at the time of bring-up.
The ExaNoDe MCM-1 prototype demonstrated a fully-integrated compute node prototype with a high-performance and high-productivity software stack adequately evaluated with representative mini-applications. The ExaNoDe MCM-2 prototype demonstrated the architecture, design, and manufacturing feasibility of advanced 3D-integration technologies. By combining and projecting performance impacts of the two ExaNoDe prototypes in a combined “strawman architecture”, ExaNoDe demonstrates a technological approach for the modular and heterogeneous designs required by computing systems targeting exascale levels.
ExaNoDe technologies and development methodologies will contribute to the EU HPC programme that will select and develop baseline for the next generation of European Processors to be used in future exascale systems. In particular, the European Processor Initiative (EPI) relies on ExaNoDe chiplet design and advanced integration knowhow to build its Common Platform, which will be the backbone of the European processor roadmap, whereas the ExaNoDe system software provides foundational components within the software platform of the EuroExa project addressing co-design for resilient exascale computing.
ExaNoDe has been collaborating with the projects ExaNeSt and ECOSCALE – covered by a Memorandum of Understanding signed by the coordinators of all of the projects – ensuring consistency in our common technological approach and reinforcing common scientific dissemination activities. The detailed technical exchange between the projects was exemplified by the intermediate use of ExaNeSt’s QFDB boards to allow ExaNoDe software developments to proceed when ExaNoDe hardware was delayed.
ExaNoDe technologies and development methodologies will contribute to the EU HPC programme that will select and develop baseline for the next generation of European Processors to be used in future exascale systems. In particular, the European Processor Initiative (EPI) relies on ExaNoDe chiplet design and advanced integration knowhow to build its Common Platform, which will be the backbone of the European processor roadmap, whereas the ExaNoDe system software provides foundational components within the software platform of the EuroExa project addressing co-design for resilient exascale computing.
ExaNoDe has been collaborating with the projects ExaNeSt and ECOSCALE – covered by a Memorandum of Understanding signed by the coordinators of all of the projects – ensuring consistency in our common technological approach and reinforcing common scientific dissemination activities. The detailed technical exchange between the projects was exemplified by the intermediate use of ExaNeSt’s QFDB boards to allow ExaNoDe software developments to proceed when ExaNoDe hardware was delayed.