Final Report Summary - HPC-EUROPA (Pan-European research infrastructure on high performance computing for the science of 21st century)
The HPC-EUROPA was an Integrated Infrastructure Initiative operated by six leading HPC centres and five centres of excellence working on research on HPC tools and methods. The fundamental aim of the project was to provide advanced computational services in an integrated way to the European research community working at the forefront of science. The project's aim was to deliver a wide spectrum of services, including access to first-class HPC platforms for all European researchers, provision of an advanced computational environment, technical support and training. Main focus was to support European researchers to preserve a competitive edge on the world stage. Moreover a background objective of the joint research activities (JRA) and networking activities (NA) was a contribute towards fostering a culture of cooperation, generating critical mass for the evolution of computational activities, and driving new advances in HPC in the context of the European Research Area (ERA).
The aim of the two JRAs was to improve the overall performance of HPC in Europe. The first of these had to do with tools for measuring the performance of programs, especially those that have recently been 'ported' to a supercomputer through the project's transnational access (TA) activities. The second was focussed on creating new and simplified ways to access the resources of grid computing systems.
Networking activities in HPC-EUROPA covered the management and evaluation of the project itself, a videoconferencing system, and new systems to ensure that data from finished projects can be stored, retrieved and shared easily.
More in detail, the very ultimate goal of the project was to allow European researchers to maintain a competitive edge on the world stage, delivering the following services:
- access to HPC platforms, unique at national level, for all European researchers;
- provision of an advanced computational environment;
- technical support and specialist tutoring in the domain of HPC.
The core project was organised around the TA activities, and more specifically around the deployment and operation of a 'virtual global infrastructure'.
The JRA and NA aim was to contribute towards the previous mentioned main deployment, fulfilling the following main roles:
- the development and deployment of new high-level services for TA users across the participants;
- the improvement and integration of existing services available at each single participant site and their integration and extension;
- the understanding of users' requirements;
- the promotion of best practices amongst the user community.
Tangible deliverables and results of HPC-EUROPA foreseen for the project were:
- TA service provision;
- Adaptation and integration of a selected toolset into the AccessGrid technology to establish a collaboration and consultation infrastructure;
- realisation of an efficient and portable data management and information retrieval infrastructure for real-life computational science applications in global networks;
- deployment of powerful performance analysis tools that are uniform across HPC platforms, extend the scalability and usability of these tools and development joint methodologies for performance analysis;
- provision of uniform, flexible and intuitive user access to HPC resources from anywhere, via single point of access portal, as well as administration tools for maintaining a grid environment of the TA centres.
HPC-EUROPA was organised around its TA, provided by six sites across Europe: BSC in Spain, CINECA in Italy, EPCC in the United Kingdom, HLRS in Germany, IDRIS in France and SARA in the Netherlands. In this model, the four networking activities and the two joint research activities were designed to facilitate and improve the TA. The NAs mainly aimed to support TA service provision directly, for example by the integration of the six TA centres through the specific development of access grid technology. The JRAs constituted an research and development (R&D) effort to develop valuable new tools to exploit the capability of the TA service provision, in example by means of new performance evaluation tools such as Paraver, especially deployed according to the technical specifications defined by the HPC-EUROPA partners.
The first important achievement reached by the consortium was the set up of an effective, impartial and renowned selection panel, made up by 14 eminent European professors from different scientific fields.
Besides the AUs requested for each project, the six TA partners provided to their visitors a lot of 'side utilities' that helped to create a real HPC focused network and global environment. Samples of those benefits were additional activities tailored specifically to the needs of visitors:
- named contact providing individual technical consultancy;
- access to local training courses;
- collaborative virtual 'surgeries' via AccessGrid;
- centralised VLE providing interactive online training material, technical reports, discussion forum etc.
During the first year, a total of 270 applications were made to the TA programme. The selection panel offered places to 188 user projects, 96 of which started - and in most cases were completed - in 2004, with an acceptance rate of 70 %. These users spent some 155 visitor-months at the 6 different infrastructures, and used about 750 000 Allocation Units (AUs). The AU is defined as 'the computational power delivered by a computer executing for one hour at the sustained rate of one GFlops/s'.
During the second year, 305 applications were made to the TA programme. The selection panel offered places to 226 user projects, with an acceptance rate of 73 %. These users spent 351,7 visitor-months at the 6 different infrastructures, and used about 2 395 913 AUs.
During the third year, 303 applications were made to the TA programme. The selection panel offered places to 198 user projects, with an acceptance rate of 66 %. These users spent 284,4 visitor-months at the six different infrastructures, and used about 4 170 136,22 AUs.
During the fourth year, 189 applications were made to the TA programme. The selection panel offered places to 132 user projects, with an acceptance rate of 70 %, in line with the average over the whole project.
The scientific users selection panel definitely succeeded in representing a possible stable and comprehensive (transnational, multidisciplinary, intra-centre) model for access provision assessment and selection for any Integrated Infrastructure Initiative, due to its composition - including renowned scientists from many different fields of the European scientific research community - and its selection methodology. These users spent 330,8 visitor-months at the six different infrastructures, significantly more than the planned value of 270,5 VM.
The average visit length in 2007 was 6.7 weeks (compared to 7.0 weeks in 2006) and somewhat longer than the estimate of 6 weeks planned at the time of providing the budget. The access provision granted in 2007 to the visitors was significantly exceeding the contractual targets. In facts, the visitors used about 5 502 332 AUs more than 1 000 000 respect to 2006 and 4 times more than the amount planned originally by the project. This fact is in line with the expectations, in facts different TA centres upgraded their facilities.
In total, within the 4 years of project lifetime, a total of 1 067 applications were received and evaluated and 738 visits, after some cancellations, have been performed in the 6 supercomputing centres.
With a target on the 4 years of 3 997 892 AUs to be delivered by the consortium, a total of 13 380 106 AUs were actually provided to the users, more than 3 times than the minimum contractual due quantity.
The objectives of the JRA1, performance analysis tools, were to deliver scalable uniform tools across all sites for the performance analysis of applications and to develop joint analysis methodologies and systems performance monitoring. During the first half of the project, there was important work on development to port and increase the scalability of the tools. This work was completed at the end of 2005, and during the last two years of HPC-EUROPA, the main focus of the JRA1 activity has been related to use, validation and extension of the defined methodology.
The main achievement of JRA2, single point of access tools was the design and the development of a portal that provides uniform access to heterogeneous computational infrastructures. To this end, JRA2 built a framework based on standards (such as JSDL) allowing test users to run their applications using diverse middleware such as UNICORE and gLite through the uniform interface. The concept of the framework was proved to be valuable and its development was continued within OMII-EUROPE and BEINGRID projects. The portal also includes useful tools for file management, resource information, and single sign-on authentication.
The aim of the two JRAs was to improve the overall performance of HPC in Europe. The first of these had to do with tools for measuring the performance of programs, especially those that have recently been 'ported' to a supercomputer through the project's transnational access (TA) activities. The second was focussed on creating new and simplified ways to access the resources of grid computing systems.
Networking activities in HPC-EUROPA covered the management and evaluation of the project itself, a videoconferencing system, and new systems to ensure that data from finished projects can be stored, retrieved and shared easily.
More in detail, the very ultimate goal of the project was to allow European researchers to maintain a competitive edge on the world stage, delivering the following services:
- access to HPC platforms, unique at national level, for all European researchers;
- provision of an advanced computational environment;
- technical support and specialist tutoring in the domain of HPC.
The core project was organised around the TA activities, and more specifically around the deployment and operation of a 'virtual global infrastructure'.
The JRA and NA aim was to contribute towards the previous mentioned main deployment, fulfilling the following main roles:
- the development and deployment of new high-level services for TA users across the participants;
- the improvement and integration of existing services available at each single participant site and their integration and extension;
- the understanding of users' requirements;
- the promotion of best practices amongst the user community.
Tangible deliverables and results of HPC-EUROPA foreseen for the project were:
- TA service provision;
- Adaptation and integration of a selected toolset into the AccessGrid technology to establish a collaboration and consultation infrastructure;
- realisation of an efficient and portable data management and information retrieval infrastructure for real-life computational science applications in global networks;
- deployment of powerful performance analysis tools that are uniform across HPC platforms, extend the scalability and usability of these tools and development joint methodologies for performance analysis;
- provision of uniform, flexible and intuitive user access to HPC resources from anywhere, via single point of access portal, as well as administration tools for maintaining a grid environment of the TA centres.
HPC-EUROPA was organised around its TA, provided by six sites across Europe: BSC in Spain, CINECA in Italy, EPCC in the United Kingdom, HLRS in Germany, IDRIS in France and SARA in the Netherlands. In this model, the four networking activities and the two joint research activities were designed to facilitate and improve the TA. The NAs mainly aimed to support TA service provision directly, for example by the integration of the six TA centres through the specific development of access grid technology. The JRAs constituted an research and development (R&D) effort to develop valuable new tools to exploit the capability of the TA service provision, in example by means of new performance evaluation tools such as Paraver, especially deployed according to the technical specifications defined by the HPC-EUROPA partners.
The first important achievement reached by the consortium was the set up of an effective, impartial and renowned selection panel, made up by 14 eminent European professors from different scientific fields.
Besides the AUs requested for each project, the six TA partners provided to their visitors a lot of 'side utilities' that helped to create a real HPC focused network and global environment. Samples of those benefits were additional activities tailored specifically to the needs of visitors:
- named contact providing individual technical consultancy;
- access to local training courses;
- collaborative virtual 'surgeries' via AccessGrid;
- centralised VLE providing interactive online training material, technical reports, discussion forum etc.
During the first year, a total of 270 applications were made to the TA programme. The selection panel offered places to 188 user projects, 96 of which started - and in most cases were completed - in 2004, with an acceptance rate of 70 %. These users spent some 155 visitor-months at the 6 different infrastructures, and used about 750 000 Allocation Units (AUs). The AU is defined as 'the computational power delivered by a computer executing for one hour at the sustained rate of one GFlops/s'.
During the second year, 305 applications were made to the TA programme. The selection panel offered places to 226 user projects, with an acceptance rate of 73 %. These users spent 351,7 visitor-months at the 6 different infrastructures, and used about 2 395 913 AUs.
During the third year, 303 applications were made to the TA programme. The selection panel offered places to 198 user projects, with an acceptance rate of 66 %. These users spent 284,4 visitor-months at the six different infrastructures, and used about 4 170 136,22 AUs.
During the fourth year, 189 applications were made to the TA programme. The selection panel offered places to 132 user projects, with an acceptance rate of 70 %, in line with the average over the whole project.
The scientific users selection panel definitely succeeded in representing a possible stable and comprehensive (transnational, multidisciplinary, intra-centre) model for access provision assessment and selection for any Integrated Infrastructure Initiative, due to its composition - including renowned scientists from many different fields of the European scientific research community - and its selection methodology. These users spent 330,8 visitor-months at the six different infrastructures, significantly more than the planned value of 270,5 VM.
The average visit length in 2007 was 6.7 weeks (compared to 7.0 weeks in 2006) and somewhat longer than the estimate of 6 weeks planned at the time of providing the budget. The access provision granted in 2007 to the visitors was significantly exceeding the contractual targets. In facts, the visitors used about 5 502 332 AUs more than 1 000 000 respect to 2006 and 4 times more than the amount planned originally by the project. This fact is in line with the expectations, in facts different TA centres upgraded their facilities.
In total, within the 4 years of project lifetime, a total of 1 067 applications were received and evaluated and 738 visits, after some cancellations, have been performed in the 6 supercomputing centres.
With a target on the 4 years of 3 997 892 AUs to be delivered by the consortium, a total of 13 380 106 AUs were actually provided to the users, more than 3 times than the minimum contractual due quantity.
The objectives of the JRA1, performance analysis tools, were to deliver scalable uniform tools across all sites for the performance analysis of applications and to develop joint analysis methodologies and systems performance monitoring. During the first half of the project, there was important work on development to port and increase the scalability of the tools. This work was completed at the end of 2005, and during the last two years of HPC-EUROPA, the main focus of the JRA1 activity has been related to use, validation and extension of the defined methodology.
The main achievement of JRA2, single point of access tools was the design and the development of a portal that provides uniform access to heterogeneous computational infrastructures. To this end, JRA2 built a framework based on standards (such as JSDL) allowing test users to run their applications using diverse middleware such as UNICORE and gLite through the uniform interface. The concept of the framework was proved to be valuable and its development was continued within OMII-EUROPE and BEINGRID projects. The portal also includes useful tools for file management, resource information, and single sign-on authentication.