Cluster Management beyond High Performance Computing

Project Information

Bright Beyond HPC

Grant agreement ID: 711315

DOI

10.3030/711315

Project closed

EC signature date 28 January 2016

Start date 1 February 2016

End date 31 October 2017

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 2 161 223,75

EU contribution

€ 1 512 856,63

Coordinated by

BRIGHT COMPUTING BV
Netherlands

Periodic Reporting for period 2 - Bright Beyond HPC (Cluster Managementbeyond High Performance Computing)

Reporting period: 2016-11-01 to 2017-10-31

Summary of the context and overall objectives of the project

The overall objective of the “Bright Beyond HPC” project was to solve the Cluster Management problem in the High Performance Computing (HPC), Big Data and Private Cloud computing markets by developing and bringing to market the readiness of Bright OpenStack & Bright Cluster Manager for Big Data.

The Bright beyond HPC project concerns the further development and scaling up of Bright Cluster Manager beyond High Performance Computing (HPC). Bright Computing developed in the project advanced functionalities in the areas of Big Data and Cloud computing. These enhancements enable Bright Computing to expand its market position significantly and the technology enables the market to benefit, in an ease-of-use manner, from Cluster Management.

The main development cycle within the project followed new releases of Bright Cluster Manager. Each release included new features and functionality. Each release went out to customers and could immediately be used. As part of the Bright beyond HPC project new functionality was being added that is especially useful for users in markets in which Bright had little presence: Big Data and Cloud computing. Hence major new Bright Cluster Manager functionality was developed for these two areas, with a focus on Apache Hadoop for the Big Data area, and a focus on OpenStack for the Cloud computing area. Some of the new features were depending on external software developments. For instance, new versions of Apache Hadoop or Apache Hadoop related tools such as Cassandra, Spark or Flink are released by Apache. Although Apache has a roadmap for most products, it was never exactly followed because it is an open source project where major developments are done by volunteer contributors and unexpected problems can always emerge. Bright not only had to follow and assess tools from Apache and OpenStack, but from many other tool providers too. Also, preference was given to tools with the biggest (potential) customer demand at a specific moment in time. So there is some flexibility in the Bright Cluster Manager release road map too. Nevertheless, the overall plan was to have three major releases of the Bright Cluster Manager during the lifetime of the project.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The main body of the work during the first period went into the Deliverable D4.1. Next gen Bright Cluster Manager HPC 7.3. that is publically available.The new software produced and described in this deliverable is available for download for customers from the Bright Computing portal, although the may prefer one of the newer releases. The new features developed for Bright Cluster Manager HPC 7.3 are briefly described in the deliverable document. They cover different technical areas.

In the second and final period we worked on several deliverables all focused on extending Bright Cluster Manager and bringing it to new markets. For the Big Data market we worked on Bright Cluster Manager 8.0 and 8.1 which has been described in Deliverables 5.1 and 5.2. Furthermore we submitted deliverable D1.1 Next gen Bright cluster Manager HPC 7.3 – 8.1 – Bright Cluster Manager with complete Hadoop management capabilities. In this deliverable we described Bright Cluster Manager for Big Data - Bright Cluster Manager with complete Hadoop management capabilities - and making it ready for the new Big Data market.

Another main objective of the project was to integrate OpenStack in such a way that OpenStack Clouds can be managed from Bright Cluster Manager. There were several steps needed to achieve this. First, Bright Cluster Manager had to be refactored to allow integration of Clouds. Then we added some basic general Cloud functionality including some for AWS. Finally, we wanted to integrate full OpenStack support. In the previous deliverable D5.1 we added Azure integration as a major new feature. Microsoft Azure is one of the major Cloud infrastructures available.
In Deliverable D2.1 Next gen Bright cluster Manager HPC 7.3 – 8.1 - Bright Cluster Manager with complete Cloud computing capabilities we described the activities to bring Bright Cluster Manager OpenStack with complete Cloud computing capabilities to the market.
Another focus was on General Cluster management. It introduced a number of new features and support for new architectures beneficial to all users of Bright Cluster Manager.

The dissemination and exploitation activities during the last period and in the future concentrate on: communicating about new features for the HPC market; Communicating about Bright Cluster Manager to the Big Data and in the Cloud computing market; Further develop the partner programme.

Especially the new tiered partner programme was successful in reaching new customers.

For an overview communication activities visit http://www.brightcomputing.com/ .

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The new release of Bright Cluster Manager and Brigh OpenStack are great steps forward. More specifically they resulted in:

Great Technology
Work on Bright Cluster Manager in the second period has lead to such an increase in the technology features included, the jump to a new major version Bright Cluster Manager 8.0 was fully justified. The 2nd release during this period, release 8.1 added significant improvements.
Work on ”Extension to Big Data and OpenStack” has been reported in deliverables D5.1 and D5.2. As a result Bright Cluster manager now has been extended with mature Hadoop management capabilities.
The OpenStack work included cloud bursting facilities for AWS and Microsoft Azure as an important feature allowing to firmly establish Bright computing in the Cloud market.

Great Scaling Up Potential
The new releases of Bright Cluster Manager 8.0 and 8.1 enable a step forward to reaching “Disproportionate growth for Bright Computing”. This has been reinforced by specific work in WP3 on marketing, communication and a new set-up of the partner programme.

Customers , including new customers, respond very positively to the new Bright Cluster Manager releases. They especially appreciate the ease of use of the well tested combination Bright Cluster Manager with OpenStack.

Great Market Potential
The integration work combining HPC, Cloud, and Big Data technology was well received by the market, resulting in customers that used a combination of those three. Furthermore, Bright Computing has now entered the new Big Data and Cloud computing markets.

Great Enabler
More efficient cluster computing: Already over 100 customers have installed Bright Cluster Manager 8.0 or have version 8.1 in use or under evaluation. Especially existing customers moving to 8.0 do so because it brings them a more efficient way of managing their clusters. The large number of evaluation versions show Bright Cluster Manager 8.1 helps to spread cluster computing to new markets.

Thanks to Bright Cluster Manager 8.0 and 8.1 HPC, Big Data and Cloud are working even more together than before to solve societal challenges. Several large societal challenges, such as improvements in the medical area that need advanced bioinformatics, or climate related research, require Big Data captured and collected by many instruments and sensors, stored locally and in the Cloud. Complex analysis often involves Cloud based analysis and HPC modelling. Bright Cluster Manager has provided an integrated environment where these three things work together. The list of Bright customers show there are many research organisations that are tackling these societal challenges.

New Bright View monitoring interface

Bright Cluster Manager 7.3. demonstrated at major events.

The new Hadoop interface in Bright Cluster Manager 7.3.