HPC and Cloud-enhanced Testbed for Extracting Value from Diverse Data at Large Scale

Project Information

EVOLVE

Grant agreement ID: 825061

DOI 10.3030/825061

Project closed

EC signature date 5 December 2018

Start date 1 December 2018

End date 30 November 2021

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost € 14 638 608,58

EU contribution € 11 045 919,63

Coordinated by

DATADIRECT NETWORKS FRANCE
France

Periodic Reporting for period 2 - EVOLVE (HPC and Cloud-enhanced Testbed for Extracting Value from Diverse Data at Large Scale)

Reporting period: 2020-06-01 to 2021-11-30

Summary of the context and overall objectives of the project

Evolve is a European project bringing together HPC, AI and Big Data experts. HPC, AI and BigData share a common point: the need for performance at scale.
However, they differentiate by many other aspects: nature of the data processed, workflow, software stack. This challenges the idea that a generic purpose processor
or storage architecture will meet their requirements.Therefore Evolve is building an heterogeneous supercomputing platform jointly with the software components to
leverage transparently this heterogeneity at the advantage of the applications.Evolve platform supports traditional CPU, with GPU and state-of-the-art FPGA.
Storage-wise, the platform is accelerated by 8 DDN's IME units and a Lustre parallel file system. More importantly, Evolve is a customer-focused project, where partners
provide 9 pilots applications to validate the approach. This crucial applications effort allows Evolve to optimize and adjust in an agile way during project’s lifetime.
With carefully designed middleware, API and connectors, Evolve is already able to accelerate Spark applications in a fully transparent manner, to containerize parallel
file systems in Docker and allow Zeppelin notebooks to access the supercomputer. Already one of the Pilot application with complex workflow including MPI parallel
code and AI can be orchestrated on the platform taking advantage of the diversity of the computing resources. Much is still on-going: resource management and
monitoring, data protecting, articulation between Kubernetes and Slurm or even deeper storage hierarchies.
Several external partners have already been engaged to test and validate the Evolve system and help us to mature the solution. The future will be diverse!

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

1.4. Key achievements
The main achievement at the time of this review can be briefly summarized as:
• The platform is full featured, all the hardware acceleration is available (FGPA, GPU, Local storage and burst buffer). The system is fully operational and used on a daily basis by consortium member and external stakeholders.
• The software run-time support all these HW accelerations as micro-services and provide transparent access to these accelerator to end-users workflows.
• The front-end is completed and optimized, complex workflows are launched through notebook. This stack is mature enough to have be made available on the marketplace of a major cloud provider.
• Resources management is data-aware and take into account workflows characteristics to map its components to the right acceleration technology. MPI components can be deployed as HPC micro-services under Kubernetes.
• All Pilots applications are running on the platform, some of them with impressive speed-up (up to 120 faster)
• Several additional application (Proof-Of-Concept) have been identified and one is already running on the platform.

All the Milestones have been reached, some KPI are considerably beyond the initial goats. The impact both in terms of Innovations and scientific output is also over the expectations.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Among the Pilot Application and Proof of Concept using EVOLVE technologies we have singled out 6 success stories with can be considered as of high societal value:

1 - Using EVOLVE technologies, CYBEL has released an innovation to detect the agricultural crop in place at a given geographical point, this agricultural cooperatives to better organize and prepare harvests and to optimize stock management.
2 - TAS EarthObserver has been released to the market using Copernicus data. One application of this innovation is to assess the damage of a frost episode to vineyards.
3- SPH has released Acritas, an innovation which considerably improve Maritime Surveillance, notably at the sea borders of Europe.
4 - MEMEX and TIEMME have released Ceslo an innovation allowing public transport operators to analyses both the traffic congestion on the public transport network and the service. Travel time and related delays can be estimated in real time.
5- -Driver Distraction Detection (VIF), this PoCs allows to assess and detect lack of concentration of the vehicle driver.
6 - CYBEL has delivered a PoC for Automatic cadastral map generation for tropical countries (Cybeletech).

Therefore, EVOLVE has demonstrated that HPC technology has an impact on non-scientific domains to extract value from data for societal/business.

Logo of the Evolve project