HPC and Cloud-enhanced Testbed for Extracting Value from Diverse Data at Large Scale

Información del proyecto

EVOLVE

Identificador del acuerdo de subvención: 825061

DOI

10.3030/825061

Proyecto cerrado

Fecha de la firma de la CE 5 Diciembre 2018

Fecha de inicio 1 Diciembre 2018

Fecha de finalización 30 Noviembre 2021

Financiado con arreglo a

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Coste total

€ 14 638 608,58

Aportación de la UE

€ 11 045 919,63

Coordinado por

DATADIRECT NETWORKS FRANCE
France

Periodic Reporting for period 2 - EVOLVE (HPC and Cloud-enhanced Testbed for Extracting Value from Diverse Data at Large Scale)

Período documentado: 2020-06-01 hasta 2021-11-30

Resumen del contexto y de los objetivos generales del proyecto

Evolve is a European project bringing together HPC, AI and Big Data experts. HPC, AI and BigData share a common point: the need for performance at scale.
However, they differentiate by many other aspects: nature of the data processed, workflow, software stack. This challenges the idea that a generic purpose processor
or storage architecture will meet their requirements.Therefore Evolve is building an heterogeneous supercomputing platform jointly with the software components to
leverage transparently this heterogeneity at the advantage of the applications.Evolve platform supports traditional CPU, with GPU and state-of-the-art FPGA.
Storage-wise, the platform is accelerated by 8 DDN's IME units and a Lustre parallel file system. More importantly, Evolve is a customer-focused project, where partners
provide 9 pilots applications to validate the approach. This crucial applications effort allows Evolve to optimize and adjust in an agile way during project’s lifetime.
With carefully designed middleware, API and connectors, Evolve is already able to accelerate Spark applications in a fully transparent manner, to containerize parallel
file systems in Docker and allow Zeppelin notebooks to access the supercomputer. Already one of the Pilot application with complex workflow including MPI parallel
code and AI can be orchestrated on the platform taking advantage of the diversity of the computing resources. Much is still on-going: resource management and
monitoring, data protecting, articulation between Kubernetes and Slurm or even deeper storage hierarchies.
Several external partners have already been engaged to test and validate the Evolve system and help us to mature the solution. The future will be diverse!

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

1.4. Key achievements
The main achievement at the time of this review can be briefly summarized as:
• The platform is full featured, all the hardware acceleration is available (FGPA, GPU, Local storage and burst buffer). The system is fully operational and used on a daily basis by consortium member and external stakeholders.
• The software run-time support all these HW accelerations as micro-services and provide transparent access to these accelerator to end-users workflows.
• The front-end is completed and optimized, complex workflows are launched through notebook. This stack is mature enough to have be made available on the marketplace of a major cloud provider.
• Resources management is data-aware and take into account workflows characteristics to map its components to the right acceleration technology. MPI components can be deployed as HPC micro-services under Kubernetes.
• All Pilots applications are running on the platform, some of them with impressive speed-up (up to 120 faster)
• Several additional application (Proof-Of-Concept) have been identified and one is already running on the platform.

All the Milestones have been reached, some KPI are considerably beyond the initial goats. The impact both in terms of Innovations and scientific output is also over the expectations.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

Among the Pilot Application and Proof of Concept using EVOLVE technologies we have singled out 6 success stories with can be considered as of high societal value:

1 - Using EVOLVE technologies, CYBEL has released an innovation to detect the agricultural crop in place at a given geographical point, this agricultural cooperatives to better organize and prepare harvests and to optimize stock management.
2 - TAS EarthObserver has been released to the market using Copernicus data. One application of this innovation is to assess the damage of a frost episode to vineyards.
3- SPH has released Acritas, an innovation which considerably improve Maritime Surveillance, notably at the sea borders of Europe.
4 - MEMEX and TIEMME have released Ceslo an innovation allowing public transport operators to analyses both the traffic congestion on the public transport network and the service. Travel time and related delays can be estimated in real time.
5- -Driver Distraction Detection (VIF), this PoCs allows to assess and detect lack of concentration of the vehicle driver.
6 - CYBEL has delivered a PoC for Automatic cadastral map generation for tropical countries (Cybeletech).

Therefore, EVOLVE has demonstrated that HPC technology has an impact on non-scientific domains to extract value from data for societal/business.

Logo of the Evolve project