Periodic Reporting for period 2 - DEEP-HybridDataCloud (Designing and Enabling E-infrastructures for intensive Processing in a Hybrid DataCloud)
Période du rapport: 2019-02-01 au 2020-04-30
The DEEP platform (https://deep-hybrid-datacloud.eu/the-platform/) is composed of several specialized high level services that can be used individually or as a whole. The ""DEEP training facility"" (https://train.deep-hybrid-datacloud.eu/) allows data scientists and machine learning practitioners to develop and train their applications exploiting existing specialized hardware in EU e-Infrastructures. This service allows to either create their own model using our development environment or to reuse an existing one. To this aim, the ""DEEP marketplace"" (https://marketplace.deep-hybrid-datacloud.eu/) allows to publish, share and reuse machine learning, deep learning and artificial intelligence models easily, creating a knowledge hub for reuse and collaboration, bringing knowledge closer to society. Lastly, the ""DEEP as a Service"" (https://deepaas.deep-hybrid-datacloud.eu/) provides an automated way to transparently publish an existing model as a service, with horizontal scalability out of the box. The project has followed a Service Oriented Architecture (SOA), with an emphasis on the quality assurance of the developed tools and applications, than undergo a comprehensive testing and validation process before being released or deployed. Moreover, all the developed applications can be transparently executed on a disparate set of resources, ranging from personal laptops, workstations, EU e-Infrastructures or HPC facilities just to cite some of them.
These services are integrated with storage and identity services in the European Open Science Cloud, and are offered through the EOSC portal, providing therefore a framework for machine learning, deep learning and artificial intelligence for the EOSC, exploiting first-class EU e-Infrastructure resources."
• The DEEP training facility (https://train.deep-hybrid-datacloud.eu/) as the service providing access to resources leveraging accelerators for intensive computing, exploiting the PaaS cloud layer or HPC resources.
• The DEEP as a Service (https://deepaas.deep-hybrid-datacloud.eu/) , which provides an easy and automated way to deploy existing applications in the catalog as services.
• The DEEP Open catalog (https://marketplace.deep-hybrid-datacloud.eu/) a marketplace where users can browse, retrieve, share and store their applications.
• The data and storage services, where user data and results can be stored that are external to the project, but integrated with us.
These services are being deployed on resource providers that are part of production pan-European e-Infrastructures. In this regard we have ensured that the deployed services are in line with the operational requirements that are enforced at the e-Infrastructure level. Besides, we have linked with industry stakeholders through the EOSC Digital Innovation Hub (DIH).
Extensive work has been performed in all the software quality and maintenance related activities, involving both software developers and users since the initial phases. This co-design methodology was carried out in order to obtain a high degree of acceptance on the software related procedures, avoiding excessive burden on software development teams. Moreover, these activities are being done in coordination with other projects, as reflected in the publication of ""A set of common software quality assurance baseline criteria for research projects"" document (http://hdl.handle.net/10261/160086). Besides, we have ensure that the developed services follow the existing recommendations to be integrated into the EOSC realm."
We bring knowledge closer to society by easing the execution of the DEEP marketplace modules. We provide easy methods (i.e. easy to users with no technical knowledge) and instructions to quickly test and use the model’s functionality and exploit its gained knowledge. Moreover, we allow scientists to deploy their applications and services, being possible therefore to build services that exploit the model’s functionality. We are fostering collaboration since all the developed modules, applications and models are directly available for download and reuse. Reproducibility, although not being directly addressed by our project, can be achieved by the publication of a model, application, data and metadata through the portal.