Periodic Reporting for period 2 - CS3MESH4EOSC (Interactive and agile/responsive sharing mesh of storage, data and applications for EOSC)
Reporting period: 2021-07-01 to 2023-06-30
The objective of CS3MESH4EOSC is to create an interoperable federation of data and higher-level services to enable friction-free collaboration between European researchers. This will be achieved by capturing the momentum of recent cloud service provisioning and uptake in National Research & Education Networks (NRENs) and public research sector providers across Europe. In particular, the availability of Enterprise File Sync&Share services have led to an increase in data storage and collaboration opportunities for mid- to long-tail science. At this time, typical service installations are in the tens of thousands of users with several Petabytes of storage held. However, the cloud services and data storage are mainly confined to collaborations within individual NRENs or between a few NRENs using custom-made access protocols. The CS3MESH4EOSC proposes to start from those locally and individually provided services and scale them up to European level, with the promise of reaching critical mass and brand recognition also among European scientists that are not usually reached by specialist eInfrastructure, and certainly not in these numbers. Moreover, CS3MESH4EOSC wants to integrate these local existing sites and services into a seamless mesh infrastructure which is fully interconnected with the EOSC-Hub, as proposed in the European Commission’s Implementation Roadmap for EOSC. We plan to provide this service delivery scale-up by interconnecting the various existing sites; this however is not trivial. The capability to do so has only recently become available at TRL6 level, and this will be the first time these new protocols and architectures are delivered at a scale anywhere near this magnitude.
Once delivered, the new capability will form an interoperable, pan-European mesh of data and higher-level services, which will allow friction-free collaboration between all European researchers, without requiring these researches to relocate their data. This means that user data remains at the primary provider's site (be that the researcher’s institute, NREN or a science domain provider), thus significantly easing policy, eligibility, data custodianship and consent issues.
Once delivered, the new capability will form an interoperable, pan-European mesh of data and higher-level services, which will allow friction-free collaboration between all European researchers, without requiring these researches to relocate their data. This means that user data remains at the primary provider's site (be that the researcher’s institute, NREN or a science domain provider), thus significantly easing policy, eligibility, data custodianship and consent issues.
The objective of the CS3MESH4EOSC is to create an interoperable federation of storage and higher-level services (“ScienceMesh”) to enable frictionless collaboration between European researchers.
The overarching principles of CS3MESH4EOSC are:
1. Re-use and build upon existing technical standards and Open-Source solutions based on EFSS (Enterprise File Sync and Share) platforms as well as other systems and services developed for Research;
2. Tap into existing user-base and collaborations, leveraging already established, operational services in the CS3 community (Cloud Services for Synchronisation and Sharing);
3. Leverage the collaboration with Industry and identify further mutually beneficial synergies between Industry and Research;
4. Extend and involve into activities, services and infrastructures in a larger scope for the European Open Science Cloud.
Project followed all of the principles and reached its objectives. ScienceMesh has been created as a pre-production infrastructure. Several applications and integration components have been integrated and deployed in production services or integrated into commercially supported EFSS platforms. The project has set up a solid foundation for its future exploitation. The executive summary of work performed by each Work Package follows below.
WP1 established a solid management process which was applied throughout the duration of the Project. The use of online tools for collaboration (video-conferencing, chat, wiki) was of key importance, especially at the beginning of the Project, with the outbreak and lasting consequences of the COVID-19 pandemic, and thus remained even when in-person events became again possible.
WP2 designed and established the foundation and key fabric components as well as architecture and operational principles of the federated infrastructure. Federation service was delivered in pre-production and it interconnects the nodes of the consortium partners (CERN, CESNET, SURF, SWITCH, WWU, PSNC) as well as an external node (SUNET) while onboarding of other external nodes is planned (e.g. HIFIS, ETHZ). WP2 also proposed formal components of the infrastructure, describing its governance structure flexible for future developments. Operational procedures, technical documentation and governance policies were published in Open Access repositories. The work package also engaged with the EOSC community to follow up on the evolving EOSC architecture, with the ambition to integrate the ScienceMesh infrastructure as part of EOSC. As a result the ScienceMesh service has been registered in the EOSC Marketplace. Additionally, interoperability with ScienceMesh was specified as one of requirements for the procured services for “Managed Collaborative Data Platform for EOSC Exchange” (CNECT/LUX/2022/CD/0023 LOT3)
WP3 set up a solid software development and deployment process, which allowed for efficient contributions both by the project members as well as by the larger community. Further work on Open Standards and protocols (OCM, CS3APIs) has been performed in collaboration with industry resulting in the formal release of two OCM versions. The Interoperability Platform (IOP) was extended following the requirements defined by WP2 and WP4. All software artefacts have been packaged in a portable way, maximising flexibility and rapid deployment. API Integration of the IOP with Nextcloud and Owncloud has been achieved. The EFSS vendors have been engaged in the integration of the ScienceMesh App within their released products. The ScienceMesh App presents the ScienceMesh functionality to end-users of these platforms. Additional service integration components have been developed for storage (for CephFS and EOS), backup (using Restic) and Kubernetes deployments (ScienceBox).
WP4 ensured technical progress in all application areas (Data Science Environments, Open Data Systems, Collaborative Document Editing, On-demand Data Transfers) as well as developed new application integrations and identified a new functional area (Event Management Systems).
WP5 established a comprehensive communication and outreach strategy, exploiting all modern communication channels (website, podcasts, social media, press articles, blog posts). The project fully leverages the CS3 community: dedicated ScienceMesh workshops and sessions were set up as at the annual CS3 conferences (2021,2022, 2023). A dedicated function of e-Research partnerships coordination was set up as part of the project steering committee. This resulted in establishing contacts and collaboration with several external projects: OSF, NGI-Atlantic, EGI-ACE, HIFIS, ESCAPE, ScieboRDS among others. The Project engaged with several ESFRI members (European Strategy Forum on Research Infrastructures) which are closely involved in the EOSC landscape: PANOSC, ESCAPE, SSHOC, ENVRI-FAIR and EOSC Life. A study and report on possible commercial exploitation of the ScienceMesh was delivered. Commercial exploitation plans have been studied and supported by hands-on experimentation by the two project members representing the industry (Cubbit and Ailleron/SoftwareMind).
The overarching principles of CS3MESH4EOSC are:
1. Re-use and build upon existing technical standards and Open-Source solutions based on EFSS (Enterprise File Sync and Share) platforms as well as other systems and services developed for Research;
2. Tap into existing user-base and collaborations, leveraging already established, operational services in the CS3 community (Cloud Services for Synchronisation and Sharing);
3. Leverage the collaboration with Industry and identify further mutually beneficial synergies between Industry and Research;
4. Extend and involve into activities, services and infrastructures in a larger scope for the European Open Science Cloud.
Project followed all of the principles and reached its objectives. ScienceMesh has been created as a pre-production infrastructure. Several applications and integration components have been integrated and deployed in production services or integrated into commercially supported EFSS platforms. The project has set up a solid foundation for its future exploitation. The executive summary of work performed by each Work Package follows below.
WP1 established a solid management process which was applied throughout the duration of the Project. The use of online tools for collaboration (video-conferencing, chat, wiki) was of key importance, especially at the beginning of the Project, with the outbreak and lasting consequences of the COVID-19 pandemic, and thus remained even when in-person events became again possible.
WP2 designed and established the foundation and key fabric components as well as architecture and operational principles of the federated infrastructure. Federation service was delivered in pre-production and it interconnects the nodes of the consortium partners (CERN, CESNET, SURF, SWITCH, WWU, PSNC) as well as an external node (SUNET) while onboarding of other external nodes is planned (e.g. HIFIS, ETHZ). WP2 also proposed formal components of the infrastructure, describing its governance structure flexible for future developments. Operational procedures, technical documentation and governance policies were published in Open Access repositories. The work package also engaged with the EOSC community to follow up on the evolving EOSC architecture, with the ambition to integrate the ScienceMesh infrastructure as part of EOSC. As a result the ScienceMesh service has been registered in the EOSC Marketplace. Additionally, interoperability with ScienceMesh was specified as one of requirements for the procured services for “Managed Collaborative Data Platform for EOSC Exchange” (CNECT/LUX/2022/CD/0023 LOT3)
WP3 set up a solid software development and deployment process, which allowed for efficient contributions both by the project members as well as by the larger community. Further work on Open Standards and protocols (OCM, CS3APIs) has been performed in collaboration with industry resulting in the formal release of two OCM versions. The Interoperability Platform (IOP) was extended following the requirements defined by WP2 and WP4. All software artefacts have been packaged in a portable way, maximising flexibility and rapid deployment. API Integration of the IOP with Nextcloud and Owncloud has been achieved. The EFSS vendors have been engaged in the integration of the ScienceMesh App within their released products. The ScienceMesh App presents the ScienceMesh functionality to end-users of these platforms. Additional service integration components have been developed for storage (for CephFS and EOS), backup (using Restic) and Kubernetes deployments (ScienceBox).
WP4 ensured technical progress in all application areas (Data Science Environments, Open Data Systems, Collaborative Document Editing, On-demand Data Transfers) as well as developed new application integrations and identified a new functional area (Event Management Systems).
WP5 established a comprehensive communication and outreach strategy, exploiting all modern communication channels (website, podcasts, social media, press articles, blog posts). The project fully leverages the CS3 community: dedicated ScienceMesh workshops and sessions were set up as at the annual CS3 conferences (2021,2022, 2023). A dedicated function of e-Research partnerships coordination was set up as part of the project steering committee. This resulted in establishing contacts and collaboration with several external projects: OSF, NGI-Atlantic, EGI-ACE, HIFIS, ESCAPE, ScieboRDS among others. The Project engaged with several ESFRI members (European Strategy Forum on Research Infrastructures) which are closely involved in the EOSC landscape: PANOSC, ESCAPE, SSHOC, ENVRI-FAIR and EOSC Life. A study and report on possible commercial exploitation of the ScienceMesh was delivered. Commercial exploitation plans have been studied and supported by hands-on experimentation by the two project members representing the industry (Cubbit and Ailleron/SoftwareMind).