Periodic Reporting for period 1 - TEADAL (Trustworthy, Energy-Aware federated DAta Lakes along the computing continuum)
Periodo di rendicontazione: 2022-09-01 al 2024-02-29
Data analytics is one of the main cornerstones in many enterprise architectures. The data lake paradigm is increasingly
adopted to assist organizations in making reliable, accurate, and fast decisions. Although the initial approaches to address these issues
saw the data lakes as the evolution of data warehouses to be implemented on-premises, cloud providers are nowadays including in
their offerings platforms able to set up and run them. Nevertheless, the increasing amount of data generated at the edge and the
need to enable data sharing among organizations are posing new challenges in terms of performance, energy efficiency, and
privacy/confidentiality, which can be properly addressed with data lakes which are deployed along the whole computing continuum
as well as building a federation of such data lakes.
The ambition of TEADAL is to provide key cornerstone technologies to create stretched data lakes spanning the cloud-edge
continuum and multi-cloud, providing privacy, confidentiality, and energy-efficient data management. The TEADAL data lake
technologies will enable trusted, verifiable, and energy efficient data flows, both in a stretched data lake and across a trustworthy
mediators federation of them, based on a shared approach for defining, enforcing, and tracking privacy/confidentiality
requirements balanced with the need for energy reduction.
adopted to assist organizations in making reliable, accurate, and fast decisions. Although the initial approaches to address these issues
saw the data lakes as the evolution of data warehouses to be implemented on-premises, cloud providers are nowadays including in
their offerings platforms able to set up and run them. Nevertheless, the increasing amount of data generated at the edge and the
need to enable data sharing among organizations are posing new challenges in terms of performance, energy efficiency, and
privacy/confidentiality, which can be properly addressed with data lakes which are deployed along the whole computing continuum
as well as building a federation of such data lakes.
The ambition of TEADAL is to provide key cornerstone technologies to create stretched data lakes spanning the cloud-edge
continuum and multi-cloud, providing privacy, confidentiality, and energy-efficient data management. The TEADAL data lake
technologies will enable trusted, verifiable, and energy efficient data flows, both in a stretched data lake and across a trustworthy
mediators federation of them, based on a shared approach for defining, enforcing, and tracking privacy/confidentiality
requirements balanced with the need for energy reduction.
Main technical achievements during RP1 are the following:
- Requirement collection from the 5 active pilots was conducted.
- Design of the Teadal Node Architecture.
- First (mvp) version of the Teadal Node was developed and integrated.
- All 5 Pilots have the Teadal Node mvp version deployed/installed.
- Requirement collection from the 5 active pilots was conducted.
- Design of the Teadal Node Architecture.
- First (mvp) version of the Teadal Node was developed and integrated.
- All 5 Pilots have the Teadal Node mvp version deployed/installed.
TEADAL will enable the creation of trusted, verifiable, and energy-efficient data flows, both inside a data lake and across federated data lakes, based on a shared approach for defining, enforcing, and tracking data governance requirements with specific emphasis on privacy/confidentiality. The proposed stretched data lake, i.e. deployed in the continuum, will be based on an innovative control plane able to exploit all the controlled/owned resources across clouds and at the edge to improve data analysis. The resulting capabilities of stretched data lakes also provide the essential basis for creating trustworthy mediatorless federations of data lakes to foster an effective data exchange among organizations while preserving privacy and confidentiality constraints without any imposed, and often not acceptable, third-party coordinator. Finally, applying to the data governance the principles of circular economy, i.e. to reuse data, application, and computation resources belonging to the data lake federation, will enable the creation of platforms for more sustainable data analytics.
The proposed solution aims to lower the effort required for efficient energy and data analysis with privacy- and confidentiality-by-design data management, thus reducing the barriers to sharing and using data among organizations to provide timely and reliable informed decisions. TEADAL will be beneficial for data lake providers, which can exploit additional functions in terms of definition, enforcement, and tracking of privacy- /confidentiality-based policies, as well as business analysts and data scientists which can make full use of data coming from several organizations without implementing complex systems to ensure privacy/confidentiality requirements.
The proposed solution aims to lower the effort required for efficient energy and data analysis with privacy- and confidentiality-by-design data management, thus reducing the barriers to sharing and using data among organizations to provide timely and reliable informed decisions. TEADAL will be beneficial for data lake providers, which can exploit additional functions in terms of definition, enforcement, and tracking of privacy- /confidentiality-based policies, as well as business analysts and data scientists which can make full use of data coming from several organizations without implementing complex systems to ensure privacy/confidentiality requirements.