Nowadays, both large enterprises and SMEs are becoming increasingly data-driven and data-intensive, relying on data and analytics throughout the whole fabric of their business (strategic planning, sales, marketing, finance, operations) to make fact-based business decisions and to better analyse and understand business conditions. Faced with a multitude of data sources, the traditional approach is to squeeze the data into a data warehouse. This requires extensive Extract-Transform-Load (ETL) processes to filter, aggregate and transform data from the original sources to a target data store. It imposes a predefined format, schema and storage for the target data and, accordingly, a predefined set of rules for data ingestion. This is not flexible for investigating new sources and accommodating changes in existing ones. Moreover, ETL processes often take several hours to complete, introducing long waiting times and overhead between the points when new data becomes available and when data scientists can query and analyse it.
Responding to these needs of an open and dynamic world, data lakes have emerged as an alternative approach. A data lake is a raw data ecosystem, where large amounts of diverse structured, unstructured and semi-structured data in its natural format and in various models can coexist. A data lake retains all data, including data that is kept because it might be of use at some point in the future, as opposed to predefined parts of data at predefined levels of granularity that are known in advance to serve specific purposes. Data is retained in its natural, raw form, following a "schema on read" rather than "schema on write" approach, and it is transformed only when the use for it arises. As opposed to data warehouses that can efficiently serve well-planned and anticipated business needs and operations, data lakes are the go-to place for so-called self-service analytics. Data scientists can directly tap into the data lake to analyse data from new sources, combine data of different types, come up with new business questions, test hypotheses and derive new insights and knowledge, improving flexible, fast and ad hoc decision making.
To this end, the overall goal of the SmartDataLake project is to design, develop and evaluate a novel framework for supporting extreme-scale analytics over Big Data Lakes, thus facilitating the journey from raw data to actionable insights. In particular, it offers a suite of tools for virtualized and adaptive data access; automated and adaptive data storage tiering; smart data discovery, exploration and mining; monitoring and assessing the impact of changes; and empowering the data scientist in the loop through scalable and interactive data visualizations.
