Periodic Reporting for period 2 - I-BiDaaS (Industrial-Driven Big Data as a Self-Service Solution)
Periodo di rendicontazione: 2019-07-01 al 2020-12-31
The I-BiDaaS project aimed to empower end-users and practitioners to easily utilize and interact with big data technologies. This was achieved by designing, building, and demonstrating a unified solution that i) increases the speed of data analysis which is necessary to cope with the rate of data asset growth, and ii) facilitates cross-domain data-flow, matching the needs of a thriving data-driven EU economy.
The objectives of the I-BiDaaS project that successfully accomplished are:
• Develop, validate, demonstrate, and support, a complete and solid big data solution that can be easily configured and adopted by practitioners.
• Break inter- and intra-sectorial data-silos, create a data market and offer new business opportunities, and support data sharing, exchange, and interoperability.
• Construct a safe environment for methodological big data experimentation, for the development of new products, services, and tools.
• Develop data processing tools and techniques applicable in real-world settings, and demonstrate significant increase of speed of data throughput and access.
• Develop technologies that will increase the efficiency and competitiveness of all EU companies and organisations that need to manage vast and complex amounts of data.
The objectives of the I-BiDaaS project that successfully accomplished are:
• Develop, validate, demonstrate, and support, a complete and solid big data solution that can be easily configured and adopted by practitioners.
• Break inter- and intra-sectorial data-silos, create a data market and offer new business opportunities, and support data sharing, exchange, and interoperability.
• Construct a safe environment for methodological big data experimentation, for the development of new products, services, and tools.
• Develop data processing tools and techniques applicable in real-world settings, and demonstrate significant increase of speed of data throughput and access.
• Develop technologies that will increase the efficiency and competitiveness of all EU companies and organisations that need to manage vast and complex amounts of data.
Summary of activities performed:
WP1: The work carried out successfully materialized the Baseline Phase of the project and led to the identification of industrial challenges, elicitation of user requirements, specification of the architecture, definition of evaluation and validation approach.
WP2: The work carried out led to the definition of the datasets for the industrial use cases; deployment of an end-to-end solution for data on-boarding; fabrication and evaluation of synthetic data for several use cases; IBM’s TDF tool was enhanced to support automatic data rules creation; visualization tools and interfaces created to support all use cases.
WP3: The work carried out led to scientific advances for data analytics algorithms and the batch processing module. Development of a rich pool of algorithms; developments on Hecuba increased the performance of the cases; visualization features of Qbeast enabled new ways of interactively conducting data analysis, now commercially exploitable; TDF is able to generate synthetic data from a preliminary analysis of the real data.
WP4: This work carried out led to the design of the I-BiDaaS streaming component which allows to analyse large number of input streams in a distributed fashion using a complex event processing engine. It allows to call external machine learning algorithms and the results can be visualized. The offloading of compute-intense tasks to GPUs was designed and tested. Several streaming use cases developed demonstrating higher quality and faster solutions over the adopted baselines.
WP5: The work carried out allowed to build and maintain the operational infrastructure environment used in I-BiDaaS, from the MVP on M12 to a fully integrated solution for the realization of the Big-Data-as-as self-service concept (M32) wherein all I-BiDaaS use cases were successfully integrated and validated. The I-BiDaaS open version released along with the I-BiDaaS Platform usage guide.
WP6: The work carried out involved the experimental protocol alignment, the implementation, operation and validation of 8 real-life industrial experiments following the requirements set by the data pilots. Two more experiments for the generic use case were conducted to show the functionalities in expert and self-service modes for the use beyond the industrial use cases. The impact analysis report was included with respect to the expected project level innovation and achievements.
WP7: The work carried out led on establishing a baseline for the market definition and analysis, finalising the design and proposition of a sound business plan for potential commercialization, including financial modelling and possible revenue streams analysis. Long-term sustainability was ensured by uptaking the solution's potential implementation. Dissemination activities provided maximum visibility and public awareness through a powerful presence on the web and social media, scientific publications, participation in events and strong collaborations with other projects and initiatives such as BDVA.
WP8: The goal of WP8 was to set up and maintain the administrative, financial and management infrastructure of the I-BiDaaS project and ensure successful Quality and Innovation Management. The I-BiDaaS Data Management Plan was defined and followed based on the FAIR principles.
WP1: The work carried out successfully materialized the Baseline Phase of the project and led to the identification of industrial challenges, elicitation of user requirements, specification of the architecture, definition of evaluation and validation approach.
WP2: The work carried out led to the definition of the datasets for the industrial use cases; deployment of an end-to-end solution for data on-boarding; fabrication and evaluation of synthetic data for several use cases; IBM’s TDF tool was enhanced to support automatic data rules creation; visualization tools and interfaces created to support all use cases.
WP3: The work carried out led to scientific advances for data analytics algorithms and the batch processing module. Development of a rich pool of algorithms; developments on Hecuba increased the performance of the cases; visualization features of Qbeast enabled new ways of interactively conducting data analysis, now commercially exploitable; TDF is able to generate synthetic data from a preliminary analysis of the real data.
WP4: This work carried out led to the design of the I-BiDaaS streaming component which allows to analyse large number of input streams in a distributed fashion using a complex event processing engine. It allows to call external machine learning algorithms and the results can be visualized. The offloading of compute-intense tasks to GPUs was designed and tested. Several streaming use cases developed demonstrating higher quality and faster solutions over the adopted baselines.
WP5: The work carried out allowed to build and maintain the operational infrastructure environment used in I-BiDaaS, from the MVP on M12 to a fully integrated solution for the realization of the Big-Data-as-as self-service concept (M32) wherein all I-BiDaaS use cases were successfully integrated and validated. The I-BiDaaS open version released along with the I-BiDaaS Platform usage guide.
WP6: The work carried out involved the experimental protocol alignment, the implementation, operation and validation of 8 real-life industrial experiments following the requirements set by the data pilots. Two more experiments for the generic use case were conducted to show the functionalities in expert and self-service modes for the use beyond the industrial use cases. The impact analysis report was included with respect to the expected project level innovation and achievements.
WP7: The work carried out led on establishing a baseline for the market definition and analysis, finalising the design and proposition of a sound business plan for potential commercialization, including financial modelling and possible revenue streams analysis. Long-term sustainability was ensured by uptaking the solution's potential implementation. Dissemination activities provided maximum visibility and public awareness through a powerful presence on the web and social media, scientific publications, participation in events and strong collaborations with other projects and initiatives such as BDVA.
WP8: The goal of WP8 was to set up and maintain the administrative, financial and management infrastructure of the I-BiDaaS project and ensure successful Quality and Innovation Management. The I-BiDaaS Data Management Plan was defined and followed based on the FAIR principles.
Major I-BiDaaS highlights:
• I-BiDaaS developed an end-to-end complete and configurable Big Data solution with three modes of operation: a) Self-service mode: allows personnel with domain knowledge and insights to easily construct Big Data pipelines, selecting a pre-defined data analytics algorithm from a list; b) Expert mode: allows experts to upload their own data analytics code based on the available highly reusable templates; c) Co-Develop mode: corresponds to an end-to-end solution for a given industry project developed by the I-BiDaaS team, wherein a non-expert user interacts with each use case through an easily understandable visual interface.
• The project successfully delivered the MVP (M12), the 1st prototype (M18) and the 2nd prototype (M32) wherein 11 technologies from 8 partners successfully integrated.
• I-BiDaaS platform offers an open-source version available to download and install and a detailed User Guide. This open-source version can be used by other entities including EU companies across multiple domains and can also impact new technological approaches.
• I-BiDaaS validated and demonstrated on 8 industrial use cases (6 batch & 5 stream) from 3 different domains (telecom, banking, manufacturing) plus 2 generic use cases; TR6 and TRL7 reached. I-BiDaaS is shown to reduce infrastructure cost (less hardware resources, maintenance of expensive infrastructure) and personnel cost (automated, efficient and scalable technologies).
• Data silos broken: TID - data from TID’s subsidiary companies and countries’ repositories, CRF - data from 2 different types of plant, located in 2 different regions, CAIXA - data from different sources integrated and provided to several business units and Security Operations Center. Two hackathons organised (CRF, TID), 12 datasets became available to the partners (8 real, 4 realistic synthetic), 9 of which open to the research community (Zenodo).
• Exploitation strategy supports long-term sustainability and potential commercial viability of the I-BiDaaS solution, creating a data market, and offering new business opportunities. A sound business plan developed. Long-term sustainability is enriched with the adoption of I-BiDaaS in industrial operational environments: CAIXA Garage Lab, CRF Training Environment and TID Innovation Call.
• All data providers disseminated their experience and the lessons learned on data sharing and the usefulness of realistic synthetic data in their relevant sectors.
• Qbeast is now a spin-off company originated at BSC and partially developed in I-BiDaaS.
• Six I-BiDaaS platform and technologies-related innovations promoted as excellent innovations by EU Innovation Radar and published at BDVA Innovation Marketplace.
• 3 Journals, 19 Conference/Workshop Papers, 4 Posters, 1 PhD Thesis, 2 Book Chapters; organisation of 9 special events; collaboration with 5 H2020 EU projects.
• I-BiDaaS developed an end-to-end complete and configurable Big Data solution with three modes of operation: a) Self-service mode: allows personnel with domain knowledge and insights to easily construct Big Data pipelines, selecting a pre-defined data analytics algorithm from a list; b) Expert mode: allows experts to upload their own data analytics code based on the available highly reusable templates; c) Co-Develop mode: corresponds to an end-to-end solution for a given industry project developed by the I-BiDaaS team, wherein a non-expert user interacts with each use case through an easily understandable visual interface.
• The project successfully delivered the MVP (M12), the 1st prototype (M18) and the 2nd prototype (M32) wherein 11 technologies from 8 partners successfully integrated.
• I-BiDaaS platform offers an open-source version available to download and install and a detailed User Guide. This open-source version can be used by other entities including EU companies across multiple domains and can also impact new technological approaches.
• I-BiDaaS validated and demonstrated on 8 industrial use cases (6 batch & 5 stream) from 3 different domains (telecom, banking, manufacturing) plus 2 generic use cases; TR6 and TRL7 reached. I-BiDaaS is shown to reduce infrastructure cost (less hardware resources, maintenance of expensive infrastructure) and personnel cost (automated, efficient and scalable technologies).
• Data silos broken: TID - data from TID’s subsidiary companies and countries’ repositories, CRF - data from 2 different types of plant, located in 2 different regions, CAIXA - data from different sources integrated and provided to several business units and Security Operations Center. Two hackathons organised (CRF, TID), 12 datasets became available to the partners (8 real, 4 realistic synthetic), 9 of which open to the research community (Zenodo).
• Exploitation strategy supports long-term sustainability and potential commercial viability of the I-BiDaaS solution, creating a data market, and offering new business opportunities. A sound business plan developed. Long-term sustainability is enriched with the adoption of I-BiDaaS in industrial operational environments: CAIXA Garage Lab, CRF Training Environment and TID Innovation Call.
• All data providers disseminated their experience and the lessons learned on data sharing and the usefulness of realistic synthetic data in their relevant sectors.
• Qbeast is now a spin-off company originated at BSC and partially developed in I-BiDaaS.
• Six I-BiDaaS platform and technologies-related innovations promoted as excellent innovations by EU Innovation Radar and published at BDVA Innovation Marketplace.
• 3 Journals, 19 Conference/Workshop Papers, 4 Posters, 1 PhD Thesis, 2 Book Chapters; organisation of 9 special events; collaboration with 5 H2020 EU projects.