"STREAMLINE will address the competitive advantage needs of European online media businesses (EOMB) by delivering fast reactive analytics suitable in solving a wide array of problems, including addressing customer
retention, personalised recommendation, and more broadly targeted services. STREAMLINE will develop cross sectorial analytics drawing on multi-source data originating from online media consumption, online games,
telecommunications services, and multilingual web content.
STREAMLINE partners face big and fast data challenges. They serve over 100 million users, offer services that produce billions of events, yielding over 10 TB of data daily, and possess over a PB of data at rest. Their business use cases are representative of EOMB, which cannot be handled efficiently & effectively by state-of-the-art technologies, as a consequence of system and human latencies.
System latency issues arise due to the lack of appropriate (data) stream-oriented analytics tools and more importantly the added complexity, cost, and burden associated with jointly supporting analytics for both “data at rest” and “data in motion.” The attached picture ""Current Lambda architecture"" is an illustration of the current state-of-the art.
Human latency results from the heterogeneity of existing tools and the low level programming languages required for development using an inordinate number of boilerplate codes that are system specific (e.g. Hadoop, SolR, Esper, Storm, and databases) and a plethora of scripts required to glue systems together. This today requires very specific expertise in the areas mentioned.
All these factors contribute to making it difficult for European businesses to find enough people with the right competence to address the need for solutions. To unlock the the potential of Big Data in Europe we need to develop better methods and tools that are easy to use even for non-experts.
STREAMLINE has formulated the following objectives, means to achievement and measurable outcomes towards the goals of reducing complexity, enabling faster results, and supporting both ""data at rest"" and ""data in motion""
in a single system in three innovative use cases of high business impact for STREAMLINE partners.
Objective I: To research, design, & develop a massively scalable, robust, and efficient processing platform for data at rest and data in motion in a single system.
Objective II: To develop a high accuracy, massively scalable data stream-oriented machine learning library based on new algorithms and approximate data structures.
Objective III: To provide a unified interactive programming environment that is user-friendly, i.e. easy to deploy in the cloud and validate its success as measured by well-defined KPIs.
Objective IV: To implement a real-time contextualisation engine, enabling analytical and predictive models to take real world context into account.
Objective V: To improve industrial partner business performance by their introducing new services, improving performance, reducing costs, and growing their business, as measured by well-defined KPIs, in a refactorable fashion that is easily reusable by other companies and industry domains.
STREAMLINE will build on the European Open Source project Apache Flink and propose enhancements to achieve the states objectives and reach maximal impact. The envisioned architecture is pictured in the attached ""The architecture for Streamline""
"
