Periodic Reporting for period 2 - EW-Shopp (EW-Shopp - Supporting Event and Weather-based Data Analytics and Marketing along the Shopper Journey)
Reporting period: 2018-07-01 to 2019-12-31
Many companies operating in the eCommerce, Retail, Customer Relationship Management (CRM) and Digital Marketing industries collect large amounts of data about customers at different touch points across the so-called consumer journey. Data analytics provides a powerful means to gain customer insights, but their effectiveness depends on the data they are fed with. Data collected by individual companies often provide a partial view on the customer journey and the analytical models often neglect factors that have an important impact on customers’ decisions. EW-Shopp aims at supporting companies to gain deeper customer insights by helping them develop analytical services that use rich models, which also consider events that impact on customer decisions, such as weather, marketing campaigns, holidays, etc.. The main project objective is to develop a toolkit to facilitate all the data processing steps required to develop reliable weather and event-based data-driven services, including data preparation and enrichment, analytics, and visualization. In addition, the project aims at demonstrating the effectiveness of weather and event-based data analytics for developing valuable business services and the usefulness of the delivered toolkit.
The EW-Shopp toolkit has been released as a set of open source interoperable components, which can be reused even beyond the scope of project. SIx business services based on analytical models that use data about events and weather have been developed by companies in the project consortium and made available for their business units and clients. Different tools have been used in the development of the different business services, which has helped testing the components of the toolkit also on data of large size compared to the kind of analytics developed in these domains.
The EW-Shopp toolkit has been released as a set of open source interoperable components, which can be reused even beyond the scope of project. SIx business services based on analytical models that use data about events and weather have been developed by companies in the project consortium and made available for their business units and clients. Different tools have been used in the development of the different business services, which has helped testing the components of the toolkit also on data of large size compared to the kind of analytics developed in these domains.
On the technical side, the EW-Shopp toolkit consists of four main open source components: the integrated tools Grafterizer and ASIA to prepare the data and enrich them with information about events and weather, the QMiner library to develop predictive models, and the KnowAge suite to set up live dashboards.
To facilitate data enrichment, ASIA (a semantic table annotation tool) has been integrated into Grafterizer, which has been extended to support graph data storage with ArangoDB. A pool of data reconciliation and extension services have been developed for ASIA to support semantic data enrichment. A solution to execute the enrichment transformations in batch mode on a scalable cloud-based infrastructure and to schedule pre-fetching of third-party data have been developed to support large-scale enrichment processes.
To support data analytics, QMiner has been extended with tools and services for building and deploy (as REST services) event and weather-based analytic models. A keyword clustering methodology has been developed to classify keywords or short phrases to improve data analytics for Digital Marketing. As for data visualization, several customized cockpits and customized views have been developed using the KnowAge suite.
APIs, ontologies and mappings have been defined to facilitate enrichment and analytics with third-party data sources: for weather data (ECMWF and OpenWeatherMap); for event data (Media Attention API based on the Event Registry and the Event Ontology and APIs); for product data (GfK product catalog publication). Finally, a methodology to support cooperation in multi-party data analytics projects has been defined to support cooperation among partners.
On the business side, six business services that harness the power of event and weather-based analytics have been developed in the context of three business cases. The services have been developed after tests with five “pilots” developed during the first period.
Enriched Purchase Insights is a widget informing users of a comparison shopping engine when a purchase is sensible. Performance Insights is a tool for planning pricing and campaigns targeted to retailers. Both services use weather information but exploit in particular price-related events tracked in the comparison shopping engine. Weather and Event-aware BI Sales Strategy Advisor is a tool enabling maximization of marketing budgets and optimization of workforce that leverages weather and event-based prediction of foot traffic and sales. A Workforce Optimization Management Service has been developed to leverage weather and several events to optimize the shifts of the agents used in contact centers, which support CRM and marketing campaigns. The Event and Weather Dashboard helps businesses optimize their ROI based on quantitative historical analysis of customer flows vs. seasonality and the prediction of foot traffic during marketing events. Campaign Booster is a set of modules to predict the performance of keywords in digital marketing campaigns based on weather and media attention data.
Each company has developed its exploitation plan to specify how its service will be further developed after the end of the project. To support the exploitation of the toolkit and the business services, videos and dedicated web pages have been realized. Finally, toolkit-related solutions have been disseminated with scientific papers, demos and tutorials at best venues in the area of semantic computing.
To facilitate data enrichment, ASIA (a semantic table annotation tool) has been integrated into Grafterizer, which has been extended to support graph data storage with ArangoDB. A pool of data reconciliation and extension services have been developed for ASIA to support semantic data enrichment. A solution to execute the enrichment transformations in batch mode on a scalable cloud-based infrastructure and to schedule pre-fetching of third-party data have been developed to support large-scale enrichment processes.
To support data analytics, QMiner has been extended with tools and services for building and deploy (as REST services) event and weather-based analytic models. A keyword clustering methodology has been developed to classify keywords or short phrases to improve data analytics for Digital Marketing. As for data visualization, several customized cockpits and customized views have been developed using the KnowAge suite.
APIs, ontologies and mappings have been defined to facilitate enrichment and analytics with third-party data sources: for weather data (ECMWF and OpenWeatherMap); for event data (Media Attention API based on the Event Registry and the Event Ontology and APIs); for product data (GfK product catalog publication). Finally, a methodology to support cooperation in multi-party data analytics projects has been defined to support cooperation among partners.
On the business side, six business services that harness the power of event and weather-based analytics have been developed in the context of three business cases. The services have been developed after tests with five “pilots” developed during the first period.
Enriched Purchase Insights is a widget informing users of a comparison shopping engine when a purchase is sensible. Performance Insights is a tool for planning pricing and campaigns targeted to retailers. Both services use weather information but exploit in particular price-related events tracked in the comparison shopping engine. Weather and Event-aware BI Sales Strategy Advisor is a tool enabling maximization of marketing budgets and optimization of workforce that leverages weather and event-based prediction of foot traffic and sales. A Workforce Optimization Management Service has been developed to leverage weather and several events to optimize the shifts of the agents used in contact centers, which support CRM and marketing campaigns. The Event and Weather Dashboard helps businesses optimize their ROI based on quantitative historical analysis of customer flows vs. seasonality and the prediction of foot traffic during marketing events. Campaign Booster is a set of modules to predict the performance of keywords in digital marketing campaigns based on weather and media attention data.
Each company has developed its exploitation plan to specify how its service will be further developed after the end of the project. To support the exploitation of the toolkit and the business services, videos and dedicated web pages have been realized. Finally, toolkit-related solutions have been disseminated with scientific papers, demos and tutorials at best venues in the area of semantic computing.
The data preparation and enrichment component provides a novel solution to support semantic data enrichment on large data volumes, thus solving known issues of the few alternatives in the market, such as OpenRefine. It supports transformation of tabular data into RDF knowledge graphs and tabular data extension in a unique solution. Compared to other UI-based data enrichment frameworks like Knime, the tool uses a unique table-first approach to data enrichment, where the users model the transformations by annotating the table and inspecting the results on a data sample.
Analytics developed for digital marketing make use of NLP to find semantic relationships between campaigns based on the semantic similarity of the targeted keywords and to compute media attention statistics from news sources.
The Event Ontology and APIs provide a novel solution to simplify web-based event data exchange. Likewise, weather APIs support access to complex data sources such as the ECMWF making their processing more amenable for data enrichment and analytics.
The business services developed within the project show that event and weather-based data analytics can help data scientists of even small and medium European companies optimize the companies’ business, by considering additional information that they can easily access to (e.g. weather and events). These services enhance the competitiveness of these companies against IT giants that have access to massive amounts of resources and data. All the analytics are based on aggregated data, which are GDPR-compliant and preserve the users’ privacy.
Several technologies and concepts developed in EW-Shopp may find application in other domains, e.g. socio-economic studies, where external factors do impact on human behavior.
Analytics developed for digital marketing make use of NLP to find semantic relationships between campaigns based on the semantic similarity of the targeted keywords and to compute media attention statistics from news sources.
The Event Ontology and APIs provide a novel solution to simplify web-based event data exchange. Likewise, weather APIs support access to complex data sources such as the ECMWF making their processing more amenable for data enrichment and analytics.
The business services developed within the project show that event and weather-based data analytics can help data scientists of even small and medium European companies optimize the companies’ business, by considering additional information that they can easily access to (e.g. weather and events). These services enhance the competitiveness of these companies against IT giants that have access to massive amounts of resources and data. All the analytics are based on aggregated data, which are GDPR-compliant and preserve the users’ privacy.
Several technologies and concepts developed in EW-Shopp may find application in other domains, e.g. socio-economic studies, where external factors do impact on human behavior.