Recent years have seen the rapid growth of interest for cloud applications built on top of Big Data technologies such as MapReduce/Hadoop, stream processing systems, and NoSQL databases. However, there is still a shortage of software engineering models, methods, and tools for developing data-intensive software systems that can harness these technologies, taking into account quality characteristics such as efficiency, reliability, and safety.
This shortcoming is significant, since the rush to hit the Big Data market with new products leads companies to reduce the attention they pay to quality aspects, which are yet critical to avoid project failures. This issue is exacerbated by the fact that quality engineering of data-intensive software systems is still in its infancy, making it difficult today to analyze, predict and guarantee quality-of-service for this class of applications.
DICE is the first open source framework that offers a quality-aware methodology to develop and operate Big Data applications. With DICE, software vendors and developers can efficiently prototype new data-intensive applications at low cost, quickly creating business cases and proofs-of-concept for Big data technologies within their organisations. DICE encompasses quality assessment, architecture enhancement, testing and agile delivery, relying on principles of the emerging DevOps paradigm. At a high-level, the DICE paradigm aims at:
- Tackling skill shortages and learning curves in quality-driven development and Big Data technologies through open source development tools, models, and methods.
- Shortening the time to market for data-intensive applications that meet quality requirements, reducing costs for independent software vendors (ISVs) and increasing value for end-users.
- Reducing the number and the severity of quality incidents by iteratively learning the quality-levels of the application at runtime, feeding this information back to the design environment.
A free WikiBook is available that describes the DICE methodology in more detail (
http://www.dice-h2020.eu/book/(odnośnik otworzy się w nowym oknie)).
Open source and commercial versions of the DICE framework can be obtained from the project website (
http://www.dice-h2020.eu/(odnośnik otworzy się w nowym oknie)). Experimental assessments of the DICE methodology using these tools have been carried out against three industrial pilots involving:
- Stream-processing systems for social media data analysis
- Batch processing for tax fraud detection
- Cloud-based management of real-time port operations
Results indicate substantial productivity gains thanks to DICE, particularly in terms of reduction of deployment and configuration time for Big data platforms, compared to manual. The DICE framework is also able to identify several violations and anti-patterns in the application designs, as well as consistently reduce manual times for testing and system evaluation.
