Periodic Reporting for period 2 - ELASTEST (ElasTest: an elastic platform for testing complex distributed large software systems)
Periodo di rendicontazione: 2018-07-01 al 2019-12-31
Large distributed software is promised to be fault-tolerant, elastic (it scales in and out) and cost-efficient, among other features. However, it is difficult to assess that distributed software complies with the quality demanded by users. For example, whenever developers want to validate non-functional features such as scalability, fault-tolerance or data consistency, they need to create complex testing architectures customizing the cloud orchestration mechanisms and managing test scalability by themselves. Things become even more complex when trying to reproduce real-world operational conditions.
ElasTest strategy for taming and diminishing the testing costs without compromising quality and even improving it, is based on three axes:
* Reducing the time-to-market: ElasTest provides useful services that can be used “out-of-the-box”, through well-documented REST APIs. These services can be used isolated or they can be combined, and they provide on demand the following tools that can be exploited from the test code: browsers, IoT devices, security checks, monitoring and data analytics.
* Reducing the bug-localization time: ElasTest provides visualizations and analytic tools that help users to find the needle in the haystack, like log analytics, task focused reporting and log comparison.
* Simplifying the orchestration of tests: by means of a domain specific language.
All in all, ElasTest makes any software development team capable of delivering software faster, more confidently, more securely and with fewer defects. This has been validated with our five applications in four different domains (web, 5G, real-time communication and IoT), and through our collaboration with external companies.
ElasTest strategy for taming and diminishing the testing costs without compromising quality and even improving it, is based on three axes:
* Reducing the time-to-market: ElasTest provides useful services that can be used “out-of-the-box”, through well-documented REST APIs. These services can be used isolated or they can be combined, and they provide on demand the following tools that can be exploited from the test code: browsers, IoT devices, security checks, monitoring and data analytics.
* Reducing the bug-localization time: ElasTest provides visualizations and analytic tools that help users to find the needle in the haystack, like log analytics, task focused reporting and log comparison.
* Simplifying the orchestration of tests: by means of a domain specific language.
All in all, ElasTest makes any software development team capable of delivering software faster, more confidently, more securely and with fewer defects. This has been validated with our five applications in four different domains (web, 5G, real-time communication and IoT), and through our collaboration with external companies.
ElasTest has created an innovative cloud architecture designed for elasticity, considering the fact that testing large distributed systems requires also highly elastic and distributed testing tools. At a very high-level perspective these are the main functions of such architecture:
* ElasTest Core Components: constitute the enablers of the platform and are responsible for providing management of jobs and the software under test.
* ElasTest Test Support Services: on demand services providing various testing capabilities to the tests.
* ElasTest Test Engines: offer additional capabilities to help in the testing process that can be used by the platform itself or by end-users.
* ElasTest integrations: Jenkins and TestLink.
* ElasTest test orchestrator: a domain specific language which allows to concisely describe complex test scenarios and expectations.
The main contributions of the project are the following:
* An elastic comprehensive tool that is mature with more than 15 public releases since the beginning of the project.
* State-of-the-art, both from an academic and industrial perspective
* Validation through demonstrators. The project made a big effort on driving a pilot experiment with demonstrators using the preliminary versions of the ElasTest platform. The results found from this validation include:
+ In some cases teams achieved 88% reusability of their test cases with ElasTest, even for exploratory testing (a manual testing activity) thanks to the cross-browser support, and a 267% of reusability for automated test cases.
+ A reduction of 90% in coding test cases for automated tests in a real-time communication vertical.
+ Maintenance effort reduced in 83%.
+ Time devoted to security testing decreased by 98%.
+ Reduction by 58% in time spent designing and coding IoT tests
* Dissemination. Since the beginning, 2 book chapters, a BsC thesis, 20 conference papers, 8 journal papers, 3MSc thesis, and 1 PhD have been published. The partners organized 4 workshops, and attended more than 50 events, both academic and industrial. There was a collaboration effort with 2 external companies, still ongoing, and a collaboration with the H2020 STAMP project (https://www.stamp-project.eu/) that led to a proof of concept integration of STAMP tools within ElasTest.
* Communication through the project’s communication channels. The consortium wrote 43 blog entries, 883 tweets and 513 followers on Twitter, 40 videos on YouTube with a total of 2133 views, 14 presentations in SlideShare with more than 7,700 views, and more than 13,900 visitors to the project’s webpage from all around the world.
* ElasTest Core Components: constitute the enablers of the platform and are responsible for providing management of jobs and the software under test.
* ElasTest Test Support Services: on demand services providing various testing capabilities to the tests.
* ElasTest Test Engines: offer additional capabilities to help in the testing process that can be used by the platform itself or by end-users.
* ElasTest integrations: Jenkins and TestLink.
* ElasTest test orchestrator: a domain specific language which allows to concisely describe complex test scenarios and expectations.
The main contributions of the project are the following:
* An elastic comprehensive tool that is mature with more than 15 public releases since the beginning of the project.
* State-of-the-art, both from an academic and industrial perspective
* Validation through demonstrators. The project made a big effort on driving a pilot experiment with demonstrators using the preliminary versions of the ElasTest platform. The results found from this validation include:
+ In some cases teams achieved 88% reusability of their test cases with ElasTest, even for exploratory testing (a manual testing activity) thanks to the cross-browser support, and a 267% of reusability for automated test cases.
+ A reduction of 90% in coding test cases for automated tests in a real-time communication vertical.
+ Maintenance effort reduced in 83%.
+ Time devoted to security testing decreased by 98%.
+ Reduction by 58% in time spent designing and coding IoT tests
* Dissemination. Since the beginning, 2 book chapters, a BsC thesis, 20 conference papers, 8 journal papers, 3MSc thesis, and 1 PhD have been published. The partners organized 4 workshops, and attended more than 50 events, both academic and industrial. There was a collaboration effort with 2 external companies, still ongoing, and a collaboration with the H2020 STAMP project (https://www.stamp-project.eu/) that led to a proof of concept integration of STAMP tools within ElasTest.
* Communication through the project’s communication channels. The consortium wrote 43 blog entries, 883 tweets and 513 followers on Twitter, 40 videos on YouTube with a total of 2133 views, 14 presentations in SlideShare with more than 7,700 views, and more than 13,900 visitors to the project’s webpage from all around the world.
Progress beyond the state of the art was achieved in the following areas:
* Some Test Support Services, namely: ElasTest Monitoring Service (EMS), ElasTest Security Service (ESS), and ElasTest User Impersonation Service (EUS). ESS provided a new tool for performing COSI attacks, EUS provided a new algorithm adapted from Netflix for measuring Quality of Experience (QoE) in WebRTC applications, and EMS provided novel ways through its DSL to build monitors.
* Test orchestration: this concept revolves around the idea of building complex test suites from simple test cases. To the best of our knowledge, there is no formalization of this concept of orchestrating test cases. The test orchestration mechanisms allow users to augment their test suites with real world conditions effortlessly. It also allows to prioritize test cases based on context-aware information.
* Test recommendation: many applications are complex in nature. Being sure that the tests cover all the functional and non-functional aspects of the software can be difficult when managing a big set of test cases. Test recommenders provided by ElasTest intend to support test automation, as well as to help testers and developers to identify those tests that are missing from the test cases and provide examples from previous projects. These features are provided by the ElasTest Recommendation Engine and the ElasTest Question and Answer (QA) Engine. These tools can advise testers using crowdsourced knowledge, and learn from open source projects as well as from the user own projects, in order to recommend accurate test implementations. These tools are a step forward in the state of the art of test generation tools.
* Additional research has been carried on in other areas, specifically:
+ Testing & cloud: CNR and URJC collaboratively built a systematic literature review of testing related to the cloud. This review has been submitted to a high impact journal as described in D2.2 [1].
+ Testing 5G infrastructures: URJC, TUB and FOKUS presented a preliminary work on testing 5G infrastructures at IEEE SDN/NFV conference, and we expect relevant synergies with the 5G stakeholders. Giving the expectations arisen for 5G, and the fact that most operators are migrating from hardware to software assets, testing activities might have a big impact on this area.
The project is expected to have an economic impact on the cost of testing and testing-related activities, like bug-localization (finding the root cause of a failed test). Current testing approaches avoid end-to-end testing as much as possible, due to the costs of implementing these end-to-end tests, or do it manually to avoid the costs of setup and automating end-to-end test cases. ElasTest allows even the most complex test scenarios to be developed or run with a fraction of the efforts, and enhances communication across teams, due to the support of exploratory testing with the advantage of the observability features built-in.
* Some Test Support Services, namely: ElasTest Monitoring Service (EMS), ElasTest Security Service (ESS), and ElasTest User Impersonation Service (EUS). ESS provided a new tool for performing COSI attacks, EUS provided a new algorithm adapted from Netflix for measuring Quality of Experience (QoE) in WebRTC applications, and EMS provided novel ways through its DSL to build monitors.
* Test orchestration: this concept revolves around the idea of building complex test suites from simple test cases. To the best of our knowledge, there is no formalization of this concept of orchestrating test cases. The test orchestration mechanisms allow users to augment their test suites with real world conditions effortlessly. It also allows to prioritize test cases based on context-aware information.
* Test recommendation: many applications are complex in nature. Being sure that the tests cover all the functional and non-functional aspects of the software can be difficult when managing a big set of test cases. Test recommenders provided by ElasTest intend to support test automation, as well as to help testers and developers to identify those tests that are missing from the test cases and provide examples from previous projects. These features are provided by the ElasTest Recommendation Engine and the ElasTest Question and Answer (QA) Engine. These tools can advise testers using crowdsourced knowledge, and learn from open source projects as well as from the user own projects, in order to recommend accurate test implementations. These tools are a step forward in the state of the art of test generation tools.
* Additional research has been carried on in other areas, specifically:
+ Testing & cloud: CNR and URJC collaboratively built a systematic literature review of testing related to the cloud. This review has been submitted to a high impact journal as described in D2.2 [1].
+ Testing 5G infrastructures: URJC, TUB and FOKUS presented a preliminary work on testing 5G infrastructures at IEEE SDN/NFV conference, and we expect relevant synergies with the 5G stakeholders. Giving the expectations arisen for 5G, and the fact that most operators are migrating from hardware to software assets, testing activities might have a big impact on this area.
The project is expected to have an economic impact on the cost of testing and testing-related activities, like bug-localization (finding the root cause of a failed test). Current testing approaches avoid end-to-end testing as much as possible, due to the costs of implementing these end-to-end tests, or do it manually to avoid the costs of setup and automating end-to-end test cases. ElasTest allows even the most complex test scenarios to be developed or run with a fraction of the efforts, and enhances communication across teams, due to the support of exploratory testing with the advantage of the observability features built-in.