Periodic Reporting for period 2 - STAMP (Software Testing AMPlification)
Período documentado: 2018-06-01 hasta 2019-11-30
Leveraging advanced research in automatic test generation, STAMP aims at pushing automation in DevOps one step further through the novel concept of test amplification. The main objective of STAMP is to automatically transform existing test assets in order to detect regression bugs before production and drive down the cost of software testing.
We address this ambitious objective through increased test automation and research and development activities articulated around the objectives presented below:
- Objective 1. Provide an approach to automatically amplify unit test cases when a change is introduced in a program.
- Objective 2. Provide an approach to automatically generate, deploy and test large numbers of system configurations.
- Objective 3. Provide an approach to automatically amplify, optimize and analyze production logs in order to retrieve test cases that verify code changes against real world conditions.
- Objective 4. Develop test amplification microservices that can fit the DevOps pipelines.
- Objective 5. Validate the relevance and effectiveness of amplification on 5 use cases.
- Objective 6. Disseminate and exploit the open source STAMP test amplification services.
We address this ambitious objective through increased test automation and research and development activities articulated around the objectives presented below:
- Objective 1. Provide an approach to automatically amplify unit test cases when a change is introduced in a program.
- Objective 2. Provide an approach to automatically generate, deploy and test large numbers of system configurations.
- Objective 3. Provide an approach to automatically amplify, optimize and analyze production logs in order to retrieve test cases that verify code changes against real world conditions.
- Objective 4. Develop test amplification microservices that can fit the DevOps pipelines.
- Objective 5. Validate the relevance and effectiveness of amplification on 5 use cases.
- Objective 6. Disseminate and exploit the open source STAMP test amplification services.
Objective 1 has been addressed with the following activities
DSpot: In the second period of STAMP we have focused the research about amplification on the following question: can DSpot amplify test cases that focus on the code changed in one commit and can these test cases detect the behavior change introduced by the commit?
Descartes. The key research question for second period was: can we generate hints that indicate what test improvement actions developers can take to improve their test suite?
We have published a survey about test amplification in JSS, a top software engineering journal. Latest results are under review in EMSE and TSE.
Objective 2 has been addressed with the following activities
- New features in the CAMP tool, including: new strategies to reduce the number of generated configurations to be tested and to select the most relevant configurations; integration with JUnit and JMeter to execute unit or performance tests on the different configuration; runtime monitoring.
- Conduct experiments outside STAMP use cases on two large open-source projects, respectively with 13k and 36k commits, and 2.5k and 48k stars on GitHub, where we detected important issues.
- Publications at IEEE ISSRE, the top conference for software reliability engineering, tutorials and technical talks e.g. at the A-TEST workshop at FSE.
Objective 3 has been addressed with the following activities
- Development of the first crash reproduction benchmark for Java (JCrashPack).
- Extension of EvoSuite for Runtime AMPlification (RAMP) to take the behavior of the software under test into account to generate unit tests (model seeding).
- Dissemination of the results in the scientific community through high-quality publications.
Objective 4 has been addressed with the following activities.
- integrate STAMP tools with Jira Software and Github Issues issue trackers;
- a new Jenkins plugin, a pipeline library and pipelines to enhance CI/CD processes with STAMP features;
- courseware based on documentation, sample pipelines and a Docker image containing a full stack CI/CD environment enhanced with STAMP features, available in DockerHub;
Objective 5 has been addressed with the following activities:
- KPIs have been reviewed and tuned to better match the reality and to provide more interesting results. New metrics have been added to also provide better visualization of results.
- Each Industrial UCs have experimented all the STAMP tools by applying them into their different case projects they selected for evaluation.
- Industrial UCs have incorporated some of the STAMP tools (and technologies) into their CI/CD procedures (e.g. pipelines) and/or their software development processes (either by through project management tools or an IDE).
- Industrial partners have computed KPI metrics, based on conducted experiments, and reported them. From these KPIs measurements, they have drawn conclusions about the effectiveness of the STAMP test amplification techniques on their industrial software development processes.
Objective 6 has been addressed with the following activities
- Collateral. Developed a full range of communication material including an efficient visual identity, 14 videos [18 over 3 years], a tri-fold brochure, an A5 flyer for the beta-testing campaign, a roll-up poster for exhibition, t-shirts for beta-testers.
- Events. Showcased the STAMP projects in 15 industry events worldwide [27 over 3 years]
- Workshops. Three webinars and 16 workshops [20 over 3 years] were organized for in-depth presentations of the tools to academic and industry potential users.
- Communication. The consortium had an on-going communication activity directed at the press and social media.
- Scientific dissemination. The consortium published 12 papers in scientific publications [31 over three years], enabled the completion of three PhD’s papers and participated in dedicated scientific and academic conferences such as ISSRE, ISSTA, ICSE , ICST
DSpot: In the second period of STAMP we have focused the research about amplification on the following question: can DSpot amplify test cases that focus on the code changed in one commit and can these test cases detect the behavior change introduced by the commit?
Descartes. The key research question for second period was: can we generate hints that indicate what test improvement actions developers can take to improve their test suite?
We have published a survey about test amplification in JSS, a top software engineering journal. Latest results are under review in EMSE and TSE.
Objective 2 has been addressed with the following activities
- New features in the CAMP tool, including: new strategies to reduce the number of generated configurations to be tested and to select the most relevant configurations; integration with JUnit and JMeter to execute unit or performance tests on the different configuration; runtime monitoring.
- Conduct experiments outside STAMP use cases on two large open-source projects, respectively with 13k and 36k commits, and 2.5k and 48k stars on GitHub, where we detected important issues.
- Publications at IEEE ISSRE, the top conference for software reliability engineering, tutorials and technical talks e.g. at the A-TEST workshop at FSE.
Objective 3 has been addressed with the following activities
- Development of the first crash reproduction benchmark for Java (JCrashPack).
- Extension of EvoSuite for Runtime AMPlification (RAMP) to take the behavior of the software under test into account to generate unit tests (model seeding).
- Dissemination of the results in the scientific community through high-quality publications.
Objective 4 has been addressed with the following activities.
- integrate STAMP tools with Jira Software and Github Issues issue trackers;
- a new Jenkins plugin, a pipeline library and pipelines to enhance CI/CD processes with STAMP features;
- courseware based on documentation, sample pipelines and a Docker image containing a full stack CI/CD environment enhanced with STAMP features, available in DockerHub;
Objective 5 has been addressed with the following activities:
- KPIs have been reviewed and tuned to better match the reality and to provide more interesting results. New metrics have been added to also provide better visualization of results.
- Each Industrial UCs have experimented all the STAMP tools by applying them into their different case projects they selected for evaluation.
- Industrial UCs have incorporated some of the STAMP tools (and technologies) into their CI/CD procedures (e.g. pipelines) and/or their software development processes (either by through project management tools or an IDE).
- Industrial partners have computed KPI metrics, based on conducted experiments, and reported them. From these KPIs measurements, they have drawn conclusions about the effectiveness of the STAMP test amplification techniques on their industrial software development processes.
Objective 6 has been addressed with the following activities
- Collateral. Developed a full range of communication material including an efficient visual identity, 14 videos [18 over 3 years], a tri-fold brochure, an A5 flyer for the beta-testing campaign, a roll-up poster for exhibition, t-shirts for beta-testers.
- Events. Showcased the STAMP projects in 15 industry events worldwide [27 over 3 years]
- Workshops. Three webinars and 16 workshops [20 over 3 years] were organized for in-depth presentations of the tools to academic and industry potential users.
- Communication. The consortium had an on-going communication activity directed at the press and social media.
- Scientific dissemination. The consortium published 12 papers in scientific publications [31 over three years], enabled the completion of three PhD’s papers and participated in dedicated scientific and academic conferences such as ISSRE, ISSTA, ICSE , ICST
Results on Unit test amplification show
- fully automated amplification is feasible with real-world, open source software package
- automatically amplified unit tests have been integrated in the test suites of 4 open source projects
These results are under revision for the Journal on Empirical Software Engineering (EMSE).
A survey on test amplification is submitted to the Journal of Systems and Software.
Results on Test quality assessment
- the first ever analysis of thousands of pseudo-tested methods measures how pseduo-tested methods are different from the other covered methods
- a qualitative manual analysis of 525 pseudo-tested methods, involving developers, reveals that less than 40% of these methods are clearly worth of additional testing effort.
Submitted to the Journal on Empirical Software Engineering (EMSE), and tool demo accepted at ASE.
XWiki embedded Descartes in their continuous integration engine since 4 months.
Results on configuration tests amplification
- demonstrate the feasibility, with the ATOS CityGo and XWiki use-cases, of a domain specific language to specify the different dimensions of environment configurations
- automatically analyze these descriptions and generate variations of existing configurations
- automatically generate Docker images that implement these new configurations and run the integration test suites on these new configurations.
Runtime test amplification
- Botsing can replicate 41 (82%) of real-world crashes, 34 (89%) of which are useful reproductions for debugging purposes
- a benchmark of 200 crash cases
- refine the guidance of the algorithm using multi-objectivization
- add seeding strategies for EvoCrash
Integration and industrialization
- Maven and Gradle plugins for STAMP tools, to support the integration of the tools in continuous integrations pipelines.
- Visual reporting support
- Eclipse plugins for DSpot and Descartes
Validation
- Refined definitions of KPIs.
- Definition of the evaluation roadmap and framework.
- Online survey and physical workshop with software developers who are outside the STAMP consortium. At the moment, 56 responses have been collected and are being analized.
- fully automated amplification is feasible with real-world, open source software package
- automatically amplified unit tests have been integrated in the test suites of 4 open source projects
These results are under revision for the Journal on Empirical Software Engineering (EMSE).
A survey on test amplification is submitted to the Journal of Systems and Software.
Results on Test quality assessment
- the first ever analysis of thousands of pseudo-tested methods measures how pseduo-tested methods are different from the other covered methods
- a qualitative manual analysis of 525 pseudo-tested methods, involving developers, reveals that less than 40% of these methods are clearly worth of additional testing effort.
Submitted to the Journal on Empirical Software Engineering (EMSE), and tool demo accepted at ASE.
XWiki embedded Descartes in their continuous integration engine since 4 months.
Results on configuration tests amplification
- demonstrate the feasibility, with the ATOS CityGo and XWiki use-cases, of a domain specific language to specify the different dimensions of environment configurations
- automatically analyze these descriptions and generate variations of existing configurations
- automatically generate Docker images that implement these new configurations and run the integration test suites on these new configurations.
Runtime test amplification
- Botsing can replicate 41 (82%) of real-world crashes, 34 (89%) of which are useful reproductions for debugging purposes
- a benchmark of 200 crash cases
- refine the guidance of the algorithm using multi-objectivization
- add seeding strategies for EvoCrash
Integration and industrialization
- Maven and Gradle plugins for STAMP tools, to support the integration of the tools in continuous integrations pipelines.
- Visual reporting support
- Eclipse plugins for DSpot and Descartes
Validation
- Refined definitions of KPIs.
- Definition of the evaluation roadmap and framework.
- Online survey and physical workshop with software developers who are outside the STAMP consortium. At the moment, 56 responses have been collected and are being analized.