Periodic Reporting for period 3 - Learn (Learning From Failing and Passing Executions At the Speed of Internet)
Reporting period: 2018-10-01 to 2019-09-30
Summary of the context and overall objectives of the project
PROBLEM AND RELEVANCE TO THE SOCIETY
Modern software systems must be extremely flexible and easily adaptable to different user needs and environments. Such flexibility requirements are so important that it is indeed common practice to develop applications that can be updated, modified and adapted in the field, directly by the end users.
However, this flexibility also introduces relevant quality issues. Almost all computer users have had the unpleasant experience to watch their favorite applications fail and crash frequently after an update. These problems are so common that is sufficient browsing the Web to find millions of reports about failures observed after updates and incompatibilities caused by the interaction of a newly installed component with the existing components. Even worse each of these problems affected a population of thousands of users.
The impact of problems introduced by end-users (e.g. the installation of a new plug-in) can be dramatic because end-users can easily modify applications, like developers do, but end-users have neither the knowledge nor the skill of developers, and they cannot debug and fix the problems that they unintentionally introduce. It is thus necessary to timely develop novel solutions that can increase the reliability of the moderns systems, which can be extended and adapted by end-users, with the capability to automatically address problems that are unknown at development-time. The Learn project aims to produce innovative solutions for the development of systems that can work around the problems introduced by end-users when using and modifying their applications.
MAIN OBJECTIVES
The main objectives are the definition of techniques to observe and collect data from the field while running at end-users site; the definition of techniques to test the software directly in the field, to proactively reveal problems and produce additional observations; the definition of learning techniques that can distill useful models out of sets of observations; the definition of a distributed infrastructure that enables communication and data exchange between instances of a same program running at different end-users’ sites; and the definition of techniques for the automatic generation and actuation of temporary fixes and workarounds.
Modern software systems must be extremely flexible and easily adaptable to different user needs and environments. Such flexibility requirements are so important that it is indeed common practice to develop applications that can be updated, modified and adapted in the field, directly by the end users.
However, this flexibility also introduces relevant quality issues. Almost all computer users have had the unpleasant experience to watch their favorite applications fail and crash frequently after an update. These problems are so common that is sufficient browsing the Web to find millions of reports about failures observed after updates and incompatibilities caused by the interaction of a newly installed component with the existing components. Even worse each of these problems affected a population of thousands of users.
The impact of problems introduced by end-users (e.g. the installation of a new plug-in) can be dramatic because end-users can easily modify applications, like developers do, but end-users have neither the knowledge nor the skill of developers, and they cannot debug and fix the problems that they unintentionally introduce. It is thus necessary to timely develop novel solutions that can increase the reliability of the moderns systems, which can be extended and adapted by end-users, with the capability to automatically address problems that are unknown at development-time. The Learn project aims to produce innovative solutions for the development of systems that can work around the problems introduced by end-users when using and modifying their applications.
MAIN OBJECTIVES
The main objectives are the definition of techniques to observe and collect data from the field while running at end-users site; the definition of techniques to test the software directly in the field, to proactively reveal problems and produce additional observations; the definition of learning techniques that can distill useful models out of sets of observations; the definition of a distributed infrastructure that enables communication and data exchange between instances of a same program running at different end-users’ sites; and the definition of techniques for the automatic generation and actuation of temporary fixes and workarounds.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
The work proceeded consistently with the workplan. We report only small differences between the planned and actual degree of development of the individual objectives. In particular, instead of developing the design for self-repairing part entirely at the beginning of the project, we realized that given the extensive diffusion of legacy non-designed for self-repair software it is more convenient and more impactful to focus on legacy systems first, while distributing along the project the work about designing systems that support self-repairing technologies. This allowed us to put additional effort in the definition of field monitoring and field testing activities, which revealed to be extremely challenging tasks.
The project has contributed more on the definition of field procedures for the identification, detection, and repair of problems, and less on the collaboration between these field procedures, since not all the studied problems required collaboration between the procedures active in the field.
Overall, the project produced relevant outcomes in all areas considered by the project: field monitoring, test automation, behavioral analysis and runtime enforcement.
We describe the advancement according to the objectives of the project.
Design for self-repairing
We defined the novel concept of proactive library, which is a library augmented with the capability to heal executions directly in-the-field. Proactive libraries can be generated semi-automatically exploiting automatic code generators starting from specifications following a model-driven paradigm. Results have been published at the International Symposium on Self-Adaptation and Self-Managing Systems (SEAMS) 2017, the International Symposium On Leveraging Applications of Formal Methods, Verification and Validation (ISOLA) 2018, and the ACM Transactions on Autonomous and Adaptive Systems (TAAS) (to appear).
Infrastructure
We defined a customized version of the JADE agent-based P2P platform that can serve the purpose of the project, providing both distribution and autonomous behaviors. Indeed, agents can be implemented to execute the technical solutions defined in the scope of the other objectives of the project. This result has been published in the Software: Practice & Experience (SPE 2019) journal edited by Wiley.
Monitoring and model synthesis
We worked on the end-user perception of the overhead and on model learners to build various kinds of finite state models that can capture the behaviour of the software. Results about the perception of the overhead and related monitoring strategies appeared at the New Ideas and Emerging Results (NIER) track of the International Conference on Software Engineering (ICSE) 2018, the IEEE Access journal 2019, in addition to papers currently under review at the International Conference on software Testing, Verification and Validation (ICST) 2020, the Journal of Systems and Software (JSS) and the ACM transactions on Software Engineering and Methodologies (TOSEM). Monitoring has been studied also in the context of cloud systems with papers appeared at the SoHeal 2018 and GAUSS 2019 workshops, as well as the European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2019. Results about model synthesis, especially about learning timed models, appeared at ICST 2017 and IEEE Transactions on Software Engineering (TSE) 2017, in addition to a paper currently under revision at IEEE TSE.
Failure detection, testing and repairing
We studied the novel concept of doing testing in the field, and reacting to failures by activating healing and repairing procedures. Results have been published at the International Symposium on Software Reliability Engineering (ISSRE) 2017, the International Conference on software Testing, Verification and Validation (ICST) 2018, and the GAUSS workshop 2019, in addition to a paper submitted to ICST 2020. Finally, a field-testing solution for microservices has been submitted to the ACM Symposium on Applied Computing (SAC) 2020, track on Cloud computing.
We also investigated the trade-off between various test case generation strategies. Results appeared in the Wiley journal on Software Testing, Verification & Reliability (STVR) 2018.
We investigated system-level test automation by defining a solution that can exploit semantic information to overcome limitations of state-of-the-art techniques mostly based on the exploration of the execution space. This work appeared at the International Conference on Software Engineering (ICSE) 2018 and has been the basis of a successful application for an ERC PoC. In the last year it has been experienced in an industrial setting and results have been published at the 3rd International Workshop on Programming Technology for the Future Web 2019.
We further explored failure detection and testing in the context of Android apps, releasing benchmarks and solutions appeared in Software: Practice & Experience (SPE) 2019, the International Symposium on Software Reliability Engineering (ISSRE) 2019, and the International Conference on Mining Software Repositories (MSR) 2019.
Finally, the activity on program repair produced a survey paper appeared in IEEE TSE 2019.
The project has contributed more on the definition of field procedures for the identification, detection, and repair of problems, and less on the collaboration between these field procedures, since not all the studied problems required collaboration between the procedures active in the field.
Overall, the project produced relevant outcomes in all areas considered by the project: field monitoring, test automation, behavioral analysis and runtime enforcement.
We describe the advancement according to the objectives of the project.
Design for self-repairing
We defined the novel concept of proactive library, which is a library augmented with the capability to heal executions directly in-the-field. Proactive libraries can be generated semi-automatically exploiting automatic code generators starting from specifications following a model-driven paradigm. Results have been published at the International Symposium on Self-Adaptation and Self-Managing Systems (SEAMS) 2017, the International Symposium On Leveraging Applications of Formal Methods, Verification and Validation (ISOLA) 2018, and the ACM Transactions on Autonomous and Adaptive Systems (TAAS) (to appear).
Infrastructure
We defined a customized version of the JADE agent-based P2P platform that can serve the purpose of the project, providing both distribution and autonomous behaviors. Indeed, agents can be implemented to execute the technical solutions defined in the scope of the other objectives of the project. This result has been published in the Software: Practice & Experience (SPE 2019) journal edited by Wiley.
Monitoring and model synthesis
We worked on the end-user perception of the overhead and on model learners to build various kinds of finite state models that can capture the behaviour of the software. Results about the perception of the overhead and related monitoring strategies appeared at the New Ideas and Emerging Results (NIER) track of the International Conference on Software Engineering (ICSE) 2018, the IEEE Access journal 2019, in addition to papers currently under review at the International Conference on software Testing, Verification and Validation (ICST) 2020, the Journal of Systems and Software (JSS) and the ACM transactions on Software Engineering and Methodologies (TOSEM). Monitoring has been studied also in the context of cloud systems with papers appeared at the SoHeal 2018 and GAUSS 2019 workshops, as well as the European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2019. Results about model synthesis, especially about learning timed models, appeared at ICST 2017 and IEEE Transactions on Software Engineering (TSE) 2017, in addition to a paper currently under revision at IEEE TSE.
Failure detection, testing and repairing
We studied the novel concept of doing testing in the field, and reacting to failures by activating healing and repairing procedures. Results have been published at the International Symposium on Software Reliability Engineering (ISSRE) 2017, the International Conference on software Testing, Verification and Validation (ICST) 2018, and the GAUSS workshop 2019, in addition to a paper submitted to ICST 2020. Finally, a field-testing solution for microservices has been submitted to the ACM Symposium on Applied Computing (SAC) 2020, track on Cloud computing.
We also investigated the trade-off between various test case generation strategies. Results appeared in the Wiley journal on Software Testing, Verification & Reliability (STVR) 2018.
We investigated system-level test automation by defining a solution that can exploit semantic information to overcome limitations of state-of-the-art techniques mostly based on the exploration of the execution space. This work appeared at the International Conference on Software Engineering (ICSE) 2018 and has been the basis of a successful application for an ERC PoC. In the last year it has been experienced in an industrial setting and results have been published at the 3rd International Workshop on Programming Technology for the Future Web 2019.
We further explored failure detection and testing in the context of Android apps, releasing benchmarks and solutions appeared in Software: Practice & Experience (SPE) 2019, the International Symposium on Software Reliability Engineering (ISSRE) 2019, and the International Conference on Mining Software Repositories (MSR) 2019.
Finally, the activity on program repair produced a survey paper appeared in IEEE TSE 2019.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
We discuss project progresses and the socio-economic impact following the same structure used to illustrate the advances.
Design for self-repairing
While there is an extensive body of knowledge about how to design systems that can tolerate failures, there are little contributions on the design of systems that can repair themselves. The work on the definition of techniques for the development of systems that can be self-repaired can produce important advances about the construction of highly reliable software systems.
In the scope of the project we elaborated the concept of proactive library, which goes exactly in this direction. This research may produce important advances on the way modern software systems are designed, resulting in systems with an advanced level of resilience to failures. Highly resilient self-repairing software systems may have a positively socio-economic impact. In particular, software systems would require less operator maintenance and would be more tolerable to mistakes made by users.
Infrastructure
The definition of an infrastructure for collecting and exchanging data is an important technical advance. In fact, most of the platforms are not thought to operate in the end-user environment, while this is the first platform designed to operate in this context. An infrastructure with these characteristics can enable a number of important researches that exploit field data and run analyses in the field.
Monitoring and model synthesis
Monitoring and model synthesis are key enablers of techniques working in the field. Field monitoring non-intrusively captures runtime data. While model synthesis efficiently produces behavioural models that describe the (correct and failing) behaviors of the software from the captured data.
Efficiently and non-intrusively collecting data and producing models from the field can enable a range of approaches based on field data rather than in-house data, giving impulse to an extremely promising research area that is still at its infancy stage. The results obtained in the scope of the project advanced the knowledge of the area by determining the degree of intervention (e.g. overhead) that can be tolerated by the end-users. This is fundamental to design techniques that work in the field transparently. Our studies performed with human subjects further expand the knowledge in the area.
Failure detection, testing and repairing
The project is investigating the nature of the faults that must be revealed and repaired in the field. This may significantly impact future research in software testing and analysis and motivate approaches working in the field. Indeed, we identified faults that can be hardly revealed in house, but that can be more easily revealed in the field.
Our study on field-testing provides a novel perspective on the strategies that should be designed to address most of the faults that are revealed in the field. In particular, it provides clear motivation for approaches working in the field. This research might have an impact on the research in the field, in addition than this project itself.
Our survey on repair approaches is the first comprehensive and systematic study of the existing approaches. This work will be likely the basis for several other future work in the area
Design for self-repairing
While there is an extensive body of knowledge about how to design systems that can tolerate failures, there are little contributions on the design of systems that can repair themselves. The work on the definition of techniques for the development of systems that can be self-repaired can produce important advances about the construction of highly reliable software systems.
In the scope of the project we elaborated the concept of proactive library, which goes exactly in this direction. This research may produce important advances on the way modern software systems are designed, resulting in systems with an advanced level of resilience to failures. Highly resilient self-repairing software systems may have a positively socio-economic impact. In particular, software systems would require less operator maintenance and would be more tolerable to mistakes made by users.
Infrastructure
The definition of an infrastructure for collecting and exchanging data is an important technical advance. In fact, most of the platforms are not thought to operate in the end-user environment, while this is the first platform designed to operate in this context. An infrastructure with these characteristics can enable a number of important researches that exploit field data and run analyses in the field.
Monitoring and model synthesis
Monitoring and model synthesis are key enablers of techniques working in the field. Field monitoring non-intrusively captures runtime data. While model synthesis efficiently produces behavioural models that describe the (correct and failing) behaviors of the software from the captured data.
Efficiently and non-intrusively collecting data and producing models from the field can enable a range of approaches based on field data rather than in-house data, giving impulse to an extremely promising research area that is still at its infancy stage. The results obtained in the scope of the project advanced the knowledge of the area by determining the degree of intervention (e.g. overhead) that can be tolerated by the end-users. This is fundamental to design techniques that work in the field transparently. Our studies performed with human subjects further expand the knowledge in the area.
Failure detection, testing and repairing
The project is investigating the nature of the faults that must be revealed and repaired in the field. This may significantly impact future research in software testing and analysis and motivate approaches working in the field. Indeed, we identified faults that can be hardly revealed in house, but that can be more easily revealed in the field.
Our study on field-testing provides a novel perspective on the strategies that should be designed to address most of the faults that are revealed in the field. In particular, it provides clear motivation for approaches working in the field. This research might have an impact on the research in the field, in addition than this project itself.
Our survey on repair approaches is the first comprehensive and systematic study of the existing approaches. This work will be likely the basis for several other future work in the area