Periodic Reporting for period 5 - EPIC (Evolving Program Improvement Collaborators)
Reporting period: 2024-03-01 to 2024-06-30
The EPIC project combines computational search and machine learning to improve the quality of software and the users experience. The EPIC research team at UCL is supervised by Professors Harman and Sarro, working on the scientific foundations of this approach, while Prof. Harman also works at Meta (formerly Facebook), thereby providing avenues for impact and adoption of previously published scientific work.
EPIC tackles the problem of software bugs, which are time-consuming and expensive to fix using human effort but which, using the team’s techniques, can be automatically identified, fixed and re-tested by a combination of computational search on machine learning. This work builds the scientific foundation for (and develops the practice of) software systems that automatically diagnose and fix their own faults.
The project has devised novel and practical approaches to tackle emerging challenges posed by the latest technologies in the field, including producing industry-relevant work on automatically find and repair bugs and safety assurance of LLMs e.g.[1--9]. The EPIC research team has also established a rigorous scientific evaluation framework for these techniques e.g.[10-11]. As, many of these automated approaches draw on computational search and increasingly machine learning/artificial intelligence, the team has also provided fundamental scientific foundations for the evaluation of these approaches e.g.[12--15].
Ref.
[1] SapFix: automated end-to-end repair at scale. ICSE 2019
[2] The importance of accounting for real-world labelling when predicting software vulnerabilities. FSE 2019
[3] Machine Learning Testing: Survey, Landscapes and Horizons. IEEE TSE 2020
[4] A Comprehensive Empirical Study on Automatic API-Misure Repair. IEEE TSE 2020
[5] Automatic testing and improvement of machine translation ICSE 2020
[6] Enhancing Genetic Improvement of Software with Regression Test Selection. ICSE 2021
[7] Testing Web Enabled Simulation at Scale Using Metamorphic Testing, ICSE 2021
[8] User-Centric Deployment of Automated Program Repair at Bloomberg, ICSE 2024
[9] Automated Unit Test Improvement using Large Language Models at Meta FSE 2024 Companion
[10] Fairness Testing: A Comprehensive Survey and Analysis of Trends. TOSEM 2024
[11] Multi-objective search for gender-fair and semantically correct word embeddings. Applied Soft Computing 2022
[12] Ignorance and Prejudice in Software Fairness. ICSE 2021
[13] Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods. FSE 2022
[14] Fairness Improvement with Multiple Protected Attributes: How Far Are We?. ICSE 2024
[15] Green AI: Do Deep Learning Frameworks Have Different Costs? ICSE 2022
[16] Enhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurement. TOSEM 2024
EPIC tackles the problem of software bugs, which are time-consuming and expensive to fix using human effort but which, using the team’s techniques, can be automatically identified, fixed and re-tested by a combination of computational search on machine learning. This work builds the scientific foundation for (and develops the practice of) software systems that automatically diagnose and fix their own faults.
The project has devised novel and practical approaches to tackle emerging challenges posed by the latest technologies in the field, including producing industry-relevant work on automatically find and repair bugs and safety assurance of LLMs e.g.[1--9]. The EPIC research team has also established a rigorous scientific evaluation framework for these techniques e.g.[10-11]. As, many of these automated approaches draw on computational search and increasingly machine learning/artificial intelligence, the team has also provided fundamental scientific foundations for the evaluation of these approaches e.g.[12--15].
Ref.
[1] SapFix: automated end-to-end repair at scale. ICSE 2019
[2] The importance of accounting for real-world labelling when predicting software vulnerabilities. FSE 2019
[3] Machine Learning Testing: Survey, Landscapes and Horizons. IEEE TSE 2020
[4] A Comprehensive Empirical Study on Automatic API-Misure Repair. IEEE TSE 2020
[5] Automatic testing and improvement of machine translation ICSE 2020
[6] Enhancing Genetic Improvement of Software with Regression Test Selection. ICSE 2021
[7] Testing Web Enabled Simulation at Scale Using Metamorphic Testing, ICSE 2021
[8] User-Centric Deployment of Automated Program Repair at Bloomberg, ICSE 2024
[9] Automated Unit Test Improvement using Large Language Models at Meta FSE 2024 Companion
[10] Fairness Testing: A Comprehensive Survey and Analysis of Trends. TOSEM 2024
[11] Multi-objective search for gender-fair and semantically correct word embeddings. Applied Soft Computing 2022
[12] Ignorance and Prejudice in Software Fairness. ICSE 2021
[13] Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods. FSE 2022
[14] Fairness Improvement with Multiple Protected Attributes: How Far Are We?. ICSE 2024
[15] Green AI: Do Deep Learning Frameworks Have Different Costs? ICSE 2022
[16] Enhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurement. TOSEM 2024
Overall, the output of the EPIC project has been disseminated thought the production of over 100 scholarly peer-reviewed articles, published in the most renewed software engineering venues.
The EPIC's output received considerable media attention, 7 international research awards and Professors Harman and Sarro received 5 career awards in recognition of their outstanding work.
The research carried out has also attracted considerable media coverage. EPIC’s work has not only significantly pushed the boundaries of the state-of-the-art research, but has also had a significant impact on society. For example, automatic repair have been applied at scale at Meta (formerly Facebook), where they have automatically found and fixed real software bugs in tens of millions of lines of production software code, improving the communications systems used by over 2.5 billion people worldwide. Automatically finding and fixing bugs, means that they never reach production, and no user is ever impacted. The approach also frees up developers’ time so that they can use their considerable human ingenuity on new features rather than the tedious and time-consuming task of finding and fixing faults. Harman also worked on web enabled simulation, which extends the principle of automated software fault finding and fixing to so-called social bugs. With this new development Harman’s EPIC project ideas may tackle, not only bugs that impede individual users experience, but also the ways in which one user may use software to harm another user.
Awards
ACM SIGSOFT Distinguished Paper Award at IEEE/ACM ICSE 2024
IEEE Best paper award at IEEE/ACM ESEM 2024
Best RENE Paper Award at IEEE SANER 2024
Best Paper Award at SSBSE 2024 Challenge
ACM SIGSOFT Distinguished Paper Award at MSR 2023
Best Paper Award at SSBSE 2021 Challenge
ACM SIGSOFT Distinguished Artifact Award at ICSE 2021
ACM Outstanding Research award to Mark Harman. The top ACM award for SE
IEEE Harlan Mills award to Mark Harman. The top IEEE award for SE
Fellow of the Royal Academy of Engineering to Mark Harman. The highest honours an engineer can received in the UK
IEEE CS TCSE Rising star Award 2021 to Federica Sarro. The top IEEE award for young SE academics
LERO Parnas Fellowship to Federica Sarro - Highest recognition for software engineering researchers in Ireland
Keynotes (selection)
ISSTA 2019 opening keynote by M. Harman.
ICSE 2019 keynote by Mark Harman
POPL 2019 opening keynote
ICPE 2023 opening keynote by F. Sarro
RE 2023 keynote by F. Sarro
Media Coverage
SapFix was covered by over 20 media outlets including Forbes, SD Times, CNET, SiliconANGLE, TechCrunch, the Verge, Tom's Guide.
Harman's Interview with Sam Shead: 10-2-2019 Forbes article
Harman's Interview with Michael Martinez for the IEEE Computer Society
Magazine feature: IEEE Spectrum interview and write up by: Amy Nordrum. Her article was published on January 2019.
Over 60 different media outlets reported on the development of web enabled simulation at Facebook including:
The Verge: https://www.theverge.com/2020/4/15/21221992/facebook-wes-simulation-research-paper-bots-scammers-new-feature(opens in new window)
The Telegraph: https://www.telegraph.co.uk/technology/2020/04/15/facebook-reveals-has-hidden-parallel-social-network-filled-bots/(opens in new window)
MIT technology review: https://www.technologyreview.com/2020/04/15/999871/facebook-ai-bot-simulation/(opens in new window)
The Independent newspaper: https://www.independent.co.uk/life-style/gadgets-and-tech/news/facebook-secret-social-network-ai-bots-simulation-scams-trolls-a9468026.html(opens in new window)
Newsweek: https://www.newsweek.com/facebook-bot-universe-trolls-scammers-web-enabled-simulation-ww-social-media-1498287(opens in new window)
Sarro's Interview with Tim Menzies for the IEEE Software Magazine 2024
The EPIC's output received considerable media attention, 7 international research awards and Professors Harman and Sarro received 5 career awards in recognition of their outstanding work.
The research carried out has also attracted considerable media coverage. EPIC’s work has not only significantly pushed the boundaries of the state-of-the-art research, but has also had a significant impact on society. For example, automatic repair have been applied at scale at Meta (formerly Facebook), where they have automatically found and fixed real software bugs in tens of millions of lines of production software code, improving the communications systems used by over 2.5 billion people worldwide. Automatically finding and fixing bugs, means that they never reach production, and no user is ever impacted. The approach also frees up developers’ time so that they can use their considerable human ingenuity on new features rather than the tedious and time-consuming task of finding and fixing faults. Harman also worked on web enabled simulation, which extends the principle of automated software fault finding and fixing to so-called social bugs. With this new development Harman’s EPIC project ideas may tackle, not only bugs that impede individual users experience, but also the ways in which one user may use software to harm another user.
Awards
ACM SIGSOFT Distinguished Paper Award at IEEE/ACM ICSE 2024
IEEE Best paper award at IEEE/ACM ESEM 2024
Best RENE Paper Award at IEEE SANER 2024
Best Paper Award at SSBSE 2024 Challenge
ACM SIGSOFT Distinguished Paper Award at MSR 2023
Best Paper Award at SSBSE 2021 Challenge
ACM SIGSOFT Distinguished Artifact Award at ICSE 2021
ACM Outstanding Research award to Mark Harman. The top ACM award for SE
IEEE Harlan Mills award to Mark Harman. The top IEEE award for SE
Fellow of the Royal Academy of Engineering to Mark Harman. The highest honours an engineer can received in the UK
IEEE CS TCSE Rising star Award 2021 to Federica Sarro. The top IEEE award for young SE academics
LERO Parnas Fellowship to Federica Sarro - Highest recognition for software engineering researchers in Ireland
Keynotes (selection)
ISSTA 2019 opening keynote by M. Harman.
ICSE 2019 keynote by Mark Harman
POPL 2019 opening keynote
ICPE 2023 opening keynote by F. Sarro
RE 2023 keynote by F. Sarro
Media Coverage
SapFix was covered by over 20 media outlets including Forbes, SD Times, CNET, SiliconANGLE, TechCrunch, the Verge, Tom's Guide.
Harman's Interview with Sam Shead: 10-2-2019 Forbes article
Harman's Interview with Michael Martinez for the IEEE Computer Society
Magazine feature: IEEE Spectrum interview and write up by: Amy Nordrum. Her article was published on January 2019.
Over 60 different media outlets reported on the development of web enabled simulation at Facebook including:
The Verge: https://www.theverge.com/2020/4/15/21221992/facebook-wes-simulation-research-paper-bots-scammers-new-feature(opens in new window)
The Telegraph: https://www.telegraph.co.uk/technology/2020/04/15/facebook-reveals-has-hidden-parallel-social-network-filled-bots/(opens in new window)
MIT technology review: https://www.technologyreview.com/2020/04/15/999871/facebook-ai-bot-simulation/(opens in new window)
The Independent newspaper: https://www.independent.co.uk/life-style/gadgets-and-tech/news/facebook-secret-social-network-ai-bots-simulation-scams-trolls-a9468026.html(opens in new window)
Newsweek: https://www.newsweek.com/facebook-bot-universe-trolls-scammers-web-enabled-simulation-ww-social-media-1498287(opens in new window)
Sarro's Interview with Tim Menzies for the IEEE Software Magazine 2024
The project has addressed important themes like automated programming, testing and repair of modern software systems. Some of this work has been deployed in industry, thus providing billions of end-users with more robust, hence, safer software. The work on finding and fixing bugs automatically goes considerably beyond the current state of the art. which involves techniques for automatically finding faults, but the end-to-end finding and fixing these faults is a quantum leap beyond the state of the art. This is internationally recognised in the scientific awards for the PI and his team, and the high citation counts, academic recognition, and industrial impact of the research outputs.