Self-assessment Oracles for Anticipatory Testing

Informations projet

PRECRIME

N° de convention de subvention: 787703

DOI

10.3030/787703

Projet clôturé

Date de signature de la CE 13 Juillet 2018

Date de début 1 Janvier 2019

Date de fin 31 Decembre 2023

Financé au titre de

• EXCELLENT SCIENCE - European Research Council (ERC)

Coût total

€ 2 313 250,00

Contribution de l’UE

€ 2 313 250,00

Coordonné par

UNIVERSITA DELLA SVIZZERA ITALIANA
Switzerland

CORDIS fournit des liens vers les livrables publics et les publications des projets HORIZON.

Les liens vers les livrables et les publications des projets du 7e PC, ainsi que les liens vers certains types de résultats spécifiques tels que les jeux de données et les logiciels, sont récupérés dynamiquement sur OpenAIRE .

Livrables

Open Research Data Pilot (1)

Data Management Plan

Open Data associated with the empirical evaluation.

Publications

Conference proceedings (25)

A Framework for In-Vivo Testing of Mobile Applications

Auteurs: Mariano Ceccato, Davide Corradini, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orru, Paolo Tonella
Publié dans: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020, Page(s) 286-296, ISBN 978-1-7281-5778-8
Éditeur: IEEE
DOI: 10.1109/icst46399.2020.00037

An Empirical Evaluation of Mutation Operators for Deep Learning Systems

Auteurs: Gunel Jahangirova, Paolo Tonella
Publié dans: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020, Page(s) 74-84, ISBN 978-1-7281-5778-8
Éditeur: IEEE
DOI: 10.1109/icst46399.2020.00018

Repairing DNN Architecture: Are We There Yet?

Auteurs: Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella, Shin Yoo
Publié dans: Proceedings of the IEEE Conference on Software Testing, Verification and Validation, 2023, Page(s) pp. 234-245
Éditeur: IEEE
DOI: 10.1109/icst57152.2023.00030

Model-based exploration of the frontier of behaviours for deep learning system testing

Auteurs: Vincenzo Riccio, Paolo Tonella
Publié dans: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, Page(s) 876-888, ISBN 9781450370431
Éditeur: ACM
DOI: 10.1145/3368089.3409730

Evolutionary improvement of assertion oracles

Auteurs: Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè
Publié dans: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, Page(s) 1178-1189, ISBN 9781450370431
Éditeur: ACM
DOI: 10.1145/3368089.3409758

Quality Metrics and Oracles for Autonomous Vehicles Testing

Auteurs: Gunel Jahangirova, Andrea Stocco, Paolo Tonella
Publié dans: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, Page(s) 194-204, ISBN 978-1-7281-6836-4
Éditeur: IEEE
DOI: 10.1109/icst49551.2021.00030

An Empirical Study on Low- and High-Level Explanations of Deep Learning Misbehaviours

Auteurs: Tahereh Zohdinasab, Vincenzo Riccio, Paolo Tonella
Publié dans: Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2023
Éditeur: IEEE Computer Society
DOI: 10.1109/esem56168.2023.10304866

DeepHyperion: exploring the feature space of deep learning-based systems through illumination search

Auteurs: Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella
Publié dans: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, Page(s) 79-90, ISBN 9781450384599
Éditeur: ACM
DOI: 10.1145/3460319.3464811

A Review and Refinement of Surprise Adequacy

Auteurs: Michael Weiss, Rwiddhi Chakraborty, Paolo Tonella
Publié dans: 2021 IEEE/ACM Third International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest), 2021, Page(s) 17-24, ISBN 978-1-6654-4565-8
Éditeur: IEEE
DOI: 10.1109/deeptest52559.2021.00009

Run Java Applications and Test Them In-Vivo Meantime

Auteurs: Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella
Publié dans: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020, Page(s) 454-459, ISBN 978-1-7281-5778-8
Éditeur: IEEE
DOI: 10.1109/icst46399.2020.00061

Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring

Auteurs: Michael Weiss, Paolo Tonella
Publié dans: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, Page(s) 24-35, ISBN 978-1-7281-6836-4
Éditeur: IEEE
DOI: 10.1109/icst49551.2021.00015

Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification

Auteurs: Michael Weiss, Paolo Tonella
Publié dans: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, Page(s) 436-441, ISBN 978-1-7281-6836-4
Éditeur: IEEE
DOI: 10.1109/icst49551.2021.00056

Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)

Auteurs: Michael Weiss, Paolo Tonella
Publié dans: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, Page(s) pp. 139-150
Éditeur: ACM
DOI: 10.1145/3533767.3534375

DeepCrime: mutation testing of deep learning systems based on real faults

Auteurs: Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella
Publié dans: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, Page(s) 67-78, ISBN 9781450384599
Éditeur: ACM
DOI: 10.1145/3460319.3464825

DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score

Auteurs: Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella
Publié dans: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, Page(s) 355-367
Éditeur: IEEE Computer Society
DOI: 10.1109/ase51524.2021.9678764

Misbehaviour prediction for autonomous driving systems

Auteurs: Andrea Stocco, Michael Weiss, Marco Calzana, Paolo Tonella
Publié dans: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, Page(s) 359-371, ISBN 9781450371216
Éditeur: ACM
DOI: 10.1145/3377811.3380353

IFRIT: Focused Testing through Deep Reinforcement Learning

Auteurs: Andrea Romdhana, Mariano Ceccato, Alessio Merlo, Paolo Tonella
Publié dans: 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), 2022
Éditeur: IEEE
DOI: 10.1109/icst53961.2022.00013

When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study

Auteurs: Vincenzo Riccio, Paolo Tonella
Publié dans: Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023
Éditeur: IEEE Computer Society
DOI: 10.1109/icse48619.2023.00104

Toward In-Vivo Testing of Mobile Applications

Auteurs: Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orru, Paolo Tonella
Publié dans: 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 2019, Page(s) 137-143, ISBN 978-1-7281-5138-0
Éditeur: IEEE
DOI: 10.1109/issrew.2019.00063

Hypertesting of Programs: Theoretical Foundation and Automated Test Generation

Auteurs: Michele Pasqua, Mariano Ceccato, Paolo Tonella
Publié dans: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024
Éditeur: Association for Computing Machinery
DOI: 10.1145/3597503.3640323

ThirdEye: Attention Maps for Safe Autonomous Driving Systems

Auteurs: Andrea Stocco, Paulo J. Nunes, Marcelo d'Amorim, Paolo Tonella
Publié dans: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2022
Éditeur: ACM
DOI: 10.1145/3551349.3556968

DeepAtash: Focused Test Generation for Deep Learning Systems

Auteurs: Tahereh Zohdinasab, Vincenzo Riccio, Paolo Tonella
Publié dans: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023
Éditeur: ACM SIGSOFT
DOI: 10.1145/3597926.3598109

Taxonomy of real faults in deep learning systems

Auteurs: Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella
Publié dans: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, Page(s) 1110-1121, ISBN 9781450371216
Éditeur: ACM
DOI: 10.1145/3377811.3380395

Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights

Auteurs: Sajad Khatiri, Sebastiano Panichella, Paolo Tonella
Publié dans: Proceedings of the IEEE Conference on Software Testing, Verification and Validation, 2023, Page(s) pp. 281-292
Éditeur: IEEE
DOI: 10.1109/icst57152.2023.00034

GAssert: A Fully Automated Tool to Improve Assertion Oracles

Auteurs: Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezze
Publié dans: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2021, Page(s) 85-88, ISBN 978-1-6654-1219-3
Éditeur: IEEE
DOI: 10.1109/icse-companion52605.2021.00042

Peer reviewed articles (16)

Two is better than one: digital siblings to improve autonomous driving testing

Auteurs: Matteo Biagiola, Andrea Stocco, Vincenzo Riccio, Paolo Tonella
Publié dans: Empirical Software Engineering (EMSE), Numéro Vol. 29,No. 72, 2024, ISSN 1382-3256
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s10664-024-10458-4

Confidence-driven Weighted Retraining for Predicting Safety-Critical Failures in Autonomous Driving Systems

Auteurs: Andrea Stocco, Paolo Tonella
Publié dans: Journal of Software: Evolution and Process, 2021, ISSN 2047-7481
Éditeur: John Wiley and Sons Ltd
DOI: 10.1002/smr.2386

Model vs System Level Testing of Autonomous Driving Systems: A Replication and Extension Study

Auteurs: Andrea Stocco, Brian Pulfer, Paolo Tonella
Publié dans: Empirical Software Engineering, Numéro vol. 28, n. 3, 2023, ISSN 1382-3256
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s10664-023-10306-x

In vivo test and rollback of Java applications as they are

Auteurs: Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella
Publié dans: Journal of Software: Testing, Verification and Reliability (STVR), Numéro Vol. 33, No. 7, 2023, ISSN 0960-0833
Éditeur: John Wiley & Sons Inc.
DOI: 10.1002/stvr.1857

Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems

Auteurs: Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella
Publié dans: ACM Transactions on Software Engineering and Methodology, Numéro vol. 32, n. 2, 2023, Page(s) 1-38, ISSN 1049-331X
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3544792

Assessing the security of inter-app communications in android through reinforcement learning

Auteurs: Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella
Publié dans: Computer Security, Numéro vol. 131, 2023, ISSN 0167-4048
Éditeur: Pergamon Press Ltd.
DOI: 10.1016/j.cose.2023.103311

Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines

Auteurs: Michael Weiss, Paolo Tonella
Publié dans: Journal of Software: Testing, Verification and Reliability, 2023, Page(s) 1-23, ISSN 1099-1689
Éditeur: John Wiley & Sons
DOI: 10.1002/stvr.1840

Testing the Plasticity of Reinforcement Learning-based Systems

Auteurs: Matteo Biagiola, Paolo Tonella
Publié dans: ACM Transactions on Software Engineering and Methodology, Numéro vol. 31, n. 4, 2022, Page(s) 1-46, ISSN 1049-331X
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3511701

Testing machine learning based systems: a systematic mapping

Auteurs: Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, Paolo Tonella
Publié dans: Empirical Software Engineering, Numéro 25/6, 2020, Page(s) 5193-5254, ISSN 1382-3256
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s10664-020-09881-0

Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks

Auteurs: Michael Weiss, Paolo Tonella
Publié dans: ACM Transactions on Software Engineering and Methodology (TOSEM), Numéro Vol.33, No. 1, 2024, ISSN 1049-331X
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3617593

Testing of Deep Reinforcement Learning Agents with Surrogate Models

Auteurs: Matteo Biagiola, Paolo Tonella
Publié dans: ACM Transactions on Software Engineering and Methodology (TOSEM), Numéro Vol. 33, No. 3, 2024, ISSN 1049-331X
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3631970

An Empirical Validation of Oracle Improvement

Auteurs: Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella
Publié dans: IEEE Transactions on Software Engineering, 2019, Page(s) 1-1, ISSN 0098-5589
Éditeur: Institute of Electrical and Electronics Engineers
DOI: 10.1109/tse.2019.2934409

A Survey of Field-based Testing Techniques

Auteurs: Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, Paolo Tonella
Publié dans: ACM Computing Surveys, Numéro 54/5, 2021, Page(s) 1-39, ISSN 0360-0300
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3447240

Generating and detecting true ambiguity: a forgotten danger in DNN supervision testing

Auteurs: Michael Weiss, André García Gómez, Paolo Tonella
Publié dans: Empirical Software Engineering (EMSE), Numéro Vol. 28,No. 146, 2023, ISSN 1382-3256
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s10664-023-10393-w

Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems

Auteurs: Andrea Stocco, Brian Pulfer, Paolo Tonella
Publié dans: IEEE Transactions on Software Engineering, Numéro vol. 49, n. 4, 2023, Page(s) pp. 1928-1940, ISSN 0098-5589
Éditeur: Institute of Electrical and Electronics Engineers
DOI: 10.1109/tse.2022.3202311

Deep Reinforcement Learning for Black-Box Testing of Android Apps

Auteurs: Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella
Publié dans: ACM Transactions on Software Engineering and Methodology, 2022, ISSN 1049-331X
Éditeur: Association for Computing Machinary, Inc.
DOI: 10.1145/3502868