Periodic Reporting for period 4 - FOVEDIS (Formal specification and verification of distributed data structures)
Berichtszeitraum: 2020-11-01 bis 2021-10-31
The future of the computing technology relies on fast access, transformation, and exchange of data across large-scale networks such as the Internet. The design of software systems that support high-frequency parallel accesses to high-quantity data is a fundamental challenge. As more scalable alternatives to traditional relational databases, distributed data structures (DDSs) are at the basis of a wide range of automated services, for now, and for the foreseeable future.
This project aims to improve our understanding of the theoretical foundations of DDSs. The design and the usage of DDSs are based on new principles, which require rigorous engineering methodologies. Specifically, we require design procedures based on precise specifications, and automated reasoning techniques for enhancing the reliability of the engineering process.
The targeted breakthrough of this project is developing automated formal methods for rigorous engineering of DDSs. A first objective is to define coherent formal specifications that provide precise requirements at design time and explicit guarantees during their usage. Then, we will investigate practical programming principles, compatible with these specifications, for building applications that use DDSs. Finally, we will develop efficient automated reasoning techniques for debugging or validating DDS implementations against their specifications. The principles underlying automated reasoning are also important for identifying best practices in the design of these complex systems to increase confidence in their correctness. The developed methodologies based on formal specifications will thus benefit both the conception and automated validation of DDS implementations and the applications that use them.
The project succeeded in addressing all of the issues mentioned above. It introduced various frameworks to specify DDSs, investigated the theoretical limits of checking conformance to these specifications automatically along with pragmatic approaches based on suitable heuristics. For applications built on top of DDSs, it proposed a concept of robustness against weakening the guarantees of the underlying DDS and methodologies for establishing it. This concept makes it possible to retain the reliability of an application while using DDSs with bigger throughput that increase its performance. These results form the basis of new engineering methodologies that have a wide spectrum of applications and can help improving the reliability of software systems relying on DDSs.
This project aims to improve our understanding of the theoretical foundations of DDSs. The design and the usage of DDSs are based on new principles, which require rigorous engineering methodologies. Specifically, we require design procedures based on precise specifications, and automated reasoning techniques for enhancing the reliability of the engineering process.
The targeted breakthrough of this project is developing automated formal methods for rigorous engineering of DDSs. A first objective is to define coherent formal specifications that provide precise requirements at design time and explicit guarantees during their usage. Then, we will investigate practical programming principles, compatible with these specifications, for building applications that use DDSs. Finally, we will develop efficient automated reasoning techniques for debugging or validating DDS implementations against their specifications. The principles underlying automated reasoning are also important for identifying best practices in the design of these complex systems to increase confidence in their correctness. The developed methodologies based on formal specifications will thus benefit both the conception and automated validation of DDS implementations and the applications that use them.
The project succeeded in addressing all of the issues mentioned above. It introduced various frameworks to specify DDSs, investigated the theoretical limits of checking conformance to these specifications automatically along with pragmatic approaches based on suitable heuristics. For applications built on top of DDSs, it proposed a concept of robustness against weakening the guarantees of the underlying DDS and methodologies for establishing it. This concept makes it possible to retain the reliability of an application while using DDSs with bigger throughput that increase its performance. These results form the basis of new engineering methodologies that have a wide spectrum of applications and can help improving the reliability of software systems relying on DDSs.
The project investigated the foundational as well as the practical issues related to the specification and the automated verification of distributed data structures (DDS) and applications built on top of them.
To ensure persistence and availability of data, the state of a DDS is replicated at multiple sites in the network and the effect of concurrent operations is described by various notions of weak consistency. The project formalized various notions of weak consistency used in practice, e.g. variations of Causal Consistency, “heterogenous” consistency criteria which differ from one operation of a DDS to another, a widely used class of DDSs called Conflict-Free Replicated Data Types (CRDTs), isolation levels used in the context of transactional systems, etc.
It investigated the theoretical limits of checking conformance to such consistency criteria either using testing or in an exhaustive manner using standard model checking techniques. It studied the asymptotic complexity of checking whether a given execution conforms to some given consistency criterion, which is relevant in the context of testing, and several classes of DDSs and consistency criteria whose verification can be reduced to classic model checking problems like reachability or the existence of forward simulations.
Guided by these theoretical limits, it developed pragmatic approaches based on suitable heuristics for proving or disproving the correctness of DDS implementations. It introduced approximated methods for testing which are not complete in theory (may miss violations) but quite precise in practice, and various user-guided methodologies for establishing correctness in an exhaustive manner.
For applications built on top of DDSs, the project proposed algorithmic methods for checking if such an application admits precisely the same behaviors when running under a weak consistency criterion instead of a stronger one. This can simplify the proofs of their correctness since reasoning about stronger consistency criteria is generally easier and it makes it possible to retain correctness while running under a more efficient DDS that guarantees a weaker notion of consistency.
These results were presented in 32 articles that were published in top venues on programming languages, mechanized verification, concurrency theory, or distributed computing. They have been disseminated in a large number of invited talks in international workshops or conferences.
To ensure persistence and availability of data, the state of a DDS is replicated at multiple sites in the network and the effect of concurrent operations is described by various notions of weak consistency. The project formalized various notions of weak consistency used in practice, e.g. variations of Causal Consistency, “heterogenous” consistency criteria which differ from one operation of a DDS to another, a widely used class of DDSs called Conflict-Free Replicated Data Types (CRDTs), isolation levels used in the context of transactional systems, etc.
It investigated the theoretical limits of checking conformance to such consistency criteria either using testing or in an exhaustive manner using standard model checking techniques. It studied the asymptotic complexity of checking whether a given execution conforms to some given consistency criterion, which is relevant in the context of testing, and several classes of DDSs and consistency criteria whose verification can be reduced to classic model checking problems like reachability or the existence of forward simulations.
Guided by these theoretical limits, it developed pragmatic approaches based on suitable heuristics for proving or disproving the correctness of DDS implementations. It introduced approximated methods for testing which are not complete in theory (may miss violations) but quite precise in practice, and various user-guided methodologies for establishing correctness in an exhaustive manner.
For applications built on top of DDSs, the project proposed algorithmic methods for checking if such an application admits precisely the same behaviors when running under a weak consistency criterion instead of a stronger one. This can simplify the proofs of their correctness since reasoning about stronger consistency criteria is generally easier and it makes it possible to retain correctness while running under a more efficient DDS that guarantees a weaker notion of consistency.
These results were presented in 32 articles that were published in top venues on programming languages, mechanized verification, concurrency theory, or distributed computing. They have been disseminated in a large number of invited talks in international workshops or conferences.