Periodic Reporting for period 4 - Mathador (Type and Proof Structures for Concurrent Software Verification)
Reporting period: 2021-10-01 to 2023-03-31
The key goal of the project has been to design a theory of types for fine-grained shared-memory concurrent programs. Such a type theory is a system in which one can write concurrent programs and package them with their correctness proofs into an encapsulated whole, so that the types serve as interfaces that hide the internal properties of the programs as well as proofs. Such a whole, packaging a program with its correctness proof (i.e. a certified program) can be used as a black box to facilitate scaling of verification. An expert can write a proof of an algorithm, and once the proof is hidden by the types, the user of the algorithm never needs to understand it.
For the encapsulation to work, however, the specifications and proofs must be written compositionally, so that they aren't invalidated once the encapsulation is considered in a bigger context. And indeed, type theory is widely recognized as a theory of composition in logics and programming languages. Up to now, however, type theory has mostly been applied to the purely functional programming model. The goal of this project has been to apply it to the stateful and concurrent programming model (i.e. to shared-memory concurrency).
This is of essence because it is precisely in concurrency where the need for compositional reasoning is most required, and is a well-recognized and perennial problem, often considered a great challenge. The failure of composition is witnessed by the exponential combinatorial explosion in the size and number of proof obligations that the verifier must discharge. In many prior works, the problem has been attacked by developing sophisticated methods for automating the discharge of the obligations. In this project, we instead encapsulate and abstract programs and proofs together into mathematical structures that manage the proof obligations during verification to avoid this combinatorial explosion.
The project has made considerable advances in this direction, such as: (1) Formulation of a mathematical theory of stateful resources and their morphisms, and (2) Formulation of a mathematical theory of partial commutative monoids (PCMs) and their morphisms. We have put these structures to work in practice by: (3) Connecting them to linearizability, a standard concurrent consistency criteria; (4) Developing and mechanizing algebraic constructions for building linearizable objects while supporting Hoare-style reasoning over object methods, and (5) By constructing novel ways of organizing linearizability proofs that are declarative and compositional in nature.
For the encapsulation to work, however, the specifications and proofs must be written compositionally, so that they aren't invalidated once the encapsulation is considered in a bigger context. And indeed, type theory is widely recognized as a theory of composition in logics and programming languages. Up to now, however, type theory has mostly been applied to the purely functional programming model. The goal of this project has been to apply it to the stateful and concurrent programming model (i.e. to shared-memory concurrency).
This is of essence because it is precisely in concurrency where the need for compositional reasoning is most required, and is a well-recognized and perennial problem, often considered a great challenge. The failure of composition is witnessed by the exponential combinatorial explosion in the size and number of proof obligations that the verifier must discharge. In many prior works, the problem has been attacked by developing sophisticated methods for automating the discharge of the obligations. In this project, we instead encapsulate and abstract programs and proofs together into mathematical structures that manage the proof obligations during verification to avoid this combinatorial explosion.
The project has made considerable advances in this direction, such as: (1) Formulation of a mathematical theory of stateful resources and their morphisms, and (2) Formulation of a mathematical theory of partial commutative monoids (PCMs) and their morphisms. We have put these structures to work in practice by: (3) Connecting them to linearizability, a standard concurrent consistency criteria; (4) Developing and mechanizing algebraic constructions for building linearizable objects while supporting Hoare-style reasoning over object methods, and (5) By constructing novel ways of organizing linearizability proofs that are declarative and compositional in nature.
We developed the following abstractions, and employed them in our type-based concurrent model.
- Resources; which are a special form of state transition systems that we use as part of types of concurrent programs.
- Resource morphisms; which are a notion of function types. A resource morphism transports a program verified over domain resource to the range resource.
- Resource simulations; used to reason about morphed programs. The simulations feature in a novel inference rule that typechecks an application of a resource morphism to a program.
- Partial Commutative Monoids. PCMs have been used in prior work to formalize ghost state in separation logic. We developed the associated notions of PCM morphisms and associated separating relations that serve as the morphism preconditions.
- Linearizable resources; we combined the theories of resources, PCMs, and their morphism to formalize linearizability. We also developed special mathematical constructions that form the beginning of the algebra of linearizable resources.
- Visibility relations. We observed that the most common way linearizability proofs are given today is operational, as it involves describing the line of code that constitutes the linearization points (LP) for each method of the verified structure. Being operational, the approach lacks power of mathematical abstraction. We have thus considered a declarative way of proving linearizability by axiomatizing the dependence (i.e. visibility) relations between the methods. We developed a generic and systematic approach to obtain declarative concurrent axiomatizations starting from sequential ones. We applied it to verify challenging concurrent structures: RDCSS and MCAS, timestamped stack, and three Janati snapshot algorithms. In all cases, we obtained high-level proofs that apply to several different implementations.
- In a cross-over application, we considered how our stateful type theory can be generalized to an abstract notion of algebraic effects. We developed a modal type system, where types keep track of the signature for the algebraic effect. We built on the prior work on Contextual Modal Type Theory (CMTT), thereby establishing a surprising and useful connection between modal type theories and algebraic effects. The connection enabled us to transfer an idea that existed in the field of modal types, and apply it to solve one of they key challenges in algebraic effects.
The development of resources fits in the WP1 of the proposal. The development of resource morphisms and simulations is in WP2. A paper describing these notions in detail has been published at the OOPSLA 2019 conference.
The development of the algebra of PCMs is in WP3. A paper describing PCM algebra has been published at POPL 2021 conference.
The work on declarative proofs of linearizability is in WP4, and has been published at ECOOP 2017, OOPSLA 2020, and POPL 2022 conferences.
The work on algebraic effects is in WP5, and has been published at ICFP 2021.
The work on the systematic axiomatization of visibility straddles WP3 and WP4, and is currently in submission to CONCUR 2023
The algebra of linearizable resources is in WP2 and WP3. It is currently available as a library mechanized in Coq, but has not yet been prepared for publication.
We have applied the above mathematics to formally verify some representative and challenging benchmark examples such as: concurrent stack data structures, several different variant of locks including a flat combiner structure, some non-linearizable data structures, and concurrent graph structures. All of these have been verified formally in our extension of the Coq theorem prover with concurrency, and we have made the software artifacts publicly available. All of the mechanization work fits into WP6.
For example, the library for formally reasoning about PCMs has been released as a stand-alone package, as it has applications beyond our project: https://sympa.inria.fr/sympa/arc/coq-club/2018-04/msg00111.html
The meta theory behind the designed system, as well as all the examples, are available on-line at the project's public website.
- Resources; which are a special form of state transition systems that we use as part of types of concurrent programs.
- Resource morphisms; which are a notion of function types. A resource morphism transports a program verified over domain resource to the range resource.
- Resource simulations; used to reason about morphed programs. The simulations feature in a novel inference rule that typechecks an application of a resource morphism to a program.
- Partial Commutative Monoids. PCMs have been used in prior work to formalize ghost state in separation logic. We developed the associated notions of PCM morphisms and associated separating relations that serve as the morphism preconditions.
- Linearizable resources; we combined the theories of resources, PCMs, and their morphism to formalize linearizability. We also developed special mathematical constructions that form the beginning of the algebra of linearizable resources.
- Visibility relations. We observed that the most common way linearizability proofs are given today is operational, as it involves describing the line of code that constitutes the linearization points (LP) for each method of the verified structure. Being operational, the approach lacks power of mathematical abstraction. We have thus considered a declarative way of proving linearizability by axiomatizing the dependence (i.e. visibility) relations between the methods. We developed a generic and systematic approach to obtain declarative concurrent axiomatizations starting from sequential ones. We applied it to verify challenging concurrent structures: RDCSS and MCAS, timestamped stack, and three Janati snapshot algorithms. In all cases, we obtained high-level proofs that apply to several different implementations.
- In a cross-over application, we considered how our stateful type theory can be generalized to an abstract notion of algebraic effects. We developed a modal type system, where types keep track of the signature for the algebraic effect. We built on the prior work on Contextual Modal Type Theory (CMTT), thereby establishing a surprising and useful connection between modal type theories and algebraic effects. The connection enabled us to transfer an idea that existed in the field of modal types, and apply it to solve one of they key challenges in algebraic effects.
The development of resources fits in the WP1 of the proposal. The development of resource morphisms and simulations is in WP2. A paper describing these notions in detail has been published at the OOPSLA 2019 conference.
The development of the algebra of PCMs is in WP3. A paper describing PCM algebra has been published at POPL 2021 conference.
The work on declarative proofs of linearizability is in WP4, and has been published at ECOOP 2017, OOPSLA 2020, and POPL 2022 conferences.
The work on algebraic effects is in WP5, and has been published at ICFP 2021.
The work on the systematic axiomatization of visibility straddles WP3 and WP4, and is currently in submission to CONCUR 2023
The algebra of linearizable resources is in WP2 and WP3. It is currently available as a library mechanized in Coq, but has not yet been prepared for publication.
We have applied the above mathematics to formally verify some representative and challenging benchmark examples such as: concurrent stack data structures, several different variant of locks including a flat combiner structure, some non-linearizable data structures, and concurrent graph structures. All of these have been verified formally in our extension of the Coq theorem prover with concurrency, and we have made the software artifacts publicly available. All of the mechanization work fits into WP6.
For example, the library for formally reasoning about PCMs has been released as a stand-alone package, as it has applications beyond our project: https://sympa.inria.fr/sympa/arc/coq-club/2018-04/msg00111.html
The meta theory behind the designed system, as well as all the examples, are available on-line at the project's public website.