Periodic Reporting for period 2 - VAMOS (Vigilant Algorithmic Monitoring of Software)
Okres sprawozdawczy: 2023-07-01 do 2024-12-31
Software use has become ubiquitous. Software plays a critical role in essentially every domain of human activity from transportation to finance, from healthcare to entertainment, from manufacturing to security.
Software has always come with vulnerabilities, ranging from design and implementation errors, to unintended and unexpected use cases, all the way to openings for malicious attacks. The more complex software gets, the more difficult it is to keep the software reliable, safe, unbiased, trusted, and secure. All of these issues are exacerbated by the rapid growth of artificial intelligence software.
The VAMOS project proposes to use a part of the available hardware infrastructure to monitor the software that runs on the other part. In this way, all critical software could be monitored. The monitoring software (“software watchdogs”) issues warning signals whenever the monitored software misbehaves, e.g. when it comes dangerously close to a failure, or under attack, or makes unsafe or unfair decisions.
To make the VAMOS vision a reality, the project builds (1) the theoretical foundations for monitoring software that can make quantitative judgments, (2) algorithms for specific monitoring applications such as measuring the safety and fairness of artificial intelligence software, and (3) a prototype middleware infrastructure to test these ideas.
Software has always come with vulnerabilities, ranging from design and implementation errors, to unintended and unexpected use cases, all the way to openings for malicious attacks. The more complex software gets, the more difficult it is to keep the software reliable, safe, unbiased, trusted, and secure. All of these issues are exacerbated by the rapid growth of artificial intelligence software.
The VAMOS project proposes to use a part of the available hardware infrastructure to monitor the software that runs on the other part. In this way, all critical software could be monitored. The monitoring software (“software watchdogs”) issues warning signals whenever the monitored software misbehaves, e.g. when it comes dangerously close to a failure, or under attack, or makes unsafe or unfair decisions.
To make the VAMOS vision a reality, the project builds (1) the theoretical foundations for monitoring software that can make quantitative judgments, (2) algorithms for specific monitoring applications such as measuring the safety and fairness of artificial intelligence software, and (3) a prototype middleware infrastructure to test these ideas.
We have made progress in all three directions and mention a few highlights.
As part of the theoretical foundations, we have developed a theory of quantitative safety and liveness based on monitoring. Safety and liveness are elementary concepts of computer science, which lie at the heart of most verification paradigms. The safety-liveness classification of boolean properties characterizes whether a given property can be falsified by observing a finite prefix of an infinite computation trace (always for safety, never for liveness). In quantitative specification and verification, properties assign not truth values, but quantitative values to computation traces (e.g. a cost, or the distance to a boolean property). We introduce quantitative safety and liveness and show that every quantitative property can be written as the pointwise minimum of a quantitative safety property and a quantitative liveness property. Consequently, like boolean properties, also quantitative properties - and their monitors - can be decomposed into safety and liveness parts. Moreover, quantitative properties can be approximated naturally. We prove that every quantitative property that has both safe and co-safe approximations can be monitored arbitrarily precisely by a monitor that uses only a finite amount of memory. [FOSSACS 2023]
As part of the monitoring applications, we have developed algorithms for monitoring the degree of bias in decision making software. We assume that the monitored system is unknown, but has a Markov chain structure. We introduce a specification language that can model common algorithmic fairness properties such as demographic parity, equal opportunity, and social burden. Our algorithms observe a long sequence of events as generated by the monitored system, and produce, after each observation, a quantitative estimate of how fair or biased the system was on that execution trace up to that point in time. The estimate is proven to be correct modulo an error bound and confidence level. Our monitors are of two types that use, respectively, frequentist and Bayesian statistical inference techniques. [CAV 2023]
As part of the envisioned middleware infrastructure, we have built a first prototype, called VAMOS (like the project), for the runtime monitoring of software which is designed specifically to support third-party and best-effort scenarios. In a third-party scenario, the monitoring software can observe only the inputs and outputs of the monitored software. In a best-effort scenario, the monitoring software must drop some observed events in order to keep up with the monitored software. The design goals of the VAMOS middleware are (i) efficiency (low overhead), (ii) flexibility (the connectability of monitors to different event sources and specification languages), and (iii) ease-of-use. [FASE 2023]
As part of the theoretical foundations, we have developed a theory of quantitative safety and liveness based on monitoring. Safety and liveness are elementary concepts of computer science, which lie at the heart of most verification paradigms. The safety-liveness classification of boolean properties characterizes whether a given property can be falsified by observing a finite prefix of an infinite computation trace (always for safety, never for liveness). In quantitative specification and verification, properties assign not truth values, but quantitative values to computation traces (e.g. a cost, or the distance to a boolean property). We introduce quantitative safety and liveness and show that every quantitative property can be written as the pointwise minimum of a quantitative safety property and a quantitative liveness property. Consequently, like boolean properties, also quantitative properties - and their monitors - can be decomposed into safety and liveness parts. Moreover, quantitative properties can be approximated naturally. We prove that every quantitative property that has both safe and co-safe approximations can be monitored arbitrarily precisely by a monitor that uses only a finite amount of memory. [FOSSACS 2023]
As part of the monitoring applications, we have developed algorithms for monitoring the degree of bias in decision making software. We assume that the monitored system is unknown, but has a Markov chain structure. We introduce a specification language that can model common algorithmic fairness properties such as demographic parity, equal opportunity, and social burden. Our algorithms observe a long sequence of events as generated by the monitored system, and produce, after each observation, a quantitative estimate of how fair or biased the system was on that execution trace up to that point in time. The estimate is proven to be correct modulo an error bound and confidence level. Our monitors are of two types that use, respectively, frequentist and Bayesian statistical inference techniques. [CAV 2023]
As part of the envisioned middleware infrastructure, we have built a first prototype, called VAMOS (like the project), for the runtime monitoring of software which is designed specifically to support third-party and best-effort scenarios. In a third-party scenario, the monitoring software can observe only the inputs and outputs of the monitored software. In a best-effort scenario, the monitoring software must drop some observed events in order to keep up with the monitored software. The design goals of the VAMOS middleware are (i) efficiency (low overhead), (ii) flexibility (the connectability of monitors to different event sources and specification languages), and (iii) ease-of-use. [FASE 2023]
We were the first to define and study a quantitative theory of monitoring, develop algorithms for monitoring fairness quantitatively, and build prototype middleware for third-party best-effort monitoring. We intend to continue the project along all the directions: theory, applications, and implementation.