Software is pervasive in modern society, and its reliability is critical.
Program analysis refers to automatically examining a piece of software with the goal of detecting correctness issues or verifying their absence. Program analyzers are tools that implement program-analysis techniques. In recent years, as day-to-day life increasingly depends on software, more and more program analyzers are being built and used in practice. In fact, there is an abundance of popular analyzers developed both in academia and industry.
We rely on program analyzers to “guard” software reliability, but who will guard the guards? Program analyzers are highly complex tools, implementing sophisticated algorithms and performance optimizations. In addition, analyzers typically integrate several self-contained, core analysis components, such as specialized solvers, which are already complex by themselves. Due to this overall complexity, program analyzers are all the more likely to contain correctness issues. The most dangerous kind of correctness issue in analyzers is a critical bug, which we define as a bug leading to a wrong response, e.g. returning ‘correct’ for incorrect software, or leading to a right response for the wrong reasons. The latter type of critical bug is relevant because it is also likely to result in a wrong analyzer response under different circumstances.
In modern society, critical bugs may have disastrous consequences, e.g. when analyzing software used for transportation, banking, or secure communication. As a concrete example, consider the Astrée analyzer, which has been used to verify the absence of runtime errors in the flight-control software of Airbus A340 and A380. What if it missed an error? It is, therefore, imperative to check program analyzers for critical bugs.
Verifying the absence of critical bugs in a program analyzer is prohibitively expensive. Contrary to verification, automated test generation can be used to effectively find such bugs. Existing testing approaches, however, are still limited for this application domain.
The goal of this project is to develop an overarching methodology for more rigorous testing of program analyzers than ever before. The key idea is to first expose more information about why a program analyzer reaches a particular response for a certain piece of code, e.g. why was the code found correct? This information will be used to interrogate the analyzer further aiming to force it into a contradiction. In other words, anything the analyzer says during interrogation can and will be used against it. Finding a contradiction signifies that an analyzer response or its justification for a response is wrong, and that a critical bug has been detected. We call this methodology “interrogation testing”.
If successful, this project will enable systematic testing of entire program-analyzer classes. As a result, analyzers will exhibit fewer critical bugs, potentially preventing catastrophic outcomes in safety-critical domains.
