Final Report Summary - PREDICTMP (predictive techniques for system level analysis of multiprocessors)
European competitiveness in microelectronics, such as wireless, networking, telecommunications, automobile, and consumer products relies heavily on improving the reliability of multiprocessor systems. Current electronic design automation (EDA) tools, which enable the design, verification and production of such multiprocessor systems, are incapable of handling and managing concurrency introduced due to interaction of multiple components as in the emerging multi-processor system-on-chip (MPSoC)s. Design and correctness of such systems require new electronic design automation techniques that are able to find concurrency related errors. This research has great impact on competitiveness of Europe by developing solutions that can make products more reliable. Proper solutions to deal with verification challenges of MPSoCs will allow European companies to achieve world-leading positions in computing solutions and products. Such complex systems are present in embedded systems; hence our solution will not only have impact on competitiveness of European information and communication (IC) industry, but also on competitiveness of European strongholds such as automotive, avionics, consumer electronics, telecoms and medical systems.
Raising the level of abstraction via system level design is one of the most efficient methods of reducing complexity due to concurrent multiprocessor designs. Increasing design costs and lost time are also pushing design, analysis, and verification to system level rather than the traditional microelectronics implementation levels such as register transfer level. System level design methods and tools make it easier to diagnose concurrency and protocol problems, whereas these problems are hidden at the lower implementation levels. System level modelling also allows early model generation, performance, power analysis, and architectural exploration of MPSoCs. SystemC is the most popular system level modelling language used for designing SoCs in the industry. However, system level design coupled with multi-processors is a challenge to EDA tools providers, and system level tools are still at their infancy. New system level techniques of analysis are required to improve reliability of concurrent software running on concurrent hardware.
To address reliability challenges of concurrent multiprocessors, a predictive runtime verification (PRV) technique was developed for SystemC designs. Design verification is the task of establishing that a given design accurately implements the intended functional behaviour. Design verification has grown to dominate the cost of electronic system design; however, designs continue to be released with latent bugs.
Our technique is based on the simulation of system level designs rather than a static and costly analysis of these designs as in formal verification. In PRV, the assertions (specifications) are monitored during the execution of the system. Concurrent interaction of multi-processor systems results in errors which are difficult to find. PRV differs from traditional verification techniques because it is a predictive verification technique, where we can predict potential and difficult to find concurrency errors using error-free executions. We developed scalable PRV techniques for SystemC and experimented with various designs.
Another objective of this project was to develop concurrency oriented coverage metrics for system level models of multiprocessor systems. Even if design verification is successfully completed as above, there is still a doubt whether specifications (or tests) are comprehensive and if they cover all possible behaviours of the system. Increasingly, there is a need to measure the quality of verification effort. In this project, we developed a novel concurrency aware mutation testing based coverage metric. Specifically, we developed a mutation library for concurrent SystemC designs to inject functional faults similar to the successful stuck-at fault model for manufacturing faults. Our mutation library considers the complete list of concurrency operators in SystemC and we showed the effectiveness of this mutation library by relating the mutations to actual bug patterns. We developed a novel concurrent mutation coverage metric using multiple schedules of a concurrent program that allows us to adequately measure the coverage for concurrent programmes. We performed experiments with various system level multiprocessor designs including a large industrial design to validate the effectiveness of our techniques. Our experimental results confirm the inadequacy of current verification test suites for checking concurrent features of SystemC and demonstrate the effectiveness of mutation testing based coverage metrics.
Testbench development is a major component of simulation based verification, which is the de-facto verification technique used in the industry. We developed an automated test generation framework for SystemC designs using the coverage techniques described above. This framework further extended the usability of our techniques in practice.
We also developed an integration framework between system level models of electronic designs and real electronic devices. As the complexity of industrial electronic systems increase, a rapid system level modelling and simulation environment is required to reduce the design and verification time. In traditional system level modelling, all components need to be modelled. However, modelling has no added value for components that are already physically implemented. Therefore, techniques have to be developed for incorporating real devices with virtual (system level) models. There is no established mechanism that provides communication between real devices and virtual models and this project fills this gap. Specifically, we developed synchronisation mechanisms between SystemC models and real devices so as to achieve real-time communication. We devised a coherent way of matching simulation time with real time execution using a novel hybrid channel concept. We developed mechanisms for connecting concurrent real devices to a set of virtual models and defined techniques that allow virtual models to receive inputs from real devices. Finally, we validated the effectiveness of our framework in industrial systems that employ industrial standards such as BACNet and Ethernet.
The outcomes of this project have been published in book chapters, refereed journals, conferences and workshops. Specifically, 1 book chapter, 1 conference proceedings book, 5 journal papers and 16 conference and workshop papers have been published as well as a patent application has been submitted.
The results of our research are very promising for improving reliability of multiprocessor systems.
Novel system level verification and coverage works have improved the state of the art by finding potential errors in the systems and accurately measuring the quality of verification. System level error diagnosis techniques will be developed to reduce debugging time and ultimately time-to-market. New research and tools will be developed with a focus on automation, integration and scalability while performing experiments on industrial applications. With the completion of this project, verification engineers will not only find actual and potential errors in a system composed of hardware and software described in SystemC but also will automatically direct the system to critical scenarios where coverage is low, therefore find corner cases and reveal hard to find errors. In this project, we developed techniques and related algorithms and tools that can be applied in industrial design verification.
Raising the level of abstraction via system level design is one of the most efficient methods of reducing complexity due to concurrent multiprocessor designs. Increasing design costs and lost time are also pushing design, analysis, and verification to system level rather than the traditional microelectronics implementation levels such as register transfer level. System level design methods and tools make it easier to diagnose concurrency and protocol problems, whereas these problems are hidden at the lower implementation levels. System level modelling also allows early model generation, performance, power analysis, and architectural exploration of MPSoCs. SystemC is the most popular system level modelling language used for designing SoCs in the industry. However, system level design coupled with multi-processors is a challenge to EDA tools providers, and system level tools are still at their infancy. New system level techniques of analysis are required to improve reliability of concurrent software running on concurrent hardware.
To address reliability challenges of concurrent multiprocessors, a predictive runtime verification (PRV) technique was developed for SystemC designs. Design verification is the task of establishing that a given design accurately implements the intended functional behaviour. Design verification has grown to dominate the cost of electronic system design; however, designs continue to be released with latent bugs.
Our technique is based on the simulation of system level designs rather than a static and costly analysis of these designs as in formal verification. In PRV, the assertions (specifications) are monitored during the execution of the system. Concurrent interaction of multi-processor systems results in errors which are difficult to find. PRV differs from traditional verification techniques because it is a predictive verification technique, where we can predict potential and difficult to find concurrency errors using error-free executions. We developed scalable PRV techniques for SystemC and experimented with various designs.
Another objective of this project was to develop concurrency oriented coverage metrics for system level models of multiprocessor systems. Even if design verification is successfully completed as above, there is still a doubt whether specifications (or tests) are comprehensive and if they cover all possible behaviours of the system. Increasingly, there is a need to measure the quality of verification effort. In this project, we developed a novel concurrency aware mutation testing based coverage metric. Specifically, we developed a mutation library for concurrent SystemC designs to inject functional faults similar to the successful stuck-at fault model for manufacturing faults. Our mutation library considers the complete list of concurrency operators in SystemC and we showed the effectiveness of this mutation library by relating the mutations to actual bug patterns. We developed a novel concurrent mutation coverage metric using multiple schedules of a concurrent program that allows us to adequately measure the coverage for concurrent programmes. We performed experiments with various system level multiprocessor designs including a large industrial design to validate the effectiveness of our techniques. Our experimental results confirm the inadequacy of current verification test suites for checking concurrent features of SystemC and demonstrate the effectiveness of mutation testing based coverage metrics.
Testbench development is a major component of simulation based verification, which is the de-facto verification technique used in the industry. We developed an automated test generation framework for SystemC designs using the coverage techniques described above. This framework further extended the usability of our techniques in practice.
We also developed an integration framework between system level models of electronic designs and real electronic devices. As the complexity of industrial electronic systems increase, a rapid system level modelling and simulation environment is required to reduce the design and verification time. In traditional system level modelling, all components need to be modelled. However, modelling has no added value for components that are already physically implemented. Therefore, techniques have to be developed for incorporating real devices with virtual (system level) models. There is no established mechanism that provides communication between real devices and virtual models and this project fills this gap. Specifically, we developed synchronisation mechanisms between SystemC models and real devices so as to achieve real-time communication. We devised a coherent way of matching simulation time with real time execution using a novel hybrid channel concept. We developed mechanisms for connecting concurrent real devices to a set of virtual models and defined techniques that allow virtual models to receive inputs from real devices. Finally, we validated the effectiveness of our framework in industrial systems that employ industrial standards such as BACNet and Ethernet.
The outcomes of this project have been published in book chapters, refereed journals, conferences and workshops. Specifically, 1 book chapter, 1 conference proceedings book, 5 journal papers and 16 conference and workshop papers have been published as well as a patent application has been submitted.
The results of our research are very promising for improving reliability of multiprocessor systems.
Novel system level verification and coverage works have improved the state of the art by finding potential errors in the systems and accurately measuring the quality of verification. System level error diagnosis techniques will be developed to reduce debugging time and ultimately time-to-market. New research and tools will be developed with a focus on automation, integration and scalability while performing experiments on industrial applications. With the completion of this project, verification engineers will not only find actual and potential errors in a system composed of hardware and software described in SystemC but also will automatically direct the system to critical scenarios where coverage is low, therefore find corner cases and reveal hard to find errors. In this project, we developed techniques and related algorithms and tools that can be applied in industrial design verification.