Skip to main content

Dependability Benchmarking

Objective

The objective of DBench is to define and validate dependability benchmarks for largely deployed computer systems. We consider specifically COTS and COTS-based systems, due to their widespread use in this kind of systems.

Four major steps are planned:
1) definition of the conceptual framework for system benchmarking;
2) identification and evaluation of the enabling technologies;
3) specific benchmark definition and application to pilot experiments in order to develop, experiment and validate the benchmark prototypes and;
4) consolidation of the conceptual framework with the experimental results.

The pilots are central in assessing and supporting the results of the development of the enabling technologies and of the benchmark guidelines that will be provided at the end of the project with the benchmark prototypes. Two major application areas are considered, embedded systems and transactional applications, and the benchmark prototypes will be implemented in two COTS OS (Windows and Linux).

Objectives:
The DBench project will define a conceptual framework and an experimental environment for benchmarking the dependability of COTS and COTS-based systems.

It will provide system developers and end-users with means for:
1) characterising and evaluating the dependability of a component or a system;
2) identifying malfunctioning or less weak parts, requiring more attention;
3) tuning a particular component to enhance its dependability and;
4) comparing the dependability of alternative or competing solutions.

The two final measurable objectives that will be achieved at the end of the project are a report presenting the concepts, specifications and guidelines for dependability benchmarking and a set of dependability benchmark prototype tools. The prototypes will be made widely available (e.g. through the web whenever possible) to promote their adoption by an audience as wide as possible.

Work description:
The work will be performed in four steps:
1) definition of the conceptual framework for system benchmarking;
2) identification and evaluation of the enabling technologies;
3) specific benchmark definition and application to pilot experiments in order to develop, experiment and validate the benchmark prototypes and;
4) consolidation of the conceptual framework with the experimental results.

The conceptual framework will settle the foundations of dependability benchmarks. It addresses several important aspects such as the need for an overall and global system viewpoint to correctly select the measures to be evaluated and interpret the results. To put into practice the conceptual framework, enabling technologies will be investigated and adapted in some respect.

They address: measurements to be performed on the target system, fault representativeness and workload and fault load selection. The selected enabling technologies will be evaluated. The experiments will be performed to achieve a comprehensive work: different dependability benchmark prototypes will be defined and developed for two major application areas (embedded systems and transactional applications). The benchmark prototypes will actually be implemented in two different families of COTS operating systems (Windows and Linux), allowing a cross evaluation of the concepts and the enabling technologies in a true dependability benchmark context. The final goal of the experiments is the validation of the dependability benchmark prototypes, in the sense of assuring that the benchmarks results represent a practical and meaningful characterisation of the dependability properties of the target systems, both from the end-user's and the system developer's points of view. Finally, the consolidation of the whole set of results will allow to push further the enabling technologies and to finalise the benchmark concepts and prototypes.

Milestones:
The work is composed of six main workpackages:
1) conceptual framework;
2) enabling technologies identification and evaluation;
3) benchmark definition, experimentation and validation;
4) consolidation;
5) evaluation and assessment and;
6) dissemination and implementation.

The titles reflect the activities to be completed and reported in the associated deliverables. The two final objectives to be achieved at the end of the project are a report presenting the concepts and guidelines for dependability benchmarking and a set of dependability benchmark prototypes.

Funding Scheme

CSC - Cost-sharing contracts

Coordinator

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE
Address
3, Rue Michel-ange
75794 Paris Cedex 16
France

Participants (4)

CRITICAL SOFTWARE SA
Portugal
Address
En1/ic2, Km 185,6 Banhos Secos
3040-032 Santa Clara (Coimbra)
FACULDADE CIENCIAS E TECNOLOGIA DA UNIVERSIDADE DE COIMBRA
Portugal
Address
Colegio S. Jeronimo, Largo D. Dinis
3000-141 Coimbra
FRIEDRICH-ALEXANDER UNIVERSITAET ERLANGEN - NUERNBERG
Germany
Address
Schlossplatz 4
91054 Erlangen
UNIVERSIDAD POLITECNICA DE VALENCIA
Spain
Address
Camino De Vera S/n
46022 Valencia