CORDIS Archive

This page is no longer maintained.

For recent developments, please consult ournew website in the FP6 environment:

/digicult

Benchmarking

---

introduction

Our initiative is in line with the eEurope 2002 Action Plan that has developed a benchmarking framework as a tool to improve national practices and comptetitiveness.

We also have set up a Benchmarking Workgroup with experts nominated by Member States. This workgroup is active and has already carried out some preliminary data collections and analysis.

Under point 2. a short tutorial on the rationale underpinning a benchmarking approach is given. Under point 3 we present the basic structure of our proposed model, for additional information you can look at the model itself that is available for downloading in the first section.

• See also the pages on theInformation Society website deditated to the Commission's benchmarking activities:

http://ec.europa.eu/information_society/eeurope/benchmarking/index_en.htm

---

index

1. A coordinated approach towards National Digitisation Policies & Programmes in Europe:

A benchmarking model for digitisation policies/programmes -Benchmarking Workgroup

2. Benchmarking methodology:

• Process
• Quantitative and qualitative benchmarking
• Indicators
• Implementation strategies for benchmarking exercises

3. Digitisation Policies Benchmarking Model

---

An Open Method for Benchmarking Digitisation Policies

1 - A Coordinated approach towards National Digitisation Policies & Programmes in Europe

In the context of the objective 'European content for global network' the eEurope Action Plan asked to create a co-ordination mechanism for digitisation policies. A lot of good experiences are on-going across Europe, but the experts nominated by all the Member States recognised that digitisation programmes of Cultural Heritage are implemented in different ways and through different policy instruments and financial mechanisms. The participants endorsed the need to co-ordinate digitisation policies and programmes across European Member States urgently. Co-ordination can be stimulated and encouraged in the first instance through a better understanding of practices in the Member States and through building a platform for improved collaboration between countries in terms of sharing skills, exchanging "good practices" and promoting use of standards. Benchmarking can help develop this process. Therefore an open model for benchmarking digitisation policies in the field of cultural and scientific content has been realised.

This model, proposed by the Commission to member States, is a qualitative one: the main task is to encourage the exchange of good practice as part of a continuous process of improvement. It does not at this stage consider a comparative framework of quantitative measures, either between countries or against external (global) indicators.

Benchmarking policies gives rise to two main difficulties. Firstly, a policy is more complicated to monitor than a business or work process: the definitions of productivity, costs and results (key benchmarking indicators) are often intangible, thus requiring different mechanisms from those applied to businesses. Secondly, a policy is always affected by its particular political-cultural-legal context. The focus is, therefore, on the various factors influencing the decisions for producing digital content, as well as on some important framework conditions.

The reference sets for good practice which are needed to draw the benchmarking model have been developed through informal consultations with experts and the interim results tested in an EU-wide questionnaire on digitisation policies and programmes.

In this context, some experts workgroups were launched, one of the most active (already two meetings held) is the Benchmarking Workgroup (BWG)’, coordinated by our unit, whose members are experts nominated by Member States. The BWG is working to set-up a European framework to implement the model and involve all the countries authorities and actors in the implementation of the model; the work programme of the group is at the stage of the qualitative benchmarking and data collection, long-term objective is to implement a quantitative section of the exercise.

The activity is carrying on and a new set of results will be presented at a plenary meeting in Alicante on 16-17 May in cooperation with Spanish Presidency. More information can be found on the ‘eEurope Digitisation’ section of this site.

^back to top

---

2 - Benchmarking methodology

Process

Benchmarking methodology is a continuous exercise similar to our instinctive way of learning. A benchmarking exercise can be used wherever a process can be identified. The starting point is to define the process or activity to be studied.

--> Process --> Themes

For each process, the important themes have to be selected: these are derived from an informed analysis of the context and the issues affecting the process. The procedure is iterative allowing for new themes to be added.

--> Process --> Themes --> look at "good practoces"

Identify examples of “good practice” bearing in mind that though some activities may not fully reflect good practice, they may contribute with other examples to the overall definition of good practice for the theme in question.

On the basis of selected themes, some examples that can be considered "good practices" should be identified. In this phase it is important to look at each example having at least one aspect that could be useful to the overall process, even if that example as a whole could not be considered actually a "good practice".

--> Process --> Themes --> look at "good practice" --> Model of process (issues and indicators)

From this preliminary survey of good practice and background knowledge of the process and, in particular, from input from other experts, one can identify the main issues for each theme that are important for defining a model of the process.

The next step is to define practical indicators. These need to be measurable. Three kinds of measure are proposed:

Measures may evolve as the benchmarking exercise develops. These indicators need to be supplemented by qualitative information and background information. For instance national studies may provide a better interpretation of the relative performance of the local initiative. In this way, transferability of best practice should be facilitated.

--> Process --> Themes --> look at "good practices" --> Model of process (issues and indicators) --> Implementation (feed-back)

The next phase is to implement the benchmarking exercise for real. Participants, with the help of an expert group, can compare their actual practices with the model, obtain suggestions from it, refine and develop the indicators and repeat the cycle if necessary to validate the comparative information.

Standards of practice across the different environments.

Process --> Themes --> look at "good practices" --> Model of process (issues and indicators) --> Implementation (feed-back) --> Refinement of model

The activity is a continuous exercise. It is useful for decision-makers as a means to improve practices by developing conformity with the model. At the same time, the implementation phase is useful for continued development of the model: the expert group must use the implementation feed-back to refine the model and the indicators and, if necessary, to add a new set of themes (set of processes).

If benchmarking of digitisation policies is implemented among the Member States, the model should be suitable for all. Subsequent cross-comparison of results can help exchange experiences and good practices, thereby contributing to improving standards of practice across the different environments.

Implementation (feed-back) <--cycle--> refinement of model

At this point a cyclic exercise starts with the testing of the implementation leading to the refinement of model. It is important to repeat the benchmarking periodically.

^back to top

Quantitative and qualitative benchmarking

Two different kinds of benchmarking can be distinguished:

"Quantitative benchmarking" which has exactly measurable indicators and aims at effective measurements of the process in different implementation environments. e.g. The eEurope framework uses "structural indicators".

"Qualitative benchmarking" which examines the way in which work is performed. Some trend indicators and other measures are also used but the focus is on “method” and an examination of the alternatives. In this case the input of an expert group is vital. This may be the most promising way to exchange "good practice" among Member States.

^back to top

Indicators

The role of indicators in the benchmarking process is essential for the Member States to compare, with the support of the expert group, their practice. Indicators are vital to monitor performance improvement.

While indicators are only one element of the benchmarking process, they represent an important tool and basis for measuring and comparing performance. If the benchmarking process is to be successful and credible, it needs to be underpinned by robust, comparable, up-to-date and above all policy-relevant data. For quantitative benchmarking a table with exact measures is the expected result about final user practice. For qualitative benchmarking a "good practice model" with "trend" indicators is proposed for comparison between practice and the main issues of the model in order to generate improvement and suggestions: this type of benchmarking is as a rule less complex and onerous to implement.

The approach is, therefore, pragmatic and progressive, using the most pertinent and reliable data that are available in the short term, while seeking to obtain new and better indicators over the course of the cyclic benchmarking exercise.

Though the focus is on trend indicators, there are some quantitative measures proposed in the model, specifically for the ‘productivity’ theme: these indicators directly map to the proposed eEurope indicator on digitised content.

^back to top

Implementation strategies for benchmarking exercises

A mechanism needs to be established whereby the Member States can test their practices with respect to digitisation policy for cultural and scientific content. A starting point needs to be made for improving effectiveness by comparing practice with the model based on agreed "good practice". For implementation to work, there needs to be support from a group of high level experts nominated by the Member States. The benchmarking exercise is founded on two pillars:

Rigorous analysis: the cornerstone of credible benchmarking.
Trust and commitment of all those involved, agreeing on the aims, principles and procedures, and carrying them through.

The active involvement and contribution of Member States is of course essential to the process and its success.

^back to top

---

3 - Digitisation Policies Benchmarking Model

To date much of the benchmarking work undertaken in other sectors has focused on process engineering. This is equally true in the European Commission where the most active workgroup is at DG Enterprise though work on policy benchmarks has also been carried out by DG Research. In the USA benchmarking of the public sector is already broadly applied through initiatives such as "Reinventing the Government", though policy benchmarking is a more complex task.

A digitisation campaign can be developed in many different "sectors of application" where a lot of activities are included, and probably most of them are independent of computing technologies. Strong co-ordination between the different components and actors involved is needed. In the framework of cultural and scientific content one can identify a scenario with different kinds of actors and components:

Generally, content holders lack the necessary expertise and resources to conduct a digitisation process effectively: for this process to work, some technical resources, internal or external, must be involved as actors in the production chain. Copyright protection and a suitable business model to find the budget for digitisation and long-term maintenance need to be developed.

^back to top

"Good practices" exchange by means of qualitative benchmarking

The aim of this exercise is to support Member States in improving practice (policy for digitisation) by looking at other Member States' experience and skills. This can be achieved by adopting a qualitative benchmarking programme.

It is important to recognise the heterogeneity of this environment. The same causes/actions could produce very different effects depending on the conditions in different Member States. Therefore, rather than applying what is good practice in one Member State directly to another, the approach should be, through working with experts, to extract from the various examples of good practice some general features that can be adapted to the different national environments and contexts.

Improvement in national practices is achieved through the implementation of the model; exchange of “good practice"” is obtained through the refinement of the model by the expert workgroup.

^back to top

Themes

Some initial themes for description and analysis of the context surrounding a digitisation policy process have been identified as follows. These selected themes must be addressed separately but in a co-ordinated way, reflecting their interdependence. Additional aspects regarding the digitisation process will be progressively tackled either by selecting other themes or more simply by devising new indicators and additional studies on the same themes.

Management

In order to manage a policy effectively, some elements should be clearly defined (such as objectives, milestones, workplan and timetable). These justify the implementation and help in reviewing the policy.

Human resources

This identifies the skills available to produce a digital version of content, and covers factors such as: training and mobility of expertise. National reference centres in each country are one of the expected results.

Funding

This may cover issues such as: public and private investment in digitisation activity; fiscal and tax systems; strategies for attracting private financing; costs of a digitisation campaign; long term maintenance.

Productivity

This deals with the volume of digitised content and its share/percentage in respect to the total content for the sector. The eEurope framework has defined a measurable indicator based on the amount of resources digitised.

Impact

Addresses the added value resulting from the digitisation initiative in terms of usability, improvement in access, preservation of original objects, new telematics services, encouragement of Internet use, and promotion of cultural diversity.

Priorities

Criteria are needed in order to direct scarce resources to digitising the most appropriate materials.

Technical aspects

The most appropriate technologies to suit the digitisation requirements need to be identified. This point aims at promoting “good practice technology guidelines” and at supporting policy makers in creating a framework for new technology development.

Page maintained by: Digicult
Last updated: 23|12|2003