Evaluation of Research Infrastructures in Open innovation and research systems

Final Report Summary - EVARIO (Evaluation of Research Infrastructures in Open innovation and research systems)

Executive Summary:
Since the early 2000s, the development and the coordination of large research infrastructures (RIs) have been increasingly recognized by the EC as an essential pillar of the building of the European Research Area. Considering the amount of resources invested in RIs, an evaluation of their impacts is needed to support policy decisions. The EvaRIO project developed evaluation methods and tools well suited to RIs in the currently changing context toward an open innovation and research environment. It also contributed to the understanding of the conditions of knowledge creation through RIs, in particular their ability to foster collaborations and creativity. It then provided guidelines for policy makers to optimize the resources dedicated to RIs.

The core activity of EvaRIO, consisting in the adaptation of the existing BETA evaluation method to the RI case, led to the following results: As a preliminary step, an overview of the literature about studies, methods and indicators available for evaluating RI impacts was achieved and allowed the identification of gaps, needs and challenges for the design of a new approach. Then a comprehensive method was designed for the evaluation of RI impacts, including:

- A framework allowing the mapping of the different effects of RIs, and a set of indications of possible approaches to be used to evaluate them. In particular, it distinguishes four broad families of effects (direct, capacity, performance and indirect effects) to be identified and evaluated ex post at the level of different types of individual actors interacting with the RI along its life-cycle (typically operators, suppliers and users). The approach emphasizes the learning processes generated by RI-related activities as well as their economic value.

- A coherent architecture of specific methods, metrics and indicators allowing the measuring of some of the effects, with the corresponding indications of sources of information and data collection processes.

- A protocol for conducting RI case studies via interviews, which are the main source of information for part of these effects and indicators, with corresponding interview guidelines for each type of actor to be covered.

In order to test the EvaRIO approach, the validity of the mapping in various contexts and the feasibility of the method, nine carefully selected case studies based on 76 interviews were carried out in the BMS field: 4 in-depth ones following the protocol mentioned above (SOLEIL Synchrotron, EMMA mouse archive, CERM lab and EMBL-EBI resources) and 5 smaller ones to complement them on specific points. Beyond the evaluation in itself, a series of lessons as regards the implementation of the EvaRIO approach were drawn from the case studies.

In parallel, the completion of four Focus studies largely based on the case studies led to promising results on the following topics: the analysis of the open source properties of the dynamics of knowledge creation via RIs; the mechanisms by which a large RI, here a synchrotron, manages its flexibility in a science intensive and uncertain environment while contributing to an ecosystem of RIs for the benefit of the scientific community; the role of RIs as catalysers of R&D collaborations and network brokers, analysed via an original use of social network modelling; the impact of RIs on regional development, emphasising a local beneficiary-oriented approach to evaluation design, better adapted to expectations of local authorities.

In addition, a series of policy recommendations were proposed to the EC in the perspective of the starting Horizon 2020. They deal with the improvement of evaluation approaches of RI impacts, and the possible ways through which the EU could help optimise these impacts.

Finally, the EvaRIO project created a website and organized several seminars for dissemination purposes, as well as a large final conference gathering policy makers and specialists in fields related to RI impact evaluation.
Project Context and Objectives:
Since the early 2000s, the development and the coordination of large research infrastructures (RIs hereafter) have been increasingly recognized by the European Commission as an essential pillar of the building of the European Research Area, and hence of a competitive and dynamic European knowledge based Economy, as defined by the Lisbon summit. According to the European Commission definition (cf. EC website or various ESFRI documents), "The term ‘research infrastructures’ refers to facilities, resources or services of a unique nature that are needed by the scientific and technological communities to conduct basic or applied research in the whole range of scientific and technological fields. This definition includes the associated human resources and covers:
- Major equipment or group(s) of related instruments used for research purposes
- Knowledge based-resources such as collections, archives, structured information or systems related to data management, used in scientific research
- Information and communication technology-based infrastructures such as grid computing, networks and communications.
Research infrastructures may be ‘single-sited’ physical facilities (a single resource at a specific location), ‘distributed’ (a network of distributed resources), or both (e.g. where ‘virtual’ access to a core facility is provided electronically)."

Considering the amount of resources invested in RIs, an evaluation of their direct and indirect impacts is needed to support policy decisions. The EvaRIO project aimed at developing an evaluation framework and a set of specific evaluation methods and tools well suited to these RIs in the currently changing context towards an open innovation and research environment. By developing methods and indicators to identify and measure economic impacts of RIs, EvaRIO was to provide guidelines for policy makers to optimize the resources dedicated to RIs. This study had also to improve the understanding of the conditions of knowledge creation through RIs, in particular their ability to foster collaborations and creativity.

The project had primarily to adapt the undernoted "BETA approach of evaluation" to the case of the RIs, an approach used so far for ex post evaluations of the economic impacts of a large variety of R&D programmes. The adaptation was to be carried out in connection with in-depth investigations of the evolution of RIs and the research networks around them. The underlying idea was that S&T knowledge creation in general and knowledge creation through RIs in particular result from a cumulative and interactive learning process.

More precisely the project included two parts: a Core study, that aimed at developing an original evaluation approach and the corresponding evaluation methods and indicators, and a series of Four Focus studies, to provide complementary analyses related to the role and impact of RIs in open innovation and research systems, in conneciton with their evolution and networking properties. The empirical materials used by the studies were to be built via the conduct of a series of RI case studies, based on interviews and various data collection processes.
These three types of targeted studies are now described in a more detailed way.

The intended Core Study

The planned Core Study had to consist in the development, extension and adaptation of an existing evaluation method, the BETA method. This approach had been previously developed and used for conducting retrospective evaluation of some of the economic impacts (so-called indirect effects) of a large variety of programmes launched by public authorities to support research and innovation. It was initially devised to evaluate some of the economic effects of the oldest and largest RI, the CERN. Since then the method had become one of the acknowledged methods of evaluation in the field of STI policy.
Although many different programmes were evaluated using this approach, they all included some R&D activities run in the form of projects. In BETA approach, direct effects were usually distinguished from indirect effects on the basis of the objectives of such projects. Hence, beyond the direct effects (corresponding to what was targeted when these R&D projects were designed and launched), the focus had always primarily been on indirect effects, (further exploitation of various types of knowledge, network, modes of organisation, etc. developed or acquired during these R&D projects). Basically, this approach tended to identify (and to measure the impact of) various types of learning processes triggered by the participation in R&D, and considered as the heart of the knowledge creation and diffusion dimensions of innovation.
Since the context of RIs was most probably different and richer than the context of standard public programmes supporting R&D projects, the aim of EvaRIO was to adapt the BETA approach to the context of RIs.

This adaptation had to be done along two main lines:
- BETA mapping: A comprehensive mapping of the different effects that could be generated by or thanks to RIs was intended. In particular, it had to introduce a clarification of the distinction between direct vs indirect effects and a classification of effects as they appeared along the lifespan of the RI, and to attempt to include the probable enrichment and network effects linked to an open innovation and research perspective
- EvaRIO-BETA method: Adapted from the original BETA method, a further development of methods and indicators, allowing the identification and the quantification of some of these effects, was the central objective of the project. This was to be based on a microeconomic approach as well as in-depth interviews of a sample of RI actors.

The intended Focus Studies

Focus 1: The dynamics of knowledge creation via open source research infrastructures
RIs are central elements in the dynamics of innovation. Their analysis and evaluation could not therefore be made in a purely static framework. It was important to think about the benefits of RIs in a dynamic, on-going context. Since research projects on RIs were flourishing and in the same time fed RIs, we expected that once an RI succeeded in fostering research and industrial projects, those projects would enrich the RI (via an additional content and/or reputation effect), which in turn would foster new projects, thus triggering an on-going endogenous dynamics.
A promising approach to understand and analyse the dynamics of RIs was to be found in the recent but already abundant literature on open source. Originally applied in the context of software, the principles of open source now emerged in other fields such as life sciences or pharmaceuticals for instance.
The ojective was to develop the approach of open source research infrastructure (OSRI) to evaluate the enrichment effect of RIs by ensuring that feedbacks from users effectively contributed to the improvements of RIs.

Focus 2: Flexibility of RIs, an analysis using real option reasoning
Like many large scale investments RIs are examples of complex products and systems characterised by capital, technology and knowledge intensive investments and networks involving numerous actors in their development and operation processes. As such they are long term projects and face many uncertainties throughout their duration. Since RIs are intended to support research and innovation activities their flexibility to new research and innovation needs is a critical aspect of their efficiency and effectiveness.
In this Focus the objective was to adopt a real option reasoning framework to explore and provide insights into some of the critical aspects of RIs related to the flexibility criterion. RIs probably had many option-like characteristics making their adaptability and responsiveness an important determinant of their overall performance.
In this perspective the research was to be built upon a growing literature on real option reasoning to:
- Identify different mechanisms (technical, organizational, institutional) embedded within RIs that support their flexibility to emerging innovation needs.
- Better understand how the flexibility embedded within the RI supports different types of effects analyzed by the BETA methodology.

Focus 3: RIs as catalysers of collaborations and networks
In this study the objective was to analyse the causal links between RIs and research projects through the lens of innovation and research networks. Research infrastructures are more and more expected to generate new knowledge by boosting collaborations between researchers. Thus they have to be conceptualised and evaluated in a framework of open innovation and open research, where innovators cannot operate alone but have to use inside-out and outside-in technology transfers (for instance through spinoff and respectively licensing-in policies), as well as co-creation processes (for instance through collaborations and research joint ventures).
The intended work was thus to explore and characterize the networks of research collaborations created around RIs in order to:
- Provide a better understanding and appropriate indicators of the different kinds of network effects, as derived from the adaptation of the BETA approach.
- Explore the relationship between RI success (or effectiveness) and the networks created around them, in particular identifying and analysing the conditions under which RIs may foster networks and/ or operate as a hub of research collaborations;
- Identify possible network specificities and/or collaboration rationales in the case of pan-European RIs, including possible ramifications corresponding to given scientific domains.

Focus 4: Impact of research infrastructures on regional creativity
The issue concerning the extent to which RIs significantly stimulate S&T creativity seemed to be of special interest. Novelty and creativity are usually supposed to emerge through the combination of distinct and complementary knowledge bases. This study was to focus more specifically on the benefits of RIs on creativity at regional level, i.e. beyond the community of researchers stricto sensu.
The mere presence of an RI and hence of a concentration of "knowledge workers" in a given region could in fact exert a positive influence on the creativity of companies located in the neighbourhood and more generally on the whole local economy. Boosting networks and creativity concerned S&T agents directly involved in RIs (operating most of the time at the international scale) as well as more peripheral, industrial and economic agents operating in the regional sphere.
In order to investigate the hypothesis of an impact of RIs on creativity at regional level, the initial aim of this focus study was to conduct two complementary lines of work: (i) an analysis of the impact of RIs as they are defined by the EU; and (ii) a comparison with a regional based infrastructure.

The intended RI Case Studies

A series of RI case studies were to be run via interviews to assess the validity of the mapping in various contexts, the feasibility of the evaluation method, and bring the relevant facts to feed the Focus Studies. Four cases studies were initially planned. But following a phase of methodological development, deep examination of the nature of the RIs to be evaluated and first inquiries, it was finally decided in accordance with the EU representatives to run 4 "large cases" (including more interviews and a comprehensive coverage of the evaluation topics) in parallel to 5 complementary "small cases" (covering only specific topics with a limited number of interviews).

Intended dissemination

By developing methods and indicators to identify and measure economic impacts of RIs, EvaRIO was also to devise policy recommendations regarding the design, setup and content of an evaluation system. Moreover a dedicated website was to be created and a final conference as well as some dissemination actions were to be organised at the end of the project, in order to share results and future implementation perspectives with the stakeholders, as well as to feed policy recommendations.
Project Results:

This document synthetises the main results of the project EvaRIO, dedicated to the evaluation of the impacts of research infrastructures (RIs hereafter) in the current context of open innovation and research systems.
EvaRIO provided five broad types of results in this respect. First and above all, it developed an evaluation approach, and tools well suited to RIs, based on an adaptation of the undernoted BETA method. All the methological aspects concerning impact evaluation, literature overview as well as original developments, are stored and detailed in the EvaRIO Core Study (cf. D5.2 Part I). Second, EvaRIO contributed to the understanding of the conditions of knowledge creation through RIs, in particular their ability to foster collaborations and creativity, by the mean of four complementary, exploratory Focus Studies (cf. D5.2 part II). Third, it implemented nine case studies, based on 76 interviews about RIs in the biological and medical sciences (BMS), in order to test and refine the new EvaRIO-BETA evaluation approach and also to feed the Focus Studies (cf. D5.1). Four, it provided a series of guidelines for policy makers, both to improve evaluation approaches and to optimize the resources dedicated to RIs (D5.2 Part III). Lastly, EvaRIO launched and organised a set of dissemination activities.
All these results are presented in a more detailed way below.

1 Main results of the EvaRIO Core Study: Developments of methods and tools for evaluating RI impact

A preliminary step was to conduct an extensive survey on the literature about studies, approaches, methodologies and indicators available for evaluating the impacts of RIs, which allowed the identification of gaps, needs and challenges for the design of a new evaluation approach. Then a comprehensive method was designed for the evaluation of RI impacts, based on the adaptation of the existing BETA evaluation method, and including, first, a framework allowing the mapping of the differents impacts and effects at stake, the, a coherent architecture of specific methods, metrics and indicators designed to measure some of these effects, and, finally, the protocol for conducting the RI interviews which are the main source of information for these effects and indicators.

1.1 A literature overview about RI evaluation

The evaluation of the economic or socio-economic impact of RIs is by far not a completely new domain in the field of the evaluation of impact of public actions towards R&D, but it is probably not as well developed as the evaluation of more standard "public R&D programmes", which are mostly funding tools. Then classical and frequently cited global surveys on evaluation in the field of R&D public programmes, as well as guidelines for impact assessment of public actions from OECD, EC, or other international organisations are useful but not fully adapted to this specific context. The following facts can be highlighted.

(i) Different frameworks have been proposed for mapping the different types of effects and impacts of RIs, but there is very little unity and coherence between those, and there is a lack of a unified theoretical background and/or a common perspective of analysis.

In order to take into account t he complexity and the variety of impacts, many attempts have been made to elaborate some classification on the basis of the definition of large categories. Obviously, each evaluation proposes a typology, but it is often in the form of a list of items rather than really based on a clear-cut classification, with hierarchical, logical and on purpose ordering. However there are exceptions to be noted.
For instance, different frameworks distinguish between four broad categories of impacts (with a list of corresponding sub-categories and possible evaluation methods) which more or less correspond to the following areas: Science, Technology & Innovation; Work & Population; Quality of Life; Ecological Environment.
A slightly more elaborated framework also couples categories of impact level (societal level - i.e. generating changes in behaviours, e.g. use of internet, interest in science, understanding societal changes -, health & environment level - i.e. supporting development of new technologies, helping understand climate change issues -, economic level - i.e. gains in scale and scope, increased industrial competitiveness, new technologies / products), time horizon, and a distinction between direct and indirect effects.
Different studies (especially in the UK) were inspired by the seminal SPRU-type work on the impact of science, in particular Martin, Salter and Tang, which for instance forms the foundations of the evaluation system used for some of the UK RIs (especially by the STFC - Science and Technology Facilities Council).
Other noticeable proposed categorisations suggest geographical distinctions (local vs. broader) and/or the micro, meso and macro levels of impacts; both categorizations are sometimes adopted in case studies.
Interesting also is the link established by some authors between the area or category of impact and the interest of various stakeholders. They claim that different types of effects and different types of indicators are adapted to different stakeholders of the same RI, because they have different objectives and different "mental backgrounds".
A quite different - and promising - perspective is related to the "logic model" approach, finely detailing the causal linkages between objectives, instruments and impacts, and then deriving a specifically adapted categorization of impacts, allowing for an easier connection to the analysis of effectiveness and efficiency.
One can also note the lack of approach encompassing the role of RIs in structuring and orientating the research agendas and the research communities and, more globally, the role of RIs in the dynamics of development of science (and of society) is hardly evaluated.

(ii) The panel of evaluation methods as a whole looks like a patchwork of different approaches with different levels of development.

Different families of methods seem to dominate the scene. First, there is the input-output approach, designed to assess the impact of the amount of money spent for the building of the RIs and/or the operation/maintenance of the RIs. More precisely, this refers to the impact of the expenditure of the RIs and of the suppliers of the RIs, and in some studies as well of the RIs staff or even of the visitors of the RIs, on the (mainly local) economy. On the basis of existing statistics, different waves of economic effects are estimated along the suppliers’ chain (including consumption); this is a quite classical approach that is often applied to any type of infrastructure beyond the field of research and technology.
Secondly, one can find numerous attempts to compose a sort of scoreboard of various isolated indicators supposedly reflecting on the one hand the activity and performance of the RIs, and on the other their outputs and impacts. Both aspects are often mixed up, blurring the distinction between evaluation for monitoring purposes and evaluation of impacts. Among the indicators, the standard S&T indicators are flourishing (from publications to patents and spin-offs). The most often used indicators are for instance indicators dealing with staff, funding, costs, users (hours, type…), availability and reliability of RI resources, bibliometrics, patents, jobs-activity in local economy via staff and RI expenditure, PhDs, lectures and training programmes, visitors, scientific events, start-up incubation and spinoff generation, participation in R&D contracts as well as many issues through examples/success stories.
Thirdly, one should mention the methods used to evaluate the impact of the scientific results on economic growth and welfare (e.g. the impact of a new family of drugs or of certain types of therapeutic treatment on mortality or on productivity of some classes of population or on health national expenditure). Measurement is then made via ad hoc tools and methods specific to each domain, but relying on welfare gains such as those due to longer life expectancy, cost savings, consumer surpluses and profits for suppliers and consumers. But these methods have not been specifically applied to results obtained from the use of RIs, they are in a sense applied downstream, taking the research process conducting to the research results as a black box.
Quite surprisingly, the use of standard methods such as cost-benefit analysis (in the strict sense of the term, i.e. using the whole apparatus of hypotheses and tools such as price estimation, consumer surplus, welfare, discount rate etc.) is apparently relatively rare. Similarly, the production function based approach, so widely used in the case of the estimation of the return of R&D expenditure at micro as well as meso (e.g. regional) or macro level is apparently relatively neglected. And on the other side of the spectrum of the standard evaluation tools, social network analysis, which has experienced such a rapid development in the recent period, is paradoxically not so often applied, at least not with its full potentiality.

(iii) There is a specificity of RIs compared to other public support to R&D.

RI is not a simple financial support (for research, for collaboration, …); it is above all a provision to users of multi-facet and evolving resources which are largely of a public good nature, hence the importance of the issue of access. Then the impact cannot be grasped as the impact of a standard R&D project with goals, resources, time horizon, as many evaluation methods tend to be targeted at (including the BETA approach, so far). In the same vein, issues of the "impact of RIs", "impact of RI-based R&D", "impact of R&D projects using RIs" and "impact of science" are not exactly similar. The point here is that the methods of evaluation often address preferentially one or the other dimension.

(iv) There is a huge and growing heterogeneity of RIs, while most of the evaluation approaches used so far applied to the "old standard type" of RIs (i.e. a large facility set up in one location).

In addition to the distinction mentioned in the official definition of the RI, in which the nature of resources put at the disposal of the users (i.e. instrument and equipment, data and digitalized resources, collections of samples, and competences) should particularly be taken into account, three other connected economic dimensions are important: the nature and degree of rivalry of the resources, the rules of access designed for accessing these resources, and the relative importance of the rationales backing the setup of the resources (allowing cost-saving and scale economies vs. fostering variety or learning processes). The approaches proposed for future applications have often been considered to be usable for any type of RIs, without conducting a deep examination of the diversity of the characteristics of RIs in order to check the relevance of the evaluation method proposed for such different contexts.

(v) The dynamic dimension of the RI and its relation to the generation of impact is often under-estimated by evaluations.

It should be noted that this issue is sometimes a bit blurred with the issue of the position of the evaluator, ex ante vs. ex post. But one can consider the relevance of the RI "life-cycle", which is by far more complex than the simple distinction between "construction" and "operation". Design / preparation, closing down / recycling / decommissioning, and above all the networking with other RIs, the dynamics of modifications of the RIs (upgrade, additions, recycling during the operation phase and the order of magnitude and frequency of modifications), the change in the data and/or knowledge about the functioning of a resource and about the different specificities and possibilities it offers from the point of view of its interest for scientific research, etc. are also sources of creation of impact. Hence, over time, different actors take part, different economic phenomena emerge, different trajectories of research are open, etc. This is related to the networking, diversification and enrichment of RIs.

(vi) The systemic nature of the interactions “around” the RIs is hardly addressable with too linear a causality reasoning.

Basically, at each phase of the RI life cycle, different systems involving different actors are concerned/affected because of the RIs. Those impact pathways may cross, as they may involve the same actors and to some extent the same intermediaries. Interactions between systems concerned/affected by RIs and the environment (including the flows from the system to its environment) then give birth to actual impacts along a variety of pathways. Again, the usual simple causality from input to output of R&D activities, often implicit to a lot of existing methods, should be enriched in order to be adapted to the RI case.

1.2 The design of a renewed and adapted BETA evaluation method

In the continuity of the BETA method, the EvaRIO approach is based on the following main lines:
- A micro-analytical level of analysis: the effects are studied at the level of individual actors (researchers, labs, companies…) where they are generated, and only the effects for these actors are accounted for, thus a priori not the effects for the rest of the economy or society
- These actors are considered as regards the role they play in the RI related activities along the "RI life-cycle", i.e. basically operators, users and builders/suppliers
- Central attention is paid to the learning processes and their economic dimensions, i.e. the dynamic processes by which knowledge is created, used, shared and diffused, the behavioural, structural and organizational changes that go along with those processes, and the valorisation of all this in economic terms
- A retrospective (or ex post) approach
- The development of a simplified and harmonized mapping of the different effects, whatever role is at stake, with a corresponding architecture of specific metrics and indicators
- An attempt to first identify and characterize the effects and the way they are generated, and secondly, if possible, to quantify them, including but not exclusively in monetary terms
- A collection of data essentially through direct interviews with the above mentioned actors, completed by the use of complementary data sources such as bibliometric ones
- A willingness to develop framework, typologies and metrics that could be adapted to various cases of RI, not only in the field of BMS.

As will be developed below, the renewed and adapted BETA-EvaRIO evaluation method includes three main parts:
First, a framework allowing us to map the different effects and impacts of RIs (although inevitably limited in coverage and in terms of theoretical assumptions), and a set of indications of possible general approaches to be used to evaluate them
Second, a coherent architecture of specific methods, metrics and indicators allowing us to measure some of the effects whenever possible, with the corresponding indications of sources of information and data collection processes
Third, a protocol for conducting RI case studies via interviews, which are the main source of information for part of the above mentioned effects and indicators, with corresponding interview guidelines for each type of actor to be covered.

1.2.1 The EvaRIO Evaluation Framework

(i) Framework: focus on specific groups of actors

As stated above, the BETA approach is a micro-based and actor-based one. It means the actors are the entry point for the evaluation, and the effects will be envisaged at their level (for the analysis of the generation of effects, for the identification and the measurement of the effects, and for measuring which entity is affected by the effects).
A key point is that only the actors who are performing R&D activities, at any stage of the RI life cycle, can be selected to be under scrutiny within the BETA approach. The reason is that it is the very R&D activity which engage them in the learning processes on which the BETA approach focuses. Having said that, the same actors can also experience other effects, which are more or less directly related to the learning processes. In this respect, the approach may include those other effects in the evaluation, but the focus will be on the former.
Therefore, the first task of the evaluation is to identify those key actors. Following a quite simple and well-acknowledged classification, we distinguish the following types of actors:
• The RI designers
• The RI builders, i.e. the suppliers for:
* The construction of RI elements
* The networking of RI elements
For the sake of comparison with existing methods, we refer to those actors as suppliers.
• The RI users, i.e. researchers who:
* Are "attached" to the RI (i.e. hosting research team, including researchers and supporting staff), as using the RI for their own research
* Are temporarily visiting the RI site in order to use it
* Have a remote access to the RI
• The RI operators, i.e. hosting research team (researchers and supporting staff), as supporting the users.
• Actors involved in the upgrades and additions
• Actors involved in the deconstruction/decommissioning

The three main categories covered in the present study are the RI suppliers, the RI users and the RI operators. It should be noted that other roles than the three main ones extensively discussed so far exist. The evaluation of the effects for these other actors could also be carried out (but it was not done in EvaRIO case studies).
Of course, the frontiers between the three roles may be blurred, and the same organisation may play different roles. These combinations, while making the evaluation exercise more difficult, are highly related to the nature of the resources stated above. More precisely, the most typical combinations are exhibited in Table 1.

Table 1: Nature of resources and roles of actors
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

In addition, it is very important to underline that the operator may be him/herself more or less directly involved in the use of the resources by the user (collaborating with the user, providing various ranges of services and assistance) or just maintaining the resources and putting it at the disposal of the users who use them in full autonomy.

(ii) Framework: the typology of effects

The framework designed in the EvaRIO study distinguishes between four different sets of effects. This framework is to be applied to each of the three main categories of actors, giving birth to 3 x 4 = 12 different sub-sets of effects. The four sets of effects are presented first, then the framework is applied to the three categories of actors.

- Increase in RI-based activities: The direct effects
Depending on the role of the actors who are analysed, the nature of the activities they carry out in relation to the RI (i.e. their RI-based activities) obviously varies. But typically, building, upgrading, operating and using the RI represent as such an economic activity for the actors, which has a certain volume over a certain period of time. For other actors, benefiting from the RI resources (in terms of equipment, data, services…) for conducting scientific experiments has also a value of its own, as compared to using alternative resources. All these correspond to the direct effects.

- Learning, accumulation of knowledge and changes in actors: The capacity effects
The central focus of the evaluation concerns the effects derived from the different forms of learning experienced when performing the RI-based activities. Carrying out these activities allows the involved participants to acquire and generate knowledge and competences, to create/reinforce ties with other actors, to modify their organisation and their ways of conducting their activities, to recruit new staff, to gain in reputation, etc., all these dimensions contributing to an increase in the capacity of the participants. To some extent, the change in the capacity corresponds to a "potential for future impact": the very change in the capacity of the participants that will influence the possibilities for their future actions.
Distinguishing this capacity dimension is supported by different theoretical considerations, in particular stemming from recent developments in economics and management of knowledge.

First, the nature of the capacity has to be further defined: to identify and characterize this change in capacity is a first very challenging task in any evaluation work. Following a tradition open by seminal works of knowledge-based approaches in economics and management sciences, one can basically distinguish different levels in this capacity, at the risk of oversimplifying these seminal works. Capacity is composed of :1) resources (or assets) such as scientific knowledge, technological know-how, ties and connections with other actors, reputational assets, organisational assets (forms of organization), 2) competences or capabilities (at individual but above all at collective level) that reflect the ability to mix, combine, integrate and exploit these resources for a given purpose, and 3) dynamic capabilities which denote the ability to build and make the resources evolve and to make the way to mix, evolve, combine, integrate and exploit them, in order to adapt to a changing environment. Resources may be generic or very specific to one given actor, while by nature competences and dynamic capabilities are mostly specific. A key feature is the competing advantage all of them may confer to the actors against their competitors.
Part of what is generated when carrying out R&D activities is codified, formalized, objectified and externalized, i.e. it becomes separate from the people and the organisations that generated it. And part of it remains embedded in these people and in these organizations. Part of it is then left open (at least partly) to other actors in the sense that it can be used by them without the help or the support of the participants (e.g. scientific knowledge formalized in a theorem, data made public on genome sequencing) while another part is only at the disposal of the participants (data base for internal purpose, specific procedures for carrying out an activity…). The first part is poured into the public "pool of knowledge" open to all, while the second part is archived in the ""private section" of the knowledge base of each actor. But we assume that the actors who generated this knowledge are in the best position to use it, and then they should be included as resources for these actors. Both dimensions can be exploited, even if the codified and externalized dimensions often facilitates (or is a pre-requisite for) economic exploitation.

Secondly, even if at some point in time, or during a certain period, an impact is created in terms of economic flow benefiting the users (and most probably other actors), the capacity is still "alive": once it is exploited, it is not destroyed or consumed, and it does not mean that it cannot be exploited anymore (this is the property of non-rivalry often underlined by economists). This is a key point here, also related to the very nature of what is generated, and to the impossibility to treat the capacity using the usual concept of stock. Information, knowledge, competences, methods, organisational setup, reputation, ties with other actors, etc. cannot be considered as a stock that diminishes (respectively accumulates) through an out (respectively in) measurable flow; correspondingly, they cannot be seen as an addition of measurable and strictly separable units, but exhibit some indivisibility features.

Thirdly, to build the capacity requires time, including accumulation, combination and re-combination, maturation and incubation phenomena. Furthermore, using the capacity may contribute to developing it; and last to maintain it at least requires a maintenance flow. It then requires time to be constituted (and to be dismantled), it exhibits some indivisibility, and requires a flow of maintenance.

Turning to more operational matters, we propose first to adopt a quite simplified and general definition of the capacity (i.e. knowledge-based assets and the capacity to make them evolve). Furthermore, five different categories of effects are distinguished, corresponding to the nature of the different dimensions developed and/or acquired. The first four correspond to domains of learning, development and acquisition of different components of the capacity. The fifth category is of a different nature, and will form the basis of the capacity generated (and will be underlying its exploitation corresponding to the "actual impact"). The categories are listed as follows:

* Science and & Technology capacity is related to S&T knowledge and competences
* Organization & Methods capacity is related to knowledge and competences regarding the management of activities and to organisational changes
* Network capacity is related to knowledge and competences regarding relations with other actors and to ties with other actors
* Reputation capacity is related to reputational assets
* Human capital capacity is related to the enlargement/diversification of staff

As mentioned earlier, the typology of effects is large enough to be applied to the actors involved in different RI-based activities. However, the frequency and importance of effects may vary from one type of actors to the other, and in specific cases, it could be useful to adopt some more detailed sub-categories as well as to add some specific elements of characterisation.

Exploitation of this capacity in two directions will constitute the last two categories of effects.

- Exploitation of change in capacity 1: The effects on performance
A first area in which the capacity seen above can be exploited is the conduct of the RI-based activities themselves, allowing the increase of the level of performance of the actor on those activities. The notion of performance could be envisaged in many respects (see below the section on metrics), including in its scientific, technical, managerial, economic etc. dimensions. What is at stake here is that the actors are able to perform better what they do, be it by supplying, operating or using the RI.

- Exploitation of change in capacity 2: The indirect effects
In this fourth category of effects are included economic flows that are generated by the other forms of exploitation of the capacity increased thanks to the participation in the RI-based activities. What makes the difference with the previous category of effects is that exploitation is made via other activities than those carried out "on the RI".
Keeping the categories of capacity effects put forward leads us to distinguishing between the following types of effects:
* The Science and Technological effects concern the use of S&T related knowledge developed during the activities, including the “transfer” of S&T from the RI-based activities to other activities of the actors. Following the definitions proposed by 'evolutionary economics' and 'knowledge economics', the term 'technology' here encompasses artefacts (products, systems, materials, processes) as well as codified, tacit, scientific, and technological knowledge (apart from methods, see Organization and Method Effects). What is used and/or transferred can therefore be of a very diverse nature, from scientific expertise to worker’s know-how, including technology laid down as a blueprint, new theories or 'tricks of the trade'; this broad approach is part of the originality of the BETA methodology. This leads to the design of new or improved products, processes or services, allowing actors to carry out further activities in the same field, or to contribute to research activities in more or less related domains.
* The Organization & Method effects (subsequently referred to as O&M effects) occur when experience gained through the RI-based activities allows an actor to modify his/her internal organization and/or to apply new methods in project management, quality management, industrial accounting and so on.
* The Network effects refer to the impact of projects on the cooperation among economic actors: for instance, the establishment of business links between actors in a consortium which leads to the continuation of commercial or technical collaborations after completion of the project; or cooperation between actors and organizations or firms not involved in the project, for instance, with the supplier of another actor, or as the result of a conference or workshop organized in connection with the project.
* The Reputation / label effect results from the reputation acquired by actors working on the RI-based activities: this good image or reputation is often used afterwards as a marketing tool.

The following Figure provides a simple sketch of the articulation between the different sets of effects discussed above.

Figure 1: Relation between the different types of effects
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

The typology of effects then relies on a twofold categorization, i.e. as regards type of actors and as regards type of effects. It is summarized in the table below.

Table 2: Crossing categories of effects and categories of actors - Summary
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

(iii) Framework: the RI Perimeter

The approach requires determining what is the perimeter of the RI to be taken into account as the locus where the processes of generation of effects start, i.e. which resources are open as RI and during which period of time. The scope of effects taken into account can be understood by looking at a simplified description of the process of interactions between RI and actors when referring to this perimeter, and of the learning and valorisation phenomena related. The perimeter is defined according to two dimensions: which resources are open as RI and during which period of time.

The delimitation of the perimeter is the basis for the operationalisation of the definition and the scope of the effects. And given the complexity and variety of the RIs, this apparently trivial question is actually one of the most difficult issues.

As regards the identification of the resources, a first step is to use the typology of resources presented above, which allows the identification of the nature of the resources at stake. But then, the analysis is made complex, for different reasons.
First, the resources open as RI may correspond to the whole set of resources hosted by a given organisation (i.e. the operator), or only part of them. Second, from an administrative and organisational point of view, the delineation between the two sets of resources and their associated staff may be difficult.

Regarding the timing issue, in the perspective of ESFRI, the most "progressive" opening up of resources as RIs basically follows the different steps (once the resource as such has been supplied):
a) Internal use of the resource by the organisation hosting it
b) Use of the resource in a collaborative way between the hosting organization and external partners
c) RI-access mode of use of the resources
d) Networking of resources making up the RI
e) ESFRI integrated access to the resource
Step c) should normally be taken as the starting point.
But of course, not all RIs follow these steps in a sequential way, and some of these steps may be skipped or combined, i.e. they may happen at the same time. Typically, some resources may open up as an RI as soon as they are set up, as in the extreme case of a centre fully dedicated to being an RI, a purpose for which the centre was created right from the origin. At the other extreme, one may find a research centre or a university lab which opens up some of its resources as an RI but only starting from a certain date. In the former case, the whole centre from its birth is an RI as such and then as such corresponds exactly to the perimeter, while in the latter case, the perimeter of the RI has to be delimited "within" the lab or research centre and "within" a limited sub-period of time. Various intermediary situations may occur, with for instance different resources progressively "entering" the circle of those opening up as RIs.

Another issue is the historical path of the organisation hosting the resource(s) opening up as RI. Quite frequently, organisations change and then the institutional environment in which the resource is open changes as well, providing other possible points of reference in time that may present some advantages for the identification and measurement of different evaluation indicators.

And finally on top of that, the network types of RIs (or the RIs formed of a network of individual RIs, both ways are used for labelling these organisational sets up) require adopting another level of perimeter definition, which differs from the one of the individual resource(s) and/or organisation hosting it/them). These cases also raise additional difficulties, such as for instance the fact that individual RIs may join the network at different points in time.

The main consequences of these different sources of ambiguity and complexity as regards the definition of the RI perimeter are: 1) that there are often different options possible, each option showing advantages and limits (e.g. too recent a period: no user yet; set up of the resources long before its opening up: limitation of the period putting the activity of suppliers out of scope; complex delineation "within" operators who are opening only part of their resources as RI, etc.), and consequently 2) that there is a risk of choosing ad hoc and heterogeneous perimeters across different RIs. The case studies conducted in the EvaRIO project largely illustrate these difficulties, each case being highly specific and subject to various interpretations as regards the "ideal" perimeter.

(iv) Framework: complementarity with other methods

The proposed EvaRIO approach aims at fulfilling a gap between three families of evaluations identified in the survey:

1. The classical assessment of the performance of RI operations (in terms of cost, productivity, access, quality, services etc.), related to monitoring activity: such approaches (type I in Figure 2 below) are generally carried out during the activity (but can also be made ex ante or ex post).

2. The standard evaluation of the economic impact of building and operating RIs (e.g. "what is the impact of a local synchrotron via its business, visitors, purchases, employees expenditure, etc."), dealing with:
- So-called "direct, indirect and induced" economic effects of the RI (expressed in monetary terms such as sales, revenues or income, or in jobs / FTE or in fiscal revenue and tax return);
- Measurement via the level of expenses injected into the economic system, generally in a determined geographical location;
- The use of an input-output matrix in order to calculate the impact.
This approach is most adapted to the case of large facilities installed in a given location
Such approaches (type II in Figure 2 below) are generally carried out ex ante or ex post.

3. The last family concerns the evaluation of some societal and economic impacts of scientific achievements obtained thanks to the use of RIs as one of the contributors to scientific advances (e.g. the impact of a new family of drugs or of given therapeutic treatments on mortality or on productivity of such a class of population, on the players of the relevant market - big pharmas, distributors ... -, and on national health expenditure), dealing with:
- A wide scope of long term effects of scientific advances in a given field: ICT, health, environment, energy;
- Measurement via ad hoc tools and methods specific to each domain, but relying on welfare gains such as those due to long life expectancy, productivity increase and cost savings, consumer surplus and profits for suppliers and consumers;
- Possibly put in a broader perspective of "ultimate" long term economic growth or total factor productivity effects.
Such approaches (type III in Figure 2 below) are generally carried out ex post.

The overall framework of effects sketched above allows us to connect and articulate those different effects: actually they can be related to some of the sets of effects that were presented in details (see Figure 2 below).

Figure 2: EvaRIO typology of effects and effects from other approaches
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

From a more operational point of view, it also means that there is no real need to design new evaluation tools and metrics since these approaches already cover part of the picture. Therefore, the EvaRIO development work as well as the case studies are focused on part of the effects only, precisely those which, according to us, are insufficiently or partially covered by these other approaches. Moreover, some of the data and information that are collectable via the implementation of the EvaRIO evaluation methods can feed these other approaches and/or be used as "starting points" for their implementation.

(v) Framework: the aggregation of effects

The aggregation of effects is a very complex issue because of the multiple levels at which it can be envisaged. With four broad categories of effects, three main types of actors and single RIs that can form networks of RIs, there are many possibilities of aggregation strategies, which are not all possible to conduct or are simply not relevant.
Basically, a first statement to be made is that aggregation of the different types of effects is impossible, whatever the grouping of actors or of single RIs. This is due to the fundamental differences in the intrinsic nature of the effects.
Table 3 provides a more detailed analysis of the different possibilities of aggregating the results from evaluations conducted with the EvaRIO proposed tools.

Table 3: Aggregation of effects
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

1.2.2 The EvaRIO Architecture of Metrics

A lot of different metrics are potentially usable for the different effects. The presentation of the proposed metrics follows the classification of effects presented above, and whenever relevant explains the differences in the treatment of the different actors

(i) Direct effects

Direct effects are in monetary terms, to the exception of some of the direct effects for users that cannot be ultimately expressed in such a way.

Basically, the metrics used are:

a) For the operators: budget of the organisation or sub-set of organisation hosting the resources that are included in the RI perimeter, over the period covered by the RI perimeter. Ideally, this budget should include all costs incurred for the operation of the RI (staff and maintenance included). The source for this information can be the operator itself or any official entity supporting and/or funding the operators.

b) For the suppliers: contracts awarded to the suppliers for the supply of goods and services necessary to the build up, the maintenance or the evolution (upgrades, changes etc.) of the resources that are included in the RI perimeter, over the period covered by the RI perimeter. The source for this information is the supplier and/or the operator awarding contracts.

c) For the users: as explained, there are two different sub-sets of direct effects.
The first refers to the same approach as for the operators and users, that is the evaluation of a volume of activity related to the RI. In the case of users, these activities correspond to the research carried out using the RI, in the form of projects, programmes, recurrent or one-off uses of the RI, etc. The difficulty here is that there are different types of expenditure related to RIs that can be taken into account as direct effects, exactly as there are many items that could be included in the cost of a given research project.
The approach proposes to estimate the effects using two boundaries (lower and upper limits) and to cover the expenditure for all projects that are clearly identified and are considered as not being possible without the RI. Thus the estimate is based on data available at users’ level, provided that they include the expenditure mentioned in the definition of a minimal bond.
The other sub-set of direct effects for the users corresponds to the direct advantage of using the RI (implicitly as compared to any other resource). Then the measurement could be made by measuring the gain in terms of resources used: in physical terms that can then be translated into cost savings (investment, materials, man-month…), in terms of time saved in completing the research project. All these metrics cannot be translated in monetary terms, and thus a co-existence between monetary value and qualitative or quantitative non-monetary metrics is almost unavoidable.
In any case, the sources of information will be the RI users (if possible checking with RI operators), and collecting will be mostly through interviews (with possible devising of a questionnaire-based survey).

(ii) Capacity effects

Capacity effects can hardly be measured directly, and evaluating and measuring the change in capacity of the actors who are analysed is indeed a very challenging part of the evaluation work.
We adopt an approach mostly departing from a monetary evaluation, and combining two steps.
On the one hand, qualitative evidence obtained from interviews and deep investigations of each individual actor are used to explain and reveal some of the main features of the capacity changes.
On the other, a large variety of ad hoc metrics relying on "external" data or objective observation are used. Both analyses are structured following the different dimensions of capacity defined above.
This approach then mixes qualitative assessment and different indicators and metrics, and hopefully provides a multi-faceted but structured image of the change in capacity. But it is by no means possible to add the different indicators in order to provide a unique and synthetic figure.

We provide a list of 52 possible metrics broken down into the different categories of capacity exposed above in the mapping of effects (see attached Annex 1). The list of indicators is basically the same for all actors, but of course the specificity of the role and of the nature of the different actors will affect the type of effects generated, depending on how much the changes in resources, competences and dynamic capabilities is valuable for the actors. Then some minor adaptations of the typology may be required for each of the sub-sets of actors examined, especially in the case of the operators.

The indicators were designed on the basis of the following assumptions:
- Capacity basically means pool of learning related assets (knowledge, competences, connections, routines…) and ability to use them as well as make them evolve.
- We try to measure the increase in capacity thanks to RI related activities, then all indicators (unless specified) are to be understood as reflecting a link with RI related activities; when relevant, comparison is then made between activities related to RI and activities not related to RI.
- In theory, for all quantitative variables there is the possibility to take into account:
* Total over the whole period considered
* Year by year over the whole period considered
* Sub-total by sub-periods in the whole period considered
* Year by year cumulated over the whole period considered
Total and year by year cumulated (when relevant) are used.
- In theory, increase may mean "difference in comparison with an hypothetical situation without the RI related activities" (counterfactual approach) OR "due to RI related activities" (null hypothesis); most often the null hypothesis is the only one applicable here and is therefore used.

The approach also indicates different sources of data that can be used. One source are the actors themselves, through direct interviews. Other sources of information are used: standard publicly available databases of S&T indicators (such as publications, patents or collaborative R&D projects), official/acknowledged rankings of actors (depending on the type of activities, these may be rankings of labs or firms' market positions). The possibility to complement the approach with a questionnaire-based survey is also to be considered, but was not systematically explored in the study.

The justification of the choice of the "external" sources of information is the following:

- Source for publications:
Web of Science (WoS, from THOMSON REUTERS) is mainly chosen because:
* It covers many scientific disciplines, hence the routines set up to investigate one domain/the case of one RI can be used also in other instances.
* Its limitations are largely documented, and then the limits of various analyses can be easily exposed, and if appropriate overcome by means of standardized procedures.
* It provides an optimum balance between the period covered and the amount and variety of information attached to the references.
* It allows a series of retrievals and queries, in particular regarding affiliations of researchers, which highly facilitate the analysis.
Of course other sources could be candidates, such as PubMed in the BMS field covered by the EvaRIO case studies. But although this source is widely used in this domain, and also allows connections to other sources of data, the WoS source was chosen for the reasons mentioned above.
It should be noted here that depending on the specificity of the case, different approaches and procedures were tested for identifying the relevant set of authors as well as for extracting data. These quite detailed and complex methodologies are documented in the case study reports.

- Source for patents:
The online source ESPACENET/EPO available at the European Patent Office is proposed here, for the following reasons:
* It is easily and freely accessible on line.
* It covers a wide range of patents and related documents from different sources, as well as a long enough period.
* It is likely to be maintained over the years with a high degree of continuity and traceability of the data treatment and accumulation procedures.
* It provides simple research tools for the S&T domains covered by the patents.
It should also be noted here that given the type of RIs and related activities covered by the case studies, it did not appear highly necessary to use more detailed and sophisticated sources of data, such as the classical PATSTAT, which could be more relevant in the case of other RIs.

- Source for specific S&T indicators:
Indicators particularly relevant for specific fields can obviously be added to the list. For instance in the field of BMS investigated during the case studies, large biological molecules, including proteins and nucleic acids are typical outputs of research and are registered and documented in a very standardized and easily accessible way: the Protein Data Bank (PDB) archive, a single worldwide repository of information accessible via various website including the one of the EBI.

- Source for reputation:
There is an increasing number of indicators supposedly able to take into account the visibility and the attractiveness of a given organisation measured via the web activities "around" this organisation. For instance, classical indicators could be:
* Number of links from an internet search engine
* Number of links from an external webpage
* Number of download of documents
* Number of visitors on a web page (in total or unique visitors)
* Number of visits on a webpage
* Number of pages visited
* Number of hits on a webpage
* Bandwidth used for accessing/downloading documents,...
The difficulty is that there is not yet a comprehensive and fully acknowledged source of reference for such analyses, and there are a number of service and consultant companies which provide detailed analyses, obviously not for free generally speaking.
Then one possibility is to adopt, at least for the time being, a tool that has been developed especially for the research world, and that has existed for some years now. That is the "Ranking Web of World Research Centres", an initiative of the Cybermetrics Lab, a research group belonging to the Consejo Superior de Investigaciones Científicas (CSIC), the largest public research body in Spain.
The ranking is obtained by a combination of four different indicators (which are also provided separately):
* Size, taking into account the number of pages recovered from four engines: Google, Yahoo, Live Search and Exalead.
* Visibility, taking into account the total number of unique external links received (inlinks) by a site (that can be only confidentially obtained from Yahoo Search, Live Search and Exalead).
* Rich Files, taking into account the number of accessible files of the type Adobe Acrobat (.pdf), Adobe PostScript (.ps), Microsoft Word (.doc) and Microsoft Powerpoint (.ppt).
* Scholar, taking into account the number of papers and citations for each academic domain provided by Google Scholar

- Source for EC projects:
Cordis database and research tools are undoubtedly the most relevant and easily accessible source of information about EC projects. Despite its limitations (lack of some key information such as cost per partner, missing information, poor relevance of some information about partners such as contact persons, etc.) this is the most commonly used source for such analyses.
But data have to be taken with caution, and have to be cleaned and even cross-checked with EC projects partners (whenever possible) to be used.

(iii) Effects on performance

For this category of effects, the indicators cannot be the same for the three types of actors. Actually, the central focus of this category of effect is the performance of operators. For the users and for the suppliers, performing their RI-based activity better will be integrated in the direct effects: suppliers may be more efficient and supplying goods and services to the RI, which will be ultimately reflected in new contracts; users will be able to get more direct gains from using the RI (for instance, they will be able to spend less time in preparing their experiments, to better focus their analyses or to enrich the treatment of the results, to find the data or samples corresponding to their needs more quickly, etc.).

But there is a need to develop specific metrics for the operators. Actually, most if not all operators have developed various types of scoreboards and KPI (Key Performance Indicators) systems, reflecting the dimensions of:
- Cost/pricing (including for instance reimbursement of travel and accommodation costs)
- Availability of resources
- Time for response to enquiries
- Time required for accessing services
- Quality/reliability/precision etc. of resource offered to user
- Provision of information on availability of resources
- Provision of protocols related to experiment
- Scientific, technical, administrative and logistic (office space, computing, libraries, accommodation…) support
- Confidentiality
- Number of users / of unique users
- Rate of openness / service (as compared to maintenance period)
- Rate of acceptance, oversubscription rate
- etc.
Very often, those types of metrics are based either on objective measurements or surveys about user/consumer satisfaction.

However, more sophisticated and specific metrics would be required to finely express the performance of the RI and of operators of the RI as contributing to the advancement and evolution of scientific knowledge as well as of the scientific community. In particular, it had been envisaged to further investigate dimensions such as:
* The evolution in the number, variety/diversity, renewal of resources and services provided by the RI (including experiments, training, access modes)
* The evolution in the number, variety/diversity, renewal of users and user communities.
Unfortunately, beyond many qualitative insights from the case studies and the identification of some potentially promising avenues of research for the development of such metrics, it was not possible to derive a significant and comprehensive architecture of indicators able to take into account these dimensions. But again, each RI case would require a very specific version of the system of metrics along the general lines sketched above.

(iv) Indirect effects

Indirect effects are expressed in monetary terms, measuring the economic importance of each effect observed.
The framework for defining and analysing these indirect effects also follows the general framework presented above (i.e. the distinction between Science & Technology, Organisation & Method, Network and Reputation dimensions), since these are the dimensions of the capacity that are exploited. However, some refinements (e.g. in the sub-categories of effects) may be introduced in order to cope with the specificities of the S&T context (field of research, nature of the output, nature of relations between actors…), e.g. the context of BMS research.
It is assumed that most types of these effects can be expressed in economic terms of added value generated by revenues and cost reductions, which, in turn, were achieved as a result of the knowledge gained by the actors during the RI-based activities. More precisely, the list of possible indicators is the following:
* Sales of products
* Sales of processes
* Sales of services, including training
* Cost reductions: in terms of resources (materials, instrumentation, …) and/or time (access to collection, partners seeking, …)
* Research funding (contracts, grants, etc.)
* Revenues from awards, prizes, donation etc.
* Revenues from licensing, royalties
* Revenues from spin-offs

Whenever an effect is observed, an attempt is made to perform such quantification. But it is not possible in all cases, and quite a significant number of identified effects generally remain unquantified. Accordingly, S&T effects basically lead to the design of new or improved products, processes or services, which allow the actors to increase sales, to protect market shares, to benefit from a reduction in costs, or to win new research contracts. Network and Reputation/label effects lead to new sales or research contracts, and Organizational, Method and Process effects lead to cost reductions.
When the use of S&T or method related knowledge is only partly responsible for increased sales or cost reductions, the value of the corresponding effect only amounts to a share of those sales or cost reductions. This share is in proportion to the influence on those sales or cost reductions of work performed within the framework of the evaluated RI related activities ('fatherhood coefficients' are thus used).
Whenever possible, the effects are compared to the effects that would have been generated in the alternative and hypothetical situation in which the RI would not have been available. The counterfactual is then considered at the micro-level of the actors themselves.
Most of the effects are evaluated ex post, corresponding to the focus of the indirect effects on "actual impact". However, depending on the context, a limited forecasting period could be taken into account, typically a 2 year forecast period.

The measurement of these effects performed by the BETA group must be considered as a minimum estimate for the following reasons. All estimations of figures provided by actors are systematically minimized. That is to say that these estimations are expressed mostly as a range from which only the lower boundary is used for computation. Some effects cannot be measured, for instance, because the influence of the RI related activity exists but cannot be separated from other factors. In spite of the time spent in interviewing people, some cases may also escape the interviewers, for instance when the technical aspects are very complex or when the actor has ‘forgotten’. Finally, in spite of the guarantee of confidentiality provided to all actors interviewed, they may still be reluctant to disclose strategic information.
The source of the data is the actors themselves, through direct interviews (see other section of this report). The possibility to complement the approach with a questionnaire-based survey could be explored, but past experiences lead us to very cautiously envisage the full success of such an approach for the BETA method.

1.2.3 The EvaRIO Interview Protocol:

In this section, we briefly present the general protocol that should be followed for conducting case studies using the EvaRIO approach. The choice of the case studies performed by the BETA team during the EvaRIO case study part of the project is reported separately in the next section.

(i) Protocol: Methodological approach for the case studies

As explained above, interview plays a key role in the collection of information. At least 15 to 20 interviews have to be carried out for an in-depth case study. Each category of actors should be covered via a sample of organisations which should be representative of the whole population of the corresponding category. The criteria of representativeness of these samples are to be defined according to the specificity of each RI case.

Interviews are conducted by 2 experienced members, and last about 2 hours. Each interview is recorded and leads to a confidential report. Internal debriefing meetings are regularly organised afterward to present, share and discuss the empirical material obtained. Contact with interviewees is frequently maintained in order to gather further clarification or information (especially quantitative) on specific points not fully covered during the interviews.

The interview implementation includes generally three phases.

The first step consists in several framing interviews at the level of RI officials, at the operator’s level. This exploratory step is a critical one for the continuation of the case study. This is especially true if we consider the high variety and complexity of the cases analysed in the framework of EvaRIO: large RIs which differ radically in terms of the resources they provide access to, but also in terms of operation, organisation, nature and history. So the main goals at this stage are to provide a deep and detailed understanding of the RI context including its involvement in ESFRI projects, and to identify a sample of key actors to be interviewed in the subsequent steps: RI staff, users, builders.

More precisely the key objectives of the framing interviews with RI officials are:
- To gain a detailed understanding of the origin (history, key persons, main steps, previous collaborations) and of the current operation of the RI (organisation, access rules, user community, industrial relationships). This is crucial for helping elaborate the specific indicators, identify the most relevant "external" sources of information for indicators, and ultimately interpret the results and identify the limits of the evaluation exercise.
- If required in the case of network of RIs or distributed RIs, to operate a selection of relevant sub-cases, i.e. centres or RIs which are also members of the RIs. Since evaluating the whole distributed RIs is a difficult task, the sub-cases are intended to allow for at least a partial evaluation. So candidates for further investigations and for applying/testing the method have to be rigorously selected.
- To help select the criteria of representativeness of the samples of users and suppliers, provide necessary information for this sampling, and/or giving advice on the most relevant and interesting actors to be interviewed.
- To assess the degree of feasibility of the evaluation based on the EvaRIO approach and the potential difficulties of implementation. For instance, grasping ex post realized effects might cause some timing problems. Another possible source of difficulty relates to the variety of interviewed actors and hence of subjects of analysis.

The second step focuses on the identification of effects resulting from RI creation and/or operation. This time consuming phase requires conducting numerous interviews - between 15 and 20 - at the level of three types of actors:
* Some five to ten RI users (academic but also industrial ones) - whenever possible;
* A couple of RI builders (especially industrial suppliers and in some cases academic contributors for building the resource) - whenever relevant;
* A couple of interlocutors at the level of the RI operator (scientists using and operating the RI service, as well as people responsible for TTOs (Technology Transfer Offices) and/or valorisation services.
The effects that are systematically investigated concern the experience gains/ new skills/ contacts generated while using, building, operating the RI. They can be classified in different areas: Science and Technology Effects (including publications, patents, etc.), Network Effects (number and quality of links, partners, etc.), Organization and Management Effects (procedures, project management, quality, certification, etc.), Reputation Effects (scientific awards, rankings, etc.), and Human Capital Effects (PhD students, post-doctoral researchers, other personnel hiring).

The third step is devoted to quantification of effects. It consists in quantifying some individual effects, by using or developing ad hoc indicators, some of them being defined in monetary terms whenever possible. In some cases it was also possible to analyse and/or anticipate the connection between the evaluation of the RI per se and the additional impacts due to the ESFRI project the RI belongs to.

(ii) Protocol: interview guidelines

Identification of effects is based on guidelines of interviews including a list of topics. It does not rely on fully structured, precise and closed questions because exploration is needed and can only be obtained through a discussion with the interviewees.
The discussion contributes then to feeding the identification of effects and the specification of possible indicators, i.e. to filter, select and test the most relevant ones in the EvaRIO predefined lists, as well as to enrich and develop new ones.
The guidelines for each type of interview are provided below.
Information is then supplemented by inputs from other sources such as websites or scientific articles and reports related to the topic covered.

In parallel in-depth collection of information is conducted in order to feed the completion of the evaluation of the specific metrics developed during the approach. Here the following sources are used:
• ISI Web of Science,
• PDB (Protein Data Bank, European Patent Office Espace.net
• EU Cordis database,
• Factiva (press archive),
• CSIC Ranking Web of Research Centers.
• We also designed and conducted our own EvaRIO survey in the case of EBI's trainees (an interesting category of EBI's users).

***********

- Interview guidelines: Framing interview (RI officials)
1. Presentation of BETA and EvaRIO evaluation approach
2. Interview Topics
History of the RIs XX (and its host organization): initial context, main dates and steps (creation, first operation, significant change,...) strategic background, role of EC,…
Mission and current operation: mission, type of resources, instruments and services provided,…
Access rules: access rules and criteria for RI access: open, very selective, market mechanisms, importance of remote access, specific rules for specific users…
Characterising the users: existence of different profiles/communities, place and % of industrial users, experience learned back from users, existence of sources of information on users, impacts of visitors upon the region
Relationships with industry: co-development of instruments with industry, TTO (technology transfer office), patents and licences (MTA), spin-offs, …
Connexion with other RI integration and/or ESFRI projects: role of XX and history of the RI integration project considered, relation with other members, realized/expected impact of participating in these RI
3. Discussion about the possibility to explore further the case of XX : relevance in the context of EvaRIO, selection of additional contacts for further interviews, at XX, among external visitor teams using XX facilities and among industrial providers.

- Interview guidelines: Effects on Operator (Analysis of benefits and knowledge creation for the RI XX)
Interview topics
1. Budget for operating the RI and staff
2. Evolution in the number/variety/renewal of resources and services provided by the RI (including experiments, training, access modes)
3. Evolution in the number/variety/renewal of users ad user communities
4. Experience gains / new skills / contacts generated during operation of RI XX, in different areas: Science and Technology (including publis, patents, prototypes,...) Networks (new partners, quality of links,...) Organization and Management (project management, certification,...) Reputation (prize, rankings,...) Human Capital (PhD students, post-docs,...).
5. Valuation of gains in experience / new skills / contacts acquired through operation of RI XX, when they are reused in the same RI XX: gain in time / cost
6. Valuation of gains in experience / knowledge / contacts outside RI XX (i.e. valorisation/TTO activities), in particular the development of patents, products, services, processes,... measured in revenue (sales, royalties, new contracts, scientific prize, ...) and cost savings / time / quality.

- Interview guidelines: Effects on User (Analysis of benefits and knowledge creation for a given user due to the use of RI XX)
Interview topics
1. Presentation of the user, history of relations with RI XX and rapid introduction of the project(s) carried at/with RI XX, scie & tech issues, social and economic issues.
2. Existence of alternative resources (other instrument) to perform the experiment and comparative advantage of RI XX (time saving, RI as a necessary condition,...)
3. Reminder of the different ways of access to RI XX with comparative advantages
4. Experience gains / new skills / contacts generated during a visit to/ a use of RI XX, in different areas: Science and Technology (including publis, patents,...) Networks (new partners, quality of links,...) Organization and Management (project management, certification,...) Reputation (prize, rankings,...) Human Capital (PhD students, post-docs,...).
5. Valuation of gains in experience / new skills / contacts acquired in a first use of RI XX, when they are reused in the same RI XX or another RI of the same type: gain in time / cost when making experience, or raising project funding for a new project
6. Valuation of gains in experience / knowledge / contacts outside RI XX (or outside RI of the same type), in particular the development of new research projects, or products/services/process by the user, measured in revenue (sales, royalties, new contracts, scientific prize, ...) and cost savings / time / quality.

- Interview guidelines: Effects on Supplier (Analysis of benefits and knowledge creation for a given supplier due to the provision of goods/services to the RI XX)
Interview topics
1. Quick introduction to the type of supply to the RI XX, history (incl. developmt and commissioning), scientific and technical issues, social and economic issues
2. Contract amount, rate of added value on these contracts, share of external purchase, amount of wages paid on these contracts
3. Experience gains / new skills / contacts generated during the execution (R&D, production, delivery, co-development,...) of the supply contract to RI XX, in various fields: Science and Technology (including publis, patents,...) Networks (new partners, quality of links,...) Organization and Management (project management, certification,...) Reputation (price, rankings,...) Human Capital (PhD students, post-docs,...).
4. Valuation of gains in experience / new skills / contacts acquired in a supply contract, when reused at the occasion of a new supply contract to the same RI (amount of new contract, saving time / cost when making the supply)
5. Valuation of gains in experience / knowledge / contacts outside the RI XX, including the transfer of the R&D done "for" the RI XX to other activities, products, services, process, as measured by revenues (sales, royalties, new contracts,...) and cost savings / time / quality.

2 Main results of the EvaRIO Case Studies

About ten case studies were conducted in order, first, to test the evaluation methodology described above and, second, to feed the Focus Studies described in the next section. They gave rise to several lessons regarding the implementation of the EvaRIO evaluation method.

2.1 Selection and achievement of case studies

As described in the Deliverable 4.4 nine case studies were carried out: 4 in-depth ones following the protocol just describe above, and 5 smaller ones to complement them on specific points.

The priority criteria for selecting the case study candidates were specified as follows:

The RI cases had to belong to the BMS field, especially the two major topics of structural biology and biomedical data. They had to be in full operation several years before applying the BETA EvaRIO evaluation approach, because the latter is typically an ex post approach. The RI candidates also had to be involved in ESFRI projects in order to derive additional indicators or at least grasp possible impacts of ESFRI programmes upon the existing RI landscape. Due to its focus on instrumentation INSTRUCT was of special interest for the purposes of the present evaluation exercise, and to a lesser degree ELIXIR, ECRIN and INFRAFRONTIER.
As a whole the set of case studies was to reflect as much as possible the diversity of RI situations. First it had to include the different types of resources RIs provide access to, which entails not only large facilities and instruments but also collections of biological material, biomedical data and software as well as human resources and competences. Second it had to encompass a variety of organizational statuses and spatial-temporal borders, going from longstanding stand alone RIs to facilities or instruments belonging to a larger entity (a lab) and opening as an RI.

Two in-depth case studies included facilities corresponding to the core technologies for structural biology and were connected to INSTRUCT (and/or BIOSTRUCTX): the CERM in Florence, which provides access to NMR, and SOLEIL synchrotron beamlines in Saclay (especially but not exclusively X-ray cristallography). The third in-depth case study was about EBI in Hinxton, because it deals with the key issue of biomedical data and because of its involvement in the ESFRI programme ELIXIR. The last one was EMMA, an archive of mutant mice which is now part of INFRAFRONTIER, and which represents an interesting case of distributed network together with an important type of resource in BMS: a collection of biological material (EMMA changed its status from a small to an in-depth case study during the last phase of EvaRIO interview implementation. It provided a highly complementary case of RI in BMS, making it possible to cover much better the wide diversity of BMS RIs. It substituted for the case of electronic microscopy platform of MPIB in Munich, as the latter was less and less involved in INSTRUCT at that time).
The small complementary case studies were the following: EORTC was chosen as a kind of model for evaluating the impacts of ECRIN; MPIB in Munich, OPPF in Oxford, I2PC-CSIC in Madrid and IGBMC in Strasbourg-Illkirch were selected as additional INSTRUCT centres, covering other organisational situations and resources than CERM and SOLEIL. More precisely, MPIB provides another core technology for structural biology (Electronic Microscopy). OPPF is a founder and a core centre of INSTRUCT providing a complementary technology, i.e. a protein production facility. I2PC represents a case of creation of a new INSTRUCT centre (associated with MPIB) and working in the field of image processing. Finally, IGBMC has been distinguished because it is an interesting case of multi-platforms RI configuration, whereas the previously mentioned RIs provide rather specialised resources.

The Table 4 summarizes the coverage of selection criteria on a case-by-case basis.

Table 4: List of RI cases according to the main criteria of selection
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

It should be stressed that all metrics and indicators were not tested in all case studies; different case studies were used as an experimental field for different tools and metrics; specificity of the cases and availability of required data were used as a basis for the choice of the optimal matching.

76 interviews were conducted: 60 for the in-depth case studies (15 on average per case) and 16 for the smaller ones (about 3 on average per case). Interviews were carried out by two senior members of BETA, and lasted approximately 2 hours each.

With respectively 25 and 17 two-hour face to face interviews involving all the types of actors, SOLEIL and CERM were undoubtedly the most complete and extensive case studies. Consisting in large facilities open to external users from their inception, they can be both considered as the most typical cases of RIs. They were also the most demanding ones in terms of hours of interview when using our micro-based and retrospective approach.
In both cases it was possible to:
- Understand the specific context of the RI (e.g. for SOLEIL: BMS synchrotron beamlines and their crucial role for structural biology, the conditions for users' access, the operation of synchrotron X-ray cristallography and finally the articulation of X-ray beamlines to BIOSTRUCTX);
- Identify many effects at the level of the operator, in connection with their internal and external valorisation;
- Select and/or contact many users, among which we have not only academic but also industrial users, allowing us to test our typology of effects and our evaluation method when applied to this group of actors;
- Select and contact several industrial suppliers, especially in the instrumentation sector, allowing us, again, to test the typology of effects and their quantification for these specific companies.
Also, in the case of SOLEIL, we had the opportunity to collect much information about the community of BMS users for the Focus Study "Network", regional impacts and flexibility.

The two remaining in-depth case studies, EBI and EMMA, show similar sizes and interviewees (around ten interviews with a clear focus on the operator side). They also share similar problems as far as evaluating impacts on suppliers and users are concerned. Industrial suppliers provide standard equipment (racks of memory and non specific electronic devices for EBI, freezers for EMMA) so that interviewing them was not considered a priority. At the same time RI users are particular in the sense that:
- They are difficult to identify;
- They are also "builders" of the RI biomedical resources, and thus they contribute to an "open source like" dynamics of knowledge creation.
As for the first point, anonymous users make the evaluation exercise difficult, unless the evaluators organize a short, customized online survey with the assistance of the RI operator him/herself (who is the only one being able and authorized to send an email to a set of users). Such a survey was designed and tested in the case of EBI trainees (a particular kind of highly motivated users) with a rather satisfying number of respondents. Interesting interviews could also be conducted with another kind of users: two industrial companies using EBI resources, databases and tools either for their internal R&D, or for feeding service provision (broker role).
To exemplify the second point EBI operates as an open data and open access distributed RI, where RI users should also be considered - at least collectively - as the main RI builders. They directly contribute to enriching the resource, enlarging the variety of data and services. Similarly EMMA users do not only order existing mice for their experimentation, they also deposit the lines they create in the archive in order to keep them available. Exploring knowledge creation dynamics in EMMA was the core topic of EvaRIO Focus Study about "open source like" enrichment process.

Turning now to the small case studies, EORTC, the network of clinical trials used as a substitute case for ECRIN, is rather close to EMMA / EBI in as much as it experiences mixed roles and a kind of enrichment dynamics. The main RI builders are the users since they contribute directly to the resource: extending the diversity of the archive / the database by adding a mutant mouse / DNA or protein data, in the previous cases; extending the diversity of the patients base and providing highly specialised clinical competences, in EORTC. In sum, the aim of such distributed RIs is to provide a unique access to the richest set of resources possible, according to a logic of agglomeration of diverse but similar components (following a rationale of increasing variety).
OPPF (like MPIB) gave rise to only one framing interview, the aim of which was to control for the specificity (or not) of the nature of the RI resource considered when evaluating the impacts (here a new type of service consisting in the production of biological samples).
The small case study about IGBMC concerned an existing multi-instrument platform (called Centre de Biologie Intégrative - CBI) hosted by a research centre. The platform was open to external users at the national level before being offered in the framework of INSTRUCT. Due to the very recent launch of calls for access via INSTRUCT, there have been no INSTRUCT users at CBI until now. Thus we had no interview with this type of actors. Nevertheless, the CBI was open to national users and the case provided interesting elements about the opening of a multi-purpose platform and/or the lab which hosts it. Moreover a specific interview gave interesting insights into the regional impacts of the CBI.
I2PC is a service entity hosted by CNB-CSIC in Madrid. It was created recently as an INSTRUCT centre. Like IGBMC, it is an interesting case for the study of RI regional impacts. It was possible to conduct two interviews with the operator, and these interviews were followed by interesting follow up exchanges.
MPIB is an INSTRUCT centre providing access to a core instrumentation (Cryo-Electronic Microscopy, referred to as cryo-EM). The cryo-EM facility was not used as an open RI in the past, but it used to belong to a research centre (MPIB) for its internal needs only. It opened as an RI specifically with the creation of INSTRUCT.

All the case studies led us to considering carefully the systems and sub-systems at stake, and to defining the perimeter of the evaluation study accordingly. Especially in the last three cases, it appeared relevant to assess the effects of opening the platform(s) as an RI not only on external users, but also on the research lab which hosted the RI. It should be noted that such a configuration is of particular interest for EvaRIO since it increasingly develops in the field of BMS, and more generally with the new ESFRI infrastructure programme.

2.2 Lessons drawn from the case studies as regards the implementation of the EvaRIO evaluation approach

As regards the evaluation of effects, even if it is impossible to provide a comprehensive view of a series of purposefully diverse case studies, the main following results can be highlighted (Cf. Deliverable 5.2 - Part I for details):
- The confirmation of the richness and variety of impacts generated at all levels and of all categories
- The close relation between the relative importance of effects and the specificities of the RI case (nature of resources, organisation, history …);
- The difficulty to grasp the full size of the direct effects
- The fundamental importance of Capacity effects, especially the S&T and networking ones
- The importance of S&T and network components for operators and users, and of reputation for users and suppliers, be they studied within the capacity or the indirect effects areas
- The relative weakness of indirect effects "on the market", i.e. real short term commercial exploitation of new scientific discovery or technological development for other actors than the suppliers
- The relative weakness of indirect effects "internal" to the users
- The fundamental importance of personal and close relationships/trust on generation of most of the effects, which incidentally raises the issue of the relevance of pure remote access
- The relation between generation of effects and strategies of suppliers (highly specialized/customized vs standard supply) as well as their dynamics
- The relation between generation of effects and "patterns" of relations between users and operators (long term collaboration vs occasional) as well as their dynamics.

More globally and from a more methodologically oriented perspective, the case studies revealed that the approach undoubtedly makes it possible to trace a very important and rich mass of phenomena, effects, mechanisms and in that is really highlighting "research as it works". Moreover there is a strong need of cooperation with actors for the method to be fruitfully implemented; promisingly, interviewees stressed the originality of the approach because it deals with details, history, different dimensions of learning, etc..It is worth noting also that the choice of a relevant time lag between RI related activities and evaluation is very important for implementing the approach, which is clearly an ex post one. Finally, the complementarity with other evaluation approaches is confirmed.

Some potential improvement areas have also been identified. For instance the approach should most probably better include the coverage of other actors such as the ones involved in design, decommissioning,.. Another improvement to be done relates to the analysis of the effects on performance all the more so since the avenues open by the approach in that matters prove quite relevant and promising. Finally a higher coherence and complementary could be reached with the insights from the dedicated EvaRIO Focus Study as regards the key networking dimension of the RI impact.

To conclude, it has to be acknowledged that the approach nevertheless suffers from some inherent limitations, in particular:
- The practical difficulties in defining a relevant and most of all unquestionable RI perimeter;
- The difficulties to achieve the whole set of quantification and data collection (a lot of details to find, solve…);
- The "other side of the coin" of the individual actors focus, which appears through different forms: 1) some actors are difficult to address (for instance Users of on-line resources, Suppliers of standard equipment, Suppliers and Operators of competences/expertise, Suppliers of the RI at networking stage); 2) there is often a mix of roles and multi-roles across actors 3) it made aggregation/summing up of effects quite complex and potentially difficult to show in a easy and simple way and most of all, 4) the individual appropriation of effects (often via market mechanisms) that is implicit to our actor-centred approach is sometimes in contradiction with the collective/community mechanisms (often associated to science) observable in the RI related behaviour and incentives of actors;
- Last but not least, the limitations in the coverage of the importance of training activities and their impacts.

3 Main results of the Focus Studies

The completion of the four Focus studies lead to very interesting and promising results, while being very diverse in terms of issues covered and scope of analysis (Cf. Deliverable 5.2 - Part II: EvaRIO Focus Studies, for detailed presentations)

3.1 Focus Study 1: An open source approach to RIs - The dynamics of knowledge creation via open source research infrastructures

Three main results were obtained.

First, on the theoretical side, we provided a clear definition of the open source phenomenon and explained how it could offer interesting insights for the understanding and management of RIs.
In the RI context, open source can be seen as relying on two main dimensions:
- Condition of access. First of all, an open source process entails free (i.e. non-discriminatory) access. Every potential contributor can enter the process if he wants to. In the case of software, for instance, open source means that everybody can access the source code, modify it and distribute it. Paradoxically, this open dimension is often secured by intellectual property rights (i.e. copyright or patents). Open source participants can indeed rely on IPR in order to force contributors to distribute their modifications and improvements under the open source regime.
- Community-mode of knowledge production. Second, open source follows a communitarian process of production. Participants share similar norms and objectives and frequently interact and exchange knowledge with each other. It is therefore (at least in theory) a deeply interactive and community-mode of production.
In the case of RIs, conditions of access can concern (1) the RI itself (what are the conditions for organizations and researchers to use the RI); (2) the knowledge produced by using the RI (Is it published? How easily can other researchers re-use this knowledge?); and (3) the product developed using this knowledge (Is it patented? Possible to imitate? etc.). For each of these elements, access can be controlled (private and exclusive); open without restriction (public); Copylefted (open source); or hybrid. It is important to notice that an open source approach to RIs does not mean that the access is always under copyleft rule. In a more pragmatic way, it means that openness must be secured as soon as it is compatible with incentive considerations.
RIs can be viewed as providing a platform for scientists and industrialists in order to develop knowledge and innovations. In an open source setting, as opposed to more traditional ways in which each user can use the RI independently of the others, users intensively interact, not only with the RI but also with the other users, thus generating a community or bazaar use of the RI.

Second, on the basis of the preceding point, we designed theoretical propositions which were tested on the case of one RI, namely the EMMA case, on the occasion of the in-depth case study conducted on this RI. The propositions and the main empirical results are summarized in Table 5 below

Lastly, some conclusions and lessons for further research can be derived from the analysis.
1) The OS approach leads to focusing not only on the outputs of RIs but also on their condition of re-use (open versus exclusive). When evaluating RIs, the conditions of re-use may hence bring an element of differentiation between RIs which would differ not in the output (equal number of publications or patents for instance), but in the constraint put on the re-use of those outputs. The idea is that openness has a higher value for society than exclusivity.
2) The OS approach leads to focusing on collaborations, exchanges and interactions as important outcomes of RIs. The development of a community of partners may hence be a fundamental outcome of RIs for users, which yields only outcomes in the long run, but should not be neglected by evaluators.
Furthermore, an interesting approach, which has not been extensively developed here, is to view OS as being in itself a meta-RI, i.e. a RI with a higher degree of abstraction. OS is more than a process of production. It is a philosophy, a research methodology, which is available to all (interestingly, on this point, one can note the many similarities between open source and open science). In other words, OS performs a function similar to the one of RIs, but at a different level. It helps researchers and innovators develop a research methodology, thus contributing to boosting the innovation process. Acknowledging this property of OS has huge economic implications: First of all, from a managerial perspective, it means that firms must incorporate this aspect of OS in their strategy. Like RIs, OS in most cases does not threaten firms, it does not compete with them, but complement them. Second, from a policy point of view, it means that sustaining this role of OS must be fully acknowledged by policy makers and eventually encouraged. At the European level for instance, RIs are an important cornerstone of the research policy. Why hence not include OS within this RI policy?

Table 5: Test of theoretical propositions in the EMMA case
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

3.2 Focus Study 2: Flexibility of Research Infrastructures: The case of the French Synchrotron SOLEIL.

This focus study provides a detailed analysis of the mechanisms by which a large Research Infrastructure (RI), here a synchrotron, maintains and manages its flexibility in a highly S&T intensive and uncertain environment.
The Focus Study had the objective to examine the flexibility and development of RIs through the real option lens perspective. This perspective was judicious since it envisaged the dynamics of RIs as principally emerging out of the need to manage uncertainties. However in the course of our study, the very rich input provided by the many interviews with RI actors and users led us to reorient our specific analytical approach in order both to better value our information base and better support our initial hypothesis related to the flexibility of RIs.

As an RI, synchrotrons are capital intensive and complex research technology systems, but they also are subject to rapid obsolescence self-sustained through the perpetual creativity and innovation characterizing the activities of scientific and engineering communities dedicated to their design, operation and development. We hypothesize that flexibility is a central performance feature of synchrotrons and explains the mechanisms that support it in a highly dynamic and uncertain environment. In fact this flexibility is supported and sustained by the dynamic resolution of several tensions at different levels of a synchrotron which refer both to its internal structuring mechanisms and its positioning within its wider external environment composed of the ecosystem / community of synchrotrons. Our analysis relies on the study of the French synchrotron SOLEIL.
Our study specifically explored how the process of RI creation unfolded dynamically given the uncertainties faced by RI actors. We developed and implemented an analytical framework that allowed us to understand how, from its conception, an RI incorporates the properties of an evolving system allowing participants to regularly factor in new requirements and technology into its design and implementation. We clarified as far as possible how RIs manage the tension between irreversible investments and the need to adapt to their environment through flexibility. In fact RIs have to resolve the tension between, on the one hand, reducing uncertainty by minimizing and controlling risks with respect to the dynamic and collective performance of research technologies and instruments through reactivity, reliability and cooperation and, on the other hand, cultivating uncertainty in order to widen the explorative capacity of scientific activity and its potential to create future opportunities through proactivity, versatility and competition.
We showed that as RIs, synchrotrons resort continuously to a delicate balance between different managerial mechanisms in order to resolve different types of tensions that contribute to their flexibility. These mechanisms relate on the one hand to exploitation and exploration efforts and on the other hand to technological and (intra/inter)-organizational differentiation and integration dynamics.

Our approach distinguishes different levels of analysis of the synchrotron and its environment, at which different tensions are managed by those two types of mechanisms.

The first is the internal level of a synchrotron (i.e. the beamlines and the accelerator complex which are co-constructed and co-evolve in the sense that their innovation dynamics and capabilities are tightly linked), where two tensions are central.
The tension between reactivity and proactivity relates to the need of synchrotron scientists and engineers both to satisfy efficiently the needs of the existing user community and to create and pioneer new scientific experiments. The tension between reliability and versatility reflects the necessity for research instruments on the one hand to improve standards through precision, accuracy and reproducibility of scientific results and on the other hand to increase their diversity and variety generation potential.
Demand pull factors at the beamline level (user community or beamline team) motivate and orient innovation trajectories at the accelerator complex level: the need of synchrotron scientists and engineers both to satisfy efficiently the needs of the existing user community and to create and pioneer new scientific experiments. On the other hand, technology-push factors at the machine level create new potential experimentation and thus innovation possibilities for beamlines.

The second level is the eco-system to which the RI belongs and that the RI uses in order to sustain and feed its own evolution dynamics. At this level, we focus on the tension between cooperation within the synchrotron community and competition between synchrotrons, which are structuring the interactions within the wider synchrotron community or ecosystem. We showed that it is necessary to enrich the analysis concerning flexibility by taking into account this higher synchrotron ecology level which is both the determinant and the result of the individual positioning of synchrotrons and their collective interactions. The ecosystem helps us better understand the innovation and evolution dynamics of synchrotrons. Here integrative mechanisms such as inter-synchrotron standardization efforts, the creation of multi-facility user platforms, best practice diffusion mechanisms within the synchrotron community, the coordination and pooling of research efforts all aim to improve the collective capability for exploitation and exploration. On the other hand differentiative processes leading to distinctive exploitation and exploration efforts motivate excellence, specialization at the individual synchrotron level and ensure variety and diversity at the collective ecosystem level. Through loose coupling the synchrotron ecosystem contributes to collective flexibility for the benefit of the whole scientific community, and in so doing also creates flexibility enabling conditions for each synchrotron facility taken individually.

3.3 Focus Study 3: RIs as catalysers of collaborations and network

Large-scale research infrastructures (RIs) are emphasized in European Science & Technology policy as a means for structuring the European Research Area (ERA). This political objective is grounded in the systemic perception of science, technology, and innovation. The belief that knowledge and innovations are most fruitfully considered as outcomes of a social process of interacting agents is also widely shared among academic scholars of the subject.
Despite the widespread theoretical acknowledgement of the importance of networks and traditional involvement of public bodies in the set-up and operation of large RIs, the effect of large RIs on research collaboration networks has not been investigated yet. Indicators for evaluating the effect of RIs on research networks are missing.

The focus study at hand provides in-depth case studies on the network effect of two RI components and discusses the appropriateness of (extended) social network measures as indicators for RI evaluation. We discuss to what extent such indicators capture direct as well as indirect effects of RIs on individual scientists’ collaborations as well as on the entire research network. Interpretation of network effects is based on the theory of knowledge economics. Our analysis aims at exemplifying how evaluation studies might benefit from social network analysis (SNA), and points out several pitfalls that should be avoided.

The two cases that we investigate are two beamlines at the synchrotron SOLEIL near Paris. Several data sets are leveraged for the analysis. The list of researchers operating the synchrotron as well as the list of users, in combination with their scientific publications, need to be obtained in order to calculate the proposed indicators. Publications are used to construct the longitudinal co-author network among scientific agents that are involved at the RI. Network effects of the RI and its personnel are captured through descriptive and simple inductive network statistics. In-depth interviews are used to assess the validity and explanatory power of the indicators.

We consider network perturbation analysis to assess direct network effects of the RI and we investigate the value of two brokerage measures as indicators of how RI operators affect knowledge flows by linking third parties directly and/or indirectly. We show that interpretation of the brokerage indicator as an indicator of connecting diverse knowledge sources is not always warranted and therefore argue for an extension of the brokerage measure that takes into account explicitly the knowledge background of connected agents.
Related to this issue is the implication of sampling actor-induced sub-networks on the interpretation of the brokerage measure. We argue for an interpretation of RI research networks as sub-networks (or large network components) that add more or less structure to the larger, unobserved research networks of whole scientific communities. In this sense network brokerage and network fragility from perturbation analysis indeed are valid indicators of behavioural additionality in network formation.

The above analysis and proposed indicators seem especially appropriate to enter into discussion with RI stakeholders about the contribution of large RIs for structuring the ERA, as long as all discussants are aware that the chosen indicators present a very pointed picture of reality.

3.4 Focus Study 4: Impact of research infrastructures on regional development

The study does not provide any evaluation per se of the impact of RIs at regional level, with corresponding measurements and figures, but rather highlights the difficulties and challenges for such an evaluation and proposes some original guidelines for future evaluation in this field. The conclusions were drawn from the collection of information from all case studies (which all included at least partially some questions about the regional dimension of effects) and in particular from the in-depth case studies SOLEIL and the two small case studies IGBMC/CBI and I2PC (a part of the Focus study report being devoted to those three cases).
(i) Challenges and difficulties for the evaluation of regional impact

The interviews did not easily reveal regional impacts (exhaustively, in a systematic and comparable way…). Assuming that the operators could transmit their observations about the RI’s impact on the “territory”, we asked for such perceptions. The result was rather disappointing. Most RI actors as defined in the EvaRIO project (even the operators) are not enough prepared to collect and provide a complete set of elements allowing the evaluation of the local impact, and do not make a systematic review of the interactions.
More globally, direct and short run impacts are the easiest to define and to measure, but they are probably the least interesting.

Basically, three main reasons could be put forward for explaining the difficulties of the exercise.

The first is the issue of the relevant perimeter. The perimeter is not a simple data for the study, it is part of the issue. It depends on the size of the RI, but more importantly on the size and the nature of the territory. An infrastructure established in a middle-sized agglomeration may be relatively isolated and therefore it is easier to build a model of its local impact than in the case of an infrastructure established in a larger urban area which is like a complex ecosystem, with many other “species” playing complementary or competitive roles. The notion of natural perimeter of an RI is therefore strongly connected with a rich multi-dimensional concept of territory. The probability or intensity of knowledge spillovers is less explained by the distance in kilometres than by organizational issues, in particular the modes of coordination between the actors. In this conceptual framework, regional innovation policies should be considered not only as supporting actions towards individual knowledge producers, but as strategies for revealing, networking and exploiting cognitive resources – for creating a “territory effect”. The right perimeter may then be the geographical and institutional setting of such a governance system.

Secondly, an extremely large and diverse set of possible impacts could be taken into account at regional level. In this perspective, a priori economic variables (expenses, revenues, employment…) are of course interesting and relevant, but are only one aspect of the issue, sometimes not the most important one. As compared to ordinary impact studies, the very nature of RIs implies taking into consideration evolutionary aspects, political visions, organisational and institutional settings….

The third issue, highly related to the previous one, is that the impact is not an object to be found on the field; it is a construct, built on the basis of the perception of the actors, of the researchers… Before evaluating ("attributing a value"), we need to know who cares about having an RI on the territory and for what. Evaluation methods, particularly in socio-economic fields, must avoid naïve theoretical assumptions: the reality to be measured is not an objective substance already present in the field, but an intellectual construct. Before measuring, the evaluator must know very precisely for whom the evaluation is done, for whose sake and what the perceptions of the prescriptor of the study are. Then, objective models can be devised with due indication of the hypotheses.

(ii) Guidelines for future evaluation: towards a beneficiary approach

Two main lines are proposed, taking into account the issues mentioned above. The precise connection between these will have to be designed in each future evaluation case, for which a specific and contextual coherence will have to be built.

First, there is a need to provide a global classification of potential effects. Among the various kinds of effects observable by impact studies, we propose the following classification in three levels in Table 6. The first level, direct economic effects, corresponds to an impact study that can be done but is almost useless for the core of our research. The second, classical contribution to regional development, is relevant and feasible. The third, evolutionary contribution to regional development, is very relevant but cannot be performed before one or several decades (normal observation time lag for evolutionary phenomena in economics).

Table 6: Measuring the impact: three kinds of effects
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

Secondly and probably most importantly, there is a need to develop a dedicated approach that fully takes into account the expectations and perceptions of the main actors interested in the identification of regional impact, thus to develop a beneficiary approach. All the information gathered in the various sites we visited proves that local/regional stakeholders are interested in research infrastructures and have a specific vision of it - included in their strategic development policy. Local/regional authorities are generally ready to co-finance the infrastructure because they have a perception of the benefits for the territory. Potentially, if one actor has an interest in measuring RI impact, it is the policy-maker who needs to convince the taxpayer of the interest of co-financing. We then rank the dimensions that can motivate local/regional policy-makers from the most direct and “tangible” to the most symbolic. It should be stressed that the symbolic motives are not the least important. In fact, the “utility” associated to every dimension varies from one territory to another, because they do not have the same value system (political/cultural perceptions). We must underline the fact that the list of effects is not exhaustive. Furthermore the different types of effects are sometimes overlapping.

Table 7 summarizes the main issues raised by RIs from the territory viewpoint.

Table 7: Four dimensions in the RI utility
(Cf. attached document entitled "EvaRIO main results - Tables and Figures")

These approaches reflect two recent trends in RIs’ value for the territories and the necessity to consider the policy-makers’ viewpoint: in addition to objective immediate impacts, RIs increasingly contribute to strategic targets of the territories that are more abstract (Science as part of the regional image) and remote (Science as an option for the future, allowing the creation and sustainability of new paths of development). Indicators are not always easy to find, but in the case of the image at least, there is already a metric: the ranking of cities and regions.

4 Policy recommendations

A series of policy recommendations were proposed to the EC, and were - whenever possible - put into perspective with the starting Horizon 2020 Programme (Cf. Deliverable 5.2 - Part III: EvaRIO Policy Recommendations).

EvaRIO is about evaluation methodological developments. An expected result is thus to derive several recommendations focussing on evaluation per se. But our study and especially the numerous interactions we had with RI officials and other key actors provided other interesting highlights, facts and viewpoints about RI impacts, that went beyond evaluation stricto sensu. Accordingly we distinguish between two kinds of recommendations:
Recommendations for Evaluating (REV thereafter), with the aim of improving the evaluation approaches of RI impacts, and Recommendations for Optimising Impacts (ROI thereafter), where we consider the recommendations which could be implemented by the EU in order to magnify RI positive impacts - or minimize possible negative ones. The following presentation of the recommendations is organized in three broad lines of argument, each being subdivided in different sub-items.

4.1 The core mission of RIs is to contribute to producing excellent but often far-from-market scientific outputs

(i) Evaluating Excellence for RI users

REV.1:
Inventoring, counting, characterising users and most of all tracing their publications are critical for evaluating the impact of RI on Science. The EC should improve researchers’ incentives to cite RIs, i.e. reveal and valorise RI use by:
- Making it a condition for EU project funding
- Asking national/regional authorities to promote the same rule
- Promoting agreements with journal editors and bibliographic data providers.

(ii) Fostering innovation: from scientific to marketable outputs

REV.2:
Do not expect many marketable outputs in the short run and use evaluation approaches entailing a long time lag between RI use (or RI launch) and the advent of such downstream outputs.

ROI.1:
Caution should be exercised in modifying RI mission towards more marketable outputs. Looking for excellence is attractive for industrial users too.

(iii) From RI access provision to collaboration with users

ROI.2:
Excessive market orientation can impede collaboration and learning processes with users. Attention should be devoted to:
- Preserving a collaborative space within RIs, especially when opening up equipment and resources to external users, or when implementing remote control access.
- Promoting the view of service as a springboard for S&T collaboration.

(iv) From procurement to collaboration with suppliers

REV.3:
When evaluating impacts on RI suppliers, it is necessary to take all types of impacts into account: the learning impacts as well as those in terms of price, costs and market share.

ROI.3:
The EC should promote coordination between similar RIs concerning their procurement policies, not only in terms of costs but also in terms of learning, i.e. exploiting and increasing suppliers' capacity and competences.
The EC should simultaneously attend to the fair return principle.

4.2 RIs entail a wide variety of resources and configurations

(i) Dealing with a variety of RI configurations

REV.4:
Do not try to develop a single method or a single list of indicators to evaluate the impacts of any kind of RI, even if there is a need for common evaluation standards. Consider the following propositions instead:
- Evaluation developments should be pursued in a long run project taking RI variety into account, i.e. creating ad hoc indicators for each RI configuration.
- Evaluation for a given configuration should entail a part specifically devoted to comparison with RIs most advanced counterparts in US, Japan, or China.
- Scenarii should be elaborated to explore what would happen in case there was no funding for the RIs.
- A minimal percentage of RI budget should be dedicated to the evaluation of RI impacts. The percentage should be increased in the case of a new RI for which no methodology exists.

ROI.4:
RI variety implies that the optimisation of impacts requires activating adequate policy levers according to the particular case of the RI at stake.
Example: It would be rather ineffective to prioritize and improve local impact / local transfer mechanisms in case of a distributed RI taking the form of a grid. But structuring the European Research Area is probably a relevant priority for distributed RIs.

(ii) Variety of stakeholders and of evaluation goals

REV.5:
To choose the appropriate indicators, first clarify the institutional setting of the evaluation exercise and make explicit the system of value of the user / sponsor of evaluation.

ROI.5:
While developing inter-policy coordination, try and adopt a multi level – multi actor view entailing synergies, competences, share of data (not only a vertical view in terms of return on investment).

(iii) Furthermore at the regional dimension

REV.6:
Attribution of ESIF means in fields clearly related to the RI activity constitutes a potential indicator of RI’s regional impact. Nevertheless, evaluation must take into consideration the regional context: impact in “leading” regions and in “followers” cannot be expected to be the same.

ROI.6:
Any improvement in the interface between RI policies and the ESIF activities will increase the efficiency and coherence of the EU policies and their perception at regional level.

(iv) Variety and need for integration: Evaluation as a means to share common views

REV.7:
Discussion and exchange of ideas are necessary for creating awareness and commitment, common visions and channelling the efforts within the evaluated entity.
Any evaluation exercise should entail milestones and workshops in order to share the methodological approach ex ante and the results ex post, first with the evaluated people, then with the other stakeholders.
The frequency of evaluation rounds has to be fine-tuned to avoid "evaluation fatigue".

ROI.7:
The EU should support benchmarking of good practices, training and mobility in the case of RI staff in charge of technology transfers.

4.3 Science and RIs operate through collaborative networks, communities and open systems

(i) RIs and the need for network evaluation approaches

REV.8:
Impacts and evaluation results for newcomers are not commensurable with those for incumbents. Both should not be treated as equivalent. Without incumbents newcomers would not learn. Incumbents are a source of effects for newcomers so they also deserve consideration and support.
Aggregation cannot be reduced to mere addition of effects for focal actors. A meso level of analysis of RI impacts, entailing systemic effects, should be supported by the EU. In particular, social network analytical tools could be further developed and adapted to the RI case.

(ii) Community-built knowledge and RI access

REV.9:
RIs should not only be evaluated in terms of scientific and economic outputs but also in terms of accessibility:
- Accessibility to new users, including new disciplines/communities and non-European users.
- Accessibility to the knowledge they contribute to generate.

ROI.8:
Depending on the type of RIs (especially the rivalry of resource or not), access rules and grant-back conditions should be fine-tuned in order to facilitate free access to resources, their maintenance and/or development.

- Building networks through time: Facilitating PhDs' training and access

REV.10:
RIs should be urged / supported to collect information concerning the students they train (geographical & topical mobility, and careers).
Synergies between RIs and Marie Skłodowska-Curie actions should be encouraged concerning the traceability of advanced students.

ROI.9:
Quota of RI access dedicated to excellent young researchers' projects should be implemented.
The co-funding of PhD theses and Post-Doctoral studies in RIs should be encouraged by the EU.

5 Dissemination

Throughout the course of EvaRIO, different seminars and a final conference were organized:

- In 2011, the seminar "Methods of evaluation of research infrastructures: Lessons from different case studies" took place in Strasbourg with specialists of evaluation, in particular from SFTC (UK), CERN, Technopolis, ELETTRA during the first phase of the study (Cf. Deliverable 1.2).

- In 2011 again, two seminars about "Research infrastructures as open innovation and research systems" (cf. Deliverable 2.2) gathered experts of different aspects related to the four Focus studies and methodological developments. The first one about "RI evaluation, real options and network considerations" took place in Brussels with EC representatives and the second one, about " Open source and regional creativity" took place in Strasbourg with experts from the University of Barcelona, the regional platform Iconoval and Fraunhofer ISI).

- In 2013, two "stakeholder seminars" were organised for EU and RI officials in order to present extensively, discuss and validate the results of the case studies. One seminar took place in Brussels, gathering different RI representatives; the other was dedicated to synchrotron SOLEIL and took place at the RI site.

- In October 2013, a large final conference was organised, gathering specialists as well as policy makers at different levels (cf. Deliverable 7). The methods and results obtained through EvaRIO were discussed and put into perspective with other evaluation approaches. The presentations can be downloaded from the EvaRIO website: http://evario.u-strasbg.fr/.

Potential Impact:
• Exploitation and impact

The evaluation method is ready to be implemented in the case of any existing RI, so it is at disposal of researchers and policy makers willing to use it. BETA is presently starting to seek for future applications in various cases of national or international policies.

Beside the full method (with corresponding framework, metrics and protocols), intermediate pieces or components of the method are also potentially applicable. Especially the original conceptual approach towards the EvaRIO framework (distinguishing between direct and indirect effects and between actual and capacity effects) could be re-used for elaborating a relevant framework for the evaluation of the impact of other types of Science, Technology and Innovation Policy. Similarly, some of the indicators (especially those which were specifically developed in the study, in addition to those borrowed from the pool of widely acknowledged indicators) could be re-deployed in other contexts. Moreover, the knowledge acquired by BETA in the use of existing databases (scope of relevance, limitations,…) and in the way to mix them with ad hoc sources of information could easily be re-used for a large scope of evaluation work.

In the same line of ideas, some aspects of the methods could be further refined and developed, as it was underlined in the preceding summary part about EvaRIO main S&T results. Areas of improvement have been quite clearly identified in that respect.

Regarding the conceptual and empirical development achieved as for the Focus Studies, here again potential applications are straightforward:
* at conceptual level, some insights could be re-contextualized in other contexts, in particular those of Focus 1 on open source approach (e.g. How to define an open source mode of production? How to test whether a specific activity follows the conditions, so defined? Etc.) and those of Focus 2 on flexibility (e.g. the managerial mechanisms set up for solving and articulating internal tensions and tensions in relation to the environment,…).
* at instrumental level, some metrics and modelling developed in Focus 3 on networks in order to analyse the genesis and evolution of R&D networks could be easily re-implemented in other contexts.
* at evaluation framework design level, in the case of Focus 4 on regional impact, the research agenda open via this subpart of EvaRIO study is still to be explored.

At another level, the EvaRIO project directly provides helpful elements, pieces of information and recommendations for two types of stakeholders. We refer first and essentially to the EC. Beside the results of each of the case studies as such (with their limitations related the scope of evaluation chosen in each case), the study helps to improve the understanding of the conditions of knowledge creation through RIs, in particular their ability to foster collaborations and creativity. And moreover, the recommendations proposed go beyond evaluation stricto sensu. Some of them aim at improving the evaluation approaches of RI impacts, providing useful insights to enrich the RI performance indicators mentioned in H2020 specific programme. Some others are related to actions that could be implemented by the EU in order to magnify RI positive impacts - or minimize possible negative ones. It is nevertheless up to the EC to consider whether these recommendations are relevant, and if yes, to generate a significant impact out of them. The second type of targeted stakeholders obviously are the responsible of the RIs which were investigated in the case studies. They could be (and some have already actually proved to be, see following section) interested in the results of the case studies, and they may be willing either to change some aspects of the functioning of their RI and/or to implement some of EvaRIO evaluation tools and/or to include (part of) the results in their public relations materials.

• Dissemination activities

As regards dissemination, several initiatives have already been taken since the beginnning of the project.

In 2011 a dedicated website was designed and launched in order to present the EvaRIO project, and disseminate some publishable resultslater on, in particular the communications presented at the EvaRIO final ocnference.

Throughout the course of EvaRIO, different seminars and a final conference have been organized:
- In 2011, the seminar "Methods of evaluation of research infrastructures: Lessons from different case studies" took place in Strasbourg with specialists of evaluation, in particular from SFTC (UK), CERN, Technopolis, ELETTRA during the first phase of the study (Cf. Deliverable 1.2)
- In 2011 again, two seminars about "Research infrastructures as open innovation and research systems" (cf. Delivrable 2.2) gathered experts of different aspects related to the four Focus studies and methodological developments. The first one about "RI evaluation, real options and network considerations" took place in Brussels with EC representatives and the second one, about " Open source and regional creativity" took place in Strasbourg with experts from University of Barcelona, the regional plateform Iconoval and Fraunhofer ISI).
- In October 2013, a large final conference was organised, grouping specialist as well as policy makers at different levels (cf. Delivrable 7; the presentations can be downloaded at the EvaRIO website: http://evario.u-strasbg.fr/).
In all cases, the objectives were to compare approaches, discuss the method and results obtained (at different stages of advancement) by EvaRIO, and more broadly to disseminate the results of the EvaRIO project.

In addition, also as part of the project, the results of the case studies have been extensively presented and discussed with related RI actors at the occasion of dedicated one-day "seminars with stakeholders". One of them took place in Brussels, gathering different RI representatives; another was organized at synchrotron SOLEIL. In most of the cases, contacts and further discussion have been maintained with the RI operators, the latter being quite active in providing additional details and bringing fruitful comments and corrections to some aspects of the results. In the same vein, BETA representatives were invited and officially introduced as performing evaluation studies in some internal meetings of the RIs, such as the kick-off meeting of the EMMA-follow up INFRAFRONTIER RI.

Another dissemination activitiy was performed by participating to various workshops dedicated to RI impact, at the occasion of which the BETA could present some partial results. In particular BETA representatives attended the final RIFI project conference in Bucharest in May 2011 and the ERF Workshop on the "Socio-Economic Value of Research Infrastructures" in June 2012 at Desy, Hamburg.

Finally, in a more academic perspective, some publications have already been achieved or are in preparation deriving from EvaRIO study.

Some aspects of the results of the Focus Study on Flexibility, based on the SOLEIL case, will be published soon:
Arman Avadikyan, Laurent Bach, Gilles Lambert, Christophe Lerch, Sandrine Wolff, "Dynamique des modèles d'affaires et écosystème: le cas des synchrotrons", Revue d'Economie Industrielle, Numéro thématique: "Modèles d’affaires et Ecosystèmes d’affaires : approches économiques et stratégiques ", à paraître: 2014 (juin), ISSN : 0154-3229

Some conceptual development originating from EvaRIO Focus Study on open source mode of knowledge creation have also been partly used in some publications assembled in a special issue of the journal Management International:
Management international, Gestion des Connaissances dans la Société et les Organisations / Knowledge Management in Society and Organizations / Gestión del conocimiento en la sociedad y en las organizaciones, Volume 16, numéro hors série, 2012, p. v-143, sous la direction de Claude Guittard, Eric Schenk, Pierre-Jean Benghozi, Jean-Louis Ermine et Michel Kalika, ISSN : 1206-1697.

Other publications are intended in the foreseeable future, among others: a journal article on the flexilibity of RIs resulting from the work carried out in the corresponding Focus Study, and another article presenting the highly interesting results of the Focus Study about RI networks and the use of Social Network Analysis tools.

List of Websites:
http://evario.u-strasbg.fr/

Final Report Summary - EVARIO (Evaluation of Research Infrastructures in Open innovation and research systems)

Verwandte Dokumente

Diese Seite teilen

Herunterladen