Final Report Summary - LIAISE (Linking Impact Assessment Instruments to Sustainability Expertise)
Executive Summary:
4.1.1. Executive Summary
An evidence based approach of policy development and societal decision making is becoming increasingly important, both in the EU and in Member States. The process of integrated impact assessment (IA) potentially provides a framework to compare policy options, it is applicable to all (policy) sector and able to identify potential trade-offs between different impact areas, regional and temporal scales and thereby maximizing the benefits while minimizing unwanted side effects.
Considerable funding in the framework of European research programs has been invested to support IA by means of developing models, methods, datasets, toolboxes and by studying the process of IA. However, both research and the experience of practitioners (principally desk officers) demonstrates that the results of these research projects are not being utilized to its full potential. A better uptake could be expected if the understanding of researcher of policy needs would be improved, the capabilities of tools would be adapted accordingly and awareness of IA practitioners on the potentials of tools developed in research project would be increased.
Additional gaps exists between the different scientific disciplines: Research efforts are fragmented between those that study the functioning of assessment procedures and those that design and update assessment tools. Some research projects are primarily motivated to directly inform IA for SD, but many are primarily motivated by scientific merits.
The Network of Excellence (NoE) LIAISE was funded to explore these gaps, to develop solutions and to give these solutions a structural permanence by developing an organisational setup and a business plan for a future centre of IA excellence that can continue beyond the funding for the NoE.
To achieve this goal, LIAISE activities entailed among others:
• Research on the practice of IA and the use of evidence in policy making throughout European Member States, the OECD and in developing countries
• Review of research projects and their categorization of their contributions in the context of IA for SD
• Development, improvement and testing of tools which take into account the requirements of IA for SD
• Development of a toolbox with IA relevant knowledge as a Science-Policy Interface and including standards to describe knowledge for decision making
• Organising dialogues between IA practitioners and different disciplines
• Development of a network of researchers, research organisations and IA practitioners which are committed to develop jointly knowledge for decision making
• Development and testing of training formats for IA practitioners and researchers
The major achievements of LIAISE are:
• A better understanding and collaboration between researchers from the ‘modelling’ disciplines and researchers from the policy sciences and social sciences. This resulted in a better understanding of the concept and roles of IA methods and tools in the policy process. Research needs to be contextualized in order to become relevant for decision making.
• Tools for ensuring such contextualisation, including techniques of Knowledge Brokerage, the LIAISE KIT, the Shared Research Agenda and a strategic dialogue with IA practitioners.
• The lessons learned provided building material for shaping the future of LIAISE as a Community of Practice on IA research for SD and are reflected in:
• A Charter on IA research for SD: a set of guiding principles for delivering knowledge that is relevant, credible and legitimate.
• A vision for a Community of Practice on IA research for SD and its most important roles and functions: IA knowledge and information hub, Networking and discussion forum, Innovation and testing, Tool identification and quality monitoring, Learning.
• An implementation plan and a business plan to get the CoP going. These plans have been endorsed by all 19 parties: all 15 partners of the NoE consortium and Ecologic, IEEP, IOM, and IZA. Additional partners are expected to join in the future.
Project Context and Objectives:
4.1.2 Project context and objectives
4.1.2.1 The role and position of impact assessment in European policy making
An evidence based approach is becoming increasingly important in policy development and societal decision making. The process of impact assessment (IA) is a central tool in regulatory policies and provides a framework for comparing policy options, while being adaptable to all (policy) sectors. Evidence on the likely impacts of policies enables policymakers to maximize benefits to society and minimize unwanted side-effects. Therefore the analysis should cover the impacts in the targeted policy domain, as well as unintended impacts, side effects and trade-offs in adjacent policy domains. IA systems and procedures have been implemented in the EC and all OECD countries and are spreading in other parts of the world.
Policy making in the European Union follows a complex system of multilevel governance involving interaction between the European policy system and the Member States. The overarching goals for policies at the European level are laid down in comprehensive strategy documents such as the Sustainable Development Strategy, the ‘Lisbon Strategy’ for Growth and Employment and more recently Europe 2020. These strategies recognise that policies aiming at the overarching goal of Sustainable Development (SD) should follow a three-pillar approach targeted at economic, social and environmental objectives. These three pillars of SD are recognised as equally crucial, interconnected, and urgent. All EU institutions (including the Commission, but also the Council and the Parliament) have committed themselves to base major policy proposals on a balanced assessment of the social, environmental and economic dimensions which takes into account the external dimensions of SD.
The instrument of ex-ante IA as developed by the EC aims at collecting evidence on the likely impacts of policies and thereby maximize the benefits to society and minimize unwanted side-effects. A first guideline was published by the Commission in 2003, and updated in 2005 and 2009 after a process of public consultation . It aimed to replace and integrate all sectoral assessments of potential impacts of proposed measures into one global instrument with a balanced consideration of development targets against the three dimensions of SD. The guidelines describe the organisational structure of the process and a basic process: 1) identification and analysis of the policy problem; 2) definition of the policy objectives; 3) development of alternative policy options and a baseline scenario to describe the development without policy intervention: 4) actual analysis of the impacts of the policy options with respect to the three dimensions of SD; 5) comparison of the analysed impacts of the various policy options against the baseline scenario; and 6) outlining a set of recommendations for indicators, monitoring procedures and the ex-post evaluation of the implemented policy.
Since 2003, the role and importance of IA and evidence based policy making has steadily increased within the European institutions. This is indicated inter alia by:
- the setup of an Impact Assessment Board of High level officers, reviewing the quality of IA
- the dedication of JRC as main supporting Directorate to support IA
- the setup of support units in all DGs and in the SecGen
- training of desk officers
- the setup of an Directorate for IA in the European Parliament
- awarding of numerous studies and framework contracts to support IA
- evaluations of the performance of the EC IA system by the Secretariat General and the European Court of auditor
These developments have been echoed in the Member States and beyond: Although IA requirements and practices vary considerably, the LIAISE surveys identified relevant activities and capacity building throughout Europe.
4.1.2.2 Perceived gaps between IA tool availability and actual use in the policy process
European research policies since Framework Program 6 have invested in several large scale projects to develop models, methods and data in support of Impact Assessment. A focus area was in topics related to research for Sustainable Development. Among these projects were MATISSE, SENSOR, SEAMLESS, IQ TOOLS, EVIA and others. Despite of these investments in developing new models and methods in support of IA, both research and the experience of practitioners (principally desk officers) demonstrates that in practice the broad range of potentially relevant tools to support IA is still not being used to its full potential. Compliance testing work on the reality of policy making revealed a gap between the aims of IA and its implementation. The application of IA tools is difficult and often at odds with the process of decision making . The political process is determining to a large degree the applicability of IA tools, not only the features of scientific knowledge. Relevancy of models and methods is determined by the context of use. Therefore, it is important for researchers to understand the context of use. The call text of topic ENV.2009.4.2.1.1 (Network of excellence for IA tools) and the documents provided by the Commission specified the main features of the gaps between scientific IA tools and the actual use in the policy process:
Policy orientation vs. research orientation. Policy-makers tend to look for tools with a proven record of effective use in policy. Policy relevance however is not yet a significant part of the researchers incentive system. From their point of view the development of new conceptual approaches and ever more complex tools is more rewarding than a focus on the usability of existing tools.
Complexity vs. transparency. The tools used must be based - as far as possible - on rigorous analysis while recognising explicitly where value judgements are included. However, many scientific models remain ‘black boxes’. Differences in data sources used or simple misunderstandings about the terminology used may undermine trust.
Maintaining existing investments vs. preparing for the future: promising or successful tools from a policy perspective are not always maintained or further applied by the developers. Due to sometimes limited interactions with policy-makers, researchers do not properly recognise the loss of opportunity in not further maintaining or applying existing tools.
Accessibility vs. applicability: the access to existing assessment tools is often poor. One of the reasons is the lack of structure to link the diverse and ever-changing needs of policy makers with the overwhelming abundance of existing tools on the supply side. Another important reason is the limited access to the data that are needed to apply the tool.
To conclude: high expectations are not being met with respect to the use of IA and the linking of policy assessment tools and procedures to pursue SD in the EU. The improved use of tools for SD requires an in depth understanding of policy needs, tool capabilities and the associated opportunities for (and obstacles to) linking them in the everyday policy process. Current research efforts are fragmented between those that study the functioning of assessment procedures and those that design and update assessment tools. Some of these research efforts seek to directly inform and in turn improve IA for SD, but very many do not.
The NoE LIAISE was funded to explore these gaps both between IA practice and research as well as between different disciplines. Expected impacts as formulated in the call: Increased integration of the IA research community, improved efficiency of IA tools through their integrated and mutually complementary development, structured dialogue between the research community and policy makers about IA tools development, and enhanced use of IA tools in policy processes at EU and Member State level.
4.1.2.3 LIAISE: mission, vision and strategy as formulated in the DOW
The DoW described the mission, vision and strategy of the NoE as follows. The challenge for LIAISE was to bring the above mentioned elements together in a more systematic fashion by: 1) showing decision-makers at different levels of governance and in different jurisdictions that different assessment procedures consist of similar phases and can draw on similar types of tools; 2) stimulating researchers to streamline the rich diversity of tools and communicate the potentials and limitations of individual tools; 3) further developing that framework to enable tool selection and use in close collaboration with IA users and IA researchers; and 4) fostering a deeper understanding of the role of IA (and IA tools) in the different phases of the policy process. The mission (the overarching objective of LIAISE) was to contribute to the renewed EU Sustainable Development Strategy (SDS) by bridging the gaps between science, policy making and implementation, with academically-grounded approaches to IA. The vision for LIAISE was to establish a ‘virtual centre of excellence’ with a leading position in the international IA research community and close institutional linkages with users at European and Member State levels. Together these characteristics should provide the capacity to contribute to the desired impacts as formulated by the Commission.
The vision was translated into the following strategic goals:
• Creating the hub of the NoE, comprising a core group of institutes, whose task will be to initiate and further develop the virtual centre of excellence mentioned in the vision.
• Using the NoE as a platform to initiate a more structured dialogue between the IA research
• community and IA users. This should result in: a) Communication structures, incentives and mechanisms that stimulate external institutions to join LIAISE as associate partners and thus contribute to the development of the virtual centre of excellence; b) A shared IA toolbox reaching out to policy makers and those involved in conducting IAs; and c) A shared research agenda on strategic IA tools and processes and the development of new IA tools;
• Safeguarding the achievements of 1 and 2 for the post NoE period.
Therefore, the key characteristics of LIAISE will be:
• A bridge between the IA user community and the IA research community. This requires a deeper understanding of the role and limitations of IA in policies for SD, the stakeholders involved and the possibilities for an improved use of existing tools by the various categories of stakeholders.
• A means (the shared IA toolbox) to handle the interplay between the socio-economic and
• environmental systems in combination with multi-scale problems in ways that help policy-makers implement IA to contribute to the key objectives of the EU SDS.
• A way of handling the ongoing development of new tools. The ongoing development and
• refinement of the EU SDS will throw up new strategic questions. Progress in the fields of science and technology will also lead to new opportunities for developing IA tools that answer emerging policy needs.
• A close, two way collaboration, interaction and information exchange between and within the IA research community and IA users. This will be absolutely crucial for the success of theproposed NoE.
4.1.2.4 Strategic changes during the execution of the project
During the course of the project three major developments strongly affected its strategic scope and outcomes of the network: 1) the position and role of the Policy Advisory Board; 2) changes in the EC strategies regarding SD; and 3) the changing views in the science-policy-interface on the concept of IA methods and tools. The re-organisation of DG RTD and the frequent changes of project officers (6 in 54 months) added additional complexity to the question how to adequately address these developments.
The call text explicitly requested the installation of a Policy Advisory Board (PAB): The NoE will also establish the appropriate links with policy makers to ensure the use of the shared IA toolbox. In this context, the consortium should also establish a policy-advisory board. In the DoW, this was addressed by installing a PAB that combined the interaction with the IA users as-well-as the scientific supervision of the project. The combination should prevent the risk (as observed in other FP6/7 projects) that working with separate advisory bodies (a policy advisory board and a scientific advisory board) might introduce a science-policy gap in the organisational design of the project as such. The PAB was given a central role and position in the conceptual approach for the long-term integration in the NoE . To enable the PAB to fulfil its supervisory role, an on-going evaluation by an external party (Ecologic) was included in the project design.
After the project was granted to the LIAISE consortium, DG RTD pro-actively engaged in the formation of the PAB with EC representatives from the various Policy DGs and SecGen. The first PAB meeting showed that there were no shared views between DG RTD, SecGen and the policy DGs, despite DG RTD efforts to internally fine-tune the preparation of the call text and the PAB formation. The envisioned PAB members refused to accept any 'advisory' responsibilities for the project and changed the name to Policy Board (PB). Individual members are not representing the views of their organisation, but use their personal capacities, knowledge and experience to critically reflect on the project. The status of the on-going external evaluation by Ecologic changed from a monitoring instrument at the disposal of the PAB to an activity in support of the scientific Management Board.
Subsequently, the members of the policy board were supportive in providing recommendations for the research conducted within the network and provided access to other policy officers e.g. in the context of the test cases. It was emphasised, however, on several occasions that LIAISE would not be considered as any kind of support service for IA complementing or replacing existing structures within the European Commission. LIAISE is considered to play a role in research and research programming, if its members are willing to take part in IA, they are encouraged to apply for tender.
The call text ENV.2009.4.2.1.1 (Network of excellence for Impact Assessment Tools) directly relates the activities of the NoE to the Renewed EU SDS. Indeed, the renewed EU Sustainable Development Strategy (SDS) reaffirms the need to support policy-making by scientifically sound approaches for IA. With the publication of the EU2020 Strategy the prominent position of the SDS as a driver for SD in the EU, was replaced with the EU2020 Flagship Initiatives to support smart, sustainable and inclusive growth. These changed also were reflected in organisational changes at DG RTD. The DG RTD unit that formulated the call and strongly promoted the NOE initiative, ceased to exist. The key persons for LIAISE were no longer connected to the project; the head of unit moved to another DG and the project officer left the Commission. An indifferent attitude towards the project became apparent from EC stakeholders. This could be observed in the meetings of the PB as well as in interactions with Policy DGs and JRC. In the PB meeting of 26 November 2012 the future of LIAISE was discussed. The response by EC representatives as described in the minutes is: The objective of LIAISE NOE is to establish a long lasting Network of Excellence that coordinates research and the envisaged interaction between policy and science in the field of impact assessment. From the responses of the PB to the options for a post LIAISE entity, a clear role for LIAISE cannot be distilled to achieve this objective. The Policy DGs tend to prefer LIAISE competences as part of regular consultancy contracts, JRC indicate that the Commission has sufficient in-house capacities for training and tendering procedures for consultancy. The existing procedures of DG RTD to coordinate research programming with the various DGs are considered as sufficient. Therefore, at this moment reviving existing scientific associations is the best suggested option for a future LIAISE entity. These existing scientific associations are not institute or consortium based, but consist of individual research members. For this no future LIAISE entity is required, LIAISE researchers can sign up individually to these associations and have an impact on their actions.
The statements made in the Policy Board also demonstrate that different actors have developed own support units and institutions for impact assessments during the lifetime of LIAISE and which perceived LIAISE as a potential competitor, respectively questioning the added value of LIAISE in this emerging landscape of actors related to IA.
In their report on period 2 the coordinators asked the Commission’s attention for these alarming developments. ‘In the perception of the coordinators this decline of genuine EC interest in and involvement with the long-term objectives of the NoE, is a serious threat for a successful and durable centre of excellence. The 'business concept for the shared toolbox' which was a key element in the assignment by the EC to LIAISE, can only be maintained in a situation where the EC is prepared to act like a future customer for the shared toolbox and LIAISE expertise. As coordinators we must discuss the EC involvement with DG RTD to agree on a shared strategy for achieving EC commitment in period 3 and the appropriate level of ambition.’ This message was actively taken up by the newly appointed project officer who organised a review by an external expert and a workshop with the LIAISE coordinators and other on-going projects in the field of SD in order to strengthen the coherence and synergies in April 2013. The review suggested to focus LIAISE more on sustainable development in the broader sense and relatively less on impact assessment in the European Union context, and urged LIAISE to think more strategically on the relevance of evidence production in the wider sense of achieving sustainable development. LIAISE achievements in networking, contextualizing tools and models, scientific progress, and method development for test cases were acknowledged, while at the same time stimulating LIAISE to think of a broader group of audiences for its services and knowledge.
Before outlining the follow-up of the review and workshop, it is necessary to briefly describe the third major development that strongly influenced the strategic scope of the project: the changing views in the science-policy-interface on the concept of IA methods and tools. Among the IA researchers the view predominated that IA methods and tools are developed by researchers in the scientific domain, then disseminated into the application domain where they are actually used by IA practitioners and policy-makers. The members of the PB pointed at the fact that this separation between the two domains is artificial, unproductive and not reflecting real life situations where IA methods (and scientific evidence in general) are successfully used in policy processes. From their perspective the scientific content of a method or tool cannot and should not be separated from its policy context. This PB intervention has proven extremely relevant and valuable for the scientific achievements of LIAISE. A major part of the research has been targeted at sharpening the views on the roles and responsibilities of researchers and policy-makers in policy processes, on the process of Knowledge Brokerage and on the question how to support these activities with ‘contextualized knowledge and information’. This not only had significant consequences for the tool development in LIAISE, but also for the business and quality concepts underlying a future centre of excellence. The ambition to deliver contextualized knowledge strongly limits the opportunities to develop and sell standardized IA tools as a ‘product’. Instead the focus turned to activities and processes related to the development of scientific knowledge. Research needs to be sensitive to the context of use and therefore systematically analyse the contexts of potential application and to develop and to describe scientific knowledge accordingly while at the same time meeting scientific criteria. In follow up discussions with the project officer in the summer 2013, it was agreed to focus efforts on a synthesis of main lessons learned of LIAISE for evidence production, to expand the exploration of users to member states, international organizations, and potentially business, and to expand the policy board with participants familiar with impact assessment of research investments.
Based on these experiences and considerations, a strategic focusing of LIAISE has been initiated by the consortium in close collaboration with the project officer. Major steps of this focusing were:
• The annual meeting of the consortium in Tallinn: It was agreed to explore a focusing of LIAISE on IA for Sustainable Development and to position LIAISE strategically addressing the research needs and subsequent research organisation on transdisciplinarity, long-term issues and integrated analysis along all dimensions of sustainable development.
• Selection of mini-research projects as part of an open call for remaining flexible budget focusing on Impact Assessment in international organizations, Research Impact Assessment, Impact Assessment in Emerging and Developing countries and durable concepts for networks of excellence of researchers.
• The conduct of an extensive stakeholder analysis exploring the potential services derived from a LIAISE NoE and the relevant stakeholders and potential customers for these services.
• The focusing of the LIAISE activities on mid- and long term programming of research
• The development of a charter which expresses the commitment of LIAISE researchers in developing research which is applicable for meeting societal needs in the context of sustainable development
• The development of a business plan which defines core products and services for maintaining LIAISE as a network with the specific focus in the future and which coordinates the partner contributions.
The focusing of LIAISE suggests a different positioning. Instead of developing a research institute or an organisation which provides consultancy, training or other services, a network conceived as a community of practice was selected as most appropriate. A community of practice is characterised by its diversity of members in terms of disciplines and professional background but with a shared ambition of solving similar problems.
The focusing of LIAISE as a community of practice, its commitment to a charter and the proposed business plan was well received by the stakeholders, the members of the policy board, and the members of LIAISE. The partner organisations, including the partner that were associated during the funding period expressed their willingness to provide substantial contributions to such a future LIAISE community of practice.
Project Results:
4.1.3 Main S&T results/foregrounds
4.1.3.1 General approach of the LIAISE NoE
The LIAISE consortium unites the multi-disciplinary competences of a core of 15 European institutes from 9 countries that in turn consolidate the expertise from large FP6 projects on IA tool development such as SEAMLESS, SENSOR, MATISSE, Sustainability A Test, IQ Tools and EVIA. It includes expertise from the field of environmental sciences, economics and political sciences. This makes it possible to analyse current policy needs, to link them in innovative ways to the available reservoir of IA knowledge and to test these innovative solutions in targeted, co-designed and co-produced IA test cases. Three additional elements in the project setup complete the general approach:
• The formation of a Policy Board with policy makers and IA practitioners from the EC (policy DGs, DG Research and Innovation, SecGen), Member States, the OECD and NGOs. The Policy Board provides a platform for the interaction at the strategic level between IA researchers and IA users;
• The formation of focus groups with IA researchers and IA practitioners to discuss operational issues and user experiences;
• The on-going evaluation by Ecologic (www.ecologic.eu) a think tank for applied environmental research, policy analysis and consultancy. It was commissioned by LIAISE to assess the progress towards the general project objectives. This evaluation is based on the impact chain approach and is complementary to the standard FP7 project evaluation procedures by the Commission which focus on monitoring the project progress in terms of the realisation of the project objectives. The impact chain approach gives insights in the progress towards the envisaged impacts of the project (the ambitions beyond the realisation of the project objectives, e.g. the contribution to evidence based policy development).
4.1.3.1.1 Project structure: Work Packages and deliverables
Table 1 Work Packages and deliverables
WP0: Management and coordination of LIAISE
Tasks: 1) Overall coordination of LIAISE; 2) Coordination, planning and monitoring system; 3) Set Up of a Policy Advisory Board; 4) In Built Evaluation; 5) Knowledge management activities and associate partners
WP0 Deliverables
1. Policy Advisory Board
2. Gender Action Plan
3. Letter of institutional commitment
4. Operational planning and monitoring system with guidelines 5. Quality Assurance plan
6. Activity and financial reports for every 18 months
7. An updated financial and implementation plan
8. Report on Awareness and Wider Societal Implications
9. Final report
WP1: IA for sustainability: policy needs, assessment procedures and governance contexts
Tasks: 1) Coordination of WP1; 2) Synthesize and meta-analyse existing research on the procedures and everyday practices of IA in the EU and the Member States; 3) Explore the relationship between IA and wider systems of governance for SD development in the EU and its Member States
WP1 Deliverables
1. A meta database of different cases of IA in action, drawing on results from inside and outside LIAISE including previous FP6 investments.
2. Review articles in relevant academic journals and practitioner outlets
3. Summary of user needs and expectations regarding IA processes, IA tools, IA training and WP6 test cases.
4. Review of links between IA and wider systems of governance for SD in a representative sample of juris-dictions and/or policy areas, via an edited book or similar.
WP2: Science for IA Tools and Procedures
Tasks: 1) Coordination of WP2 and framework development; 2) Synthesising emerging knowledge to better analyse policy impacts; 3) Research for IA Tools; translation of knowledge gaps into research questions; 4) Roadmap for continuous interaction between IA community and other research communities; 5) Knowledge use in different assessment venues; 6) Development of IA modules for design of research
WP2 Deliverables
1. Methodological framework for WP2 activities
2. Overview of research groups/networks producing knowledge of relevancy for IA tools and processes
3. Synthesis of research needs for IA tools in research programmes inside and beyond the IA research community
4. Procedural concept to facilitate a continuous uptake of emerging scientific and social scientific knowledge in IA tool and process improvement, beyond the lifetime of the NoE
5. A literature review on the politics and policy of evidence-based policy-making in different assessment venues, with particular reference to assessment tools
6. Modules for research designers to make research programmes and projects compatible with IA knowledge needs
WP3: Shared toolbox: back-office and improvement of selected IA tools
Tasks: 1) Coordination of WP3; 2) Assuring quality by creating and using an IA Tool Reference Model (RM-IAT); 3) Requirements analysis, creation and improvement of the back office tools; 4) Development of the back office IA toolbox;
WP3 Deliverables
1. Reference Model for IA Tools (RM-IAT) including glossary, software quality assurance (SQA) plan, standards for documentation and user requirements engineering and standards for architectural design and distributed processing
2. Analysis of requirements of tools in terms of: applica-tion aim; application domain and scale(s); flexibility;. harmonization and standardization;, usability; user scope; transparency; documentation 3. Conceptual and architectural design of the toolbox back office
4. Improved tools (software delivery and deployment) including documentation and manual
5. Implementation of shared toolbox back office (software delivery and deployment) including documentation and manual
6. New tools (software delivery and deployment) including documentation and manual
WP4: Shared toolbox: front office
Tasks: 1) Coordination of WP4; Conceptualising and designing the toolbox front office; 3) Development of the toolbox front office; 4) Implementation of the toolbox front office; 5) Maintaining the toolbox
WP4 Deliverables
1. Design options for the toolbox – paper to be presented to the Policy Advisory Board
2. Concept for the toolbox, description of (sub)catego-ries, quality criteria and outline for meta-description 3. Design options for the help desk
4. Populated toolbox for the front office with inventories models/tools/actors/good practice/data sources
5. Help desk
WP5: Shared toolbox: governance and post-project durability
Tasks: 1) Coordination of WP5; 2) Establishment of product standards for participation in the shared toolbox; 3) An operating and performance review; 4) Preparation of a Route Map to a Self-governing Entity; 5) Development and implementation of a Business Plan.
WP5 Deliverables
1. Product Standards
2. Submission of a Route Map to a Self-governing Entity
3. Report on the operating and performance review of the Shared toolbox and associated services
4. Business plan with implementation timelines
5. Report on implementation of business plan
WP6: Test cases in sustainability priority areas
Tasks: 1) Coordination of WP6; 2) Test cases: the development of options; 3) Modules for IA support; 4) Case Studies; 5) Analyse the test results with regard to LIAISE activities
WP6 Deliverables
1. Options for test cases
2. Modules for IA support
3. Results from test cases
4. Material of each closed test case for individual WP7 policy briefs
5. Final material on the test cases for WP7 policy-briefs
6. Compendium of test case studies. Form could be an edited book or an open access journal special issue
WP7: Dissemination and Training
Tasks: 1) Coordination of WP7; 2) Web site and materials; 3) Scientific publications and Policy briefs; 4) IA Training activities; 5) Events.
WP7 Deliverables
1. Dissemination Plan
2. Biannual Innovation report on IA
3. Periodical news bulletin
4. Open Access platform
5. Report on the continuous updates of the website
6. LIAISE Curricula for researchers 7. LIAISE Curricula for policy-makers
8. Concept for IA Book series and IA Journal
9. Launch of LIAISE papers series
10. LIAISE Brief Series
11. High level Conference
12. Final workshop
4.1.3.1.2 Stakeholder Analysis
The LIAISE ambition to better utilize the available potential of the research knowledge reservoir in evidence based policy-making requires a better understanding of needs, preferences, working procedures and options for action in both policy making as well as in research. LIAISE conducted a stakeholder analysis to identify relevant actors and their needs with regard to the relevancy of research. The available data collected in other LIAISE activities, provided a useful source of information for a secondary analysis from a stakeholder perspective. Hence, the data were not specifically collected for a stakeholder analysis, but for other purposes: an analysis of IA practices in EU Member States, the development of a research agenda, the testing of LIAISE services, etc. The analysis is based on data from the following sources:
• About 40 LIAISE workshops: test case workshops, agenda workshops, etc.
• Literature analysis: country studies on the design and implementation of IA processes within LIAISE and LIAISE related projects
• Individual contacts and interviews as part of surveys among 170 policy makers in 17 European Member States and more than 200 researchers
• About 30 special sessions at international conferences
• Mapping of about 200 research projects
• Evaluation of model use in EU impact assessments
The empirical basis and its interpretation for the stakeholder analysis have been validated by the work package leaders who were responsible for these activities. The stakeholder analysis found that there are relatively few stakeholders who are concerned with the meta-level of policy impact assessment, for example by issuing guidelines, researching or steering the implementation of IA. Most stakeholders rather consider IA as a means to achieve a goal in their respective thematic policy domain or research field. A wide group of actors is potentially interested in improving the information exchange in the science-policy interface. Their specific interests not only depend on the type of stakeholder, but also on the use context and the level (strategic or operational) at which a certain decision has to be made. We identified four general practices of IA or contexts of use in which LIAISE provides an added value to improving the situation:
• The development of requirements and guidelines for IA for SD (strategic level)
• The design of an IA process (operational level)
• The decision on using models in an IA process (operational level)
• The funding of research and development of models (strategic level)
Although most stakeholders have a special perspective on IA coming from their fields of disciplinary specialization or policy domain, LIAISE was able to identify general user groups in the different practices of IA, who have similar strategic interests and needs. These actors include horizontal and policy units in governments, international organisations, research funding agencies, stakeholders from civil society and business, consultants, but also individual researchers and politicians. To identify actions to better bridge between research and policy making, LIAISE studied the different practices of IA with regard to the occurring (general) stakeholder constellations, their interests, and the possible LIAISE contributions to improving the science-policy interface in each context. The results are summarized in the following Table 8 as part of Section 4.1.3.4. on Potential Impacts.
4.1.3.2 Main achievements from a research perspective
4.1.3.2.1 The Practices of IA in Europe and beyond (WP1)
WP1’s research on the practices of Impact Assessment (IA) in the countries of the EU was based on the assumption that IA should be treated as one venue among a number in which there is potential for evidence and tool use to inform policy formulation. It was also premised on the understanding that many different kinds of learning are possible through IA. Apart from ‘instrumental’ learning, in which objective information is fed to policy makers, who use it to make better decisions in a rational and straightforward way, ‘conceptual’ or ‘double loop’ learning, which challenges underlying assumptions and changes values and norms perhaps gradually, is possible. WP1 therefore championed the need to understand user needs in the broadest sense, i.e. not simply as a list of how existing and new tools could be developed in an instrumental manner, but considering the richer context and role of IA and IA tools and the wider needs of users. This thinking also led to the conclusion that Test Cases were much more than about testing tools (or instrumental learning), but also about attempting to understand the way practitioners and researchers interact to develop new questions and problem framings (‘conceptual’ or ‘double loop’ learning).
WP1 confirmed that by 2008, all 31 OECD countries had either adopted, or were in the process of adopting, a formal system of policy appraisal (OECD 2009). Building on this, D1.3 - A summary of user needs and expectations regarding IA processes, IA tools, IA training and test cases - reported on the most comprehensive survey yet conducted of user needs and expectations with regard to IA systems and tools in 17 European countries. Previous surveys of this kind had been either narrower in focus or conducted in less depth. None focused (as WP1 did) on the experience and insights of those people at national level who determine the strategic direction of IA, e.g. writing guidance documents, controlling quality and offering technical support for IA.
WP1 collected the following types of information:
- On the origins and purposes of IA: When was IA introduced and why? e.g. to reduce administrative burdens, improve the quality of legislation, improve communication or to integrate sustainable development. What is its purpose considered to be in practice?
- On the design of IA: including, is it mandatory or voluntary; how are policies selected for IA and what are the criteria used; at what stage of the policy making process is IA carried out; how is quality controlled and what impacts are included?
Thus, information was gathered not only on how IA was supposed to be conducted according to guidance and other official documents, but on how IAs are conducted in practice: how did implementation differ from the guidance; how much resource devoted; what impacts were considered in practice, and what quality was achieved? Ideas were also sought on how IA could be improved and what barriers it faces in each country.
The survey found a wide variety in the IA systems studied. The systems themselves, their underlying purposes and the tools they use vary both within and between the 17 countries. Many different factors affect the way they are structured and their functioning. These include the availability of resources (skills, time and data with which to conduct an IA) as well as the steps that have been taken to establish quality control mechanisms and institutions. In terms of underlying purpose, sustainable development was perceived as one relatively minor aim of IA. Reducing the costs of regulation appears to be a dominant motivation in most jurisdictions.
Following the mid-term review, an additional analysis examined the types of tools used in a sample of 325 published appraisals from 8 jurisdictions (this time including the Commission), using a detailed framework which includes a novel, 7-fold classification of tool types (see table 2). This created a systematic picture of precisely which tools are actually used in different systems. Again, this significantly extends the existing literature which focuses on a limited number of cases and jurisdictions.
Table 2 Percentage of cases using certain types of tools, over the periods examined. Note: PA = Physical Assessment tools; MA = Monetary Assessment tools; Model = Modelling tools; Other = Scenario, Multi-Criteria Analysis, Stakeholder analysis and other tools
Jurisdiction Number of appraisals
(period covered) Average length of report
(pages) Simple
(% of cases) PA
(% of cases) MA
(% of cases) Model
(% of cases) Other
(% of cases) No tools
(% of cases)
Cyprus 20 (2009-11) 14 0 0 0 0 0 100
Denmark 50 (2006-11) 2.5 68 4 56 12 2 28
European Commission 50 (2008-11) 84 96 4 44 18 8 4
Finland 50 (2009) 2.5 16 0 18 2 4 66
Greece 36 (2010-11) 17 19 0 14 0 0 78
Ireland 49 (2004-10) 13 33 0 45 6 0 39
Poland 20 (2008-10) 7 60 0 40 0 5 30
UK 50 (2007-10) 38 60 0 92 16 10 4
TOTAL 325
These jurisdictions were chosen for several reasons: they represent a spread of well-studied and less well-studied places; all have reasonably accessible appraisal processes and other government documents that could be studied empirically; and they represent both ‘early’ and ‘late’ adopters of appraisal systems (Adelle et al. 2012) .
Some (e.g. Cyprus, Finland, Greece) appear to use hardly any tools; in the cases of Greece and Finland, more than half the appraisals sampled reported no tools. In Finland, use of a standard checklist is reported to some extent, and other methods are occasionally reported, e.g. partial CBA, but on average the reports are extremely brief (less than three pages). In Cyprus, while an appraisal procedure is present, and a standard form has to be filled in which increases the appraisal's length, no specific tools were reported in any of the cases analysed. Other jurisdictions (e.g. Ireland, Poland, and Denmark) still show a large minority of cases using no tools, but there is more evidence of use of some simple and monetary assessment tools, in around half of the cases sampled. Table 2 shows that only 6% of the Irish cases reported use of modelling tools (all related to building regulations). In Denmark, the reports were very brief and mainly revealed use of monetary assessment and simple tools, with some quantification. A few cases (12%) mentioned modelling tools and two – relating mainly to environment and tax legislation - mentioned physical assessment. But while a wider range of tools was reported than in some other jurisdictions, a relatively large proportion (28%) of cases still reported no tools used. Poland exhibited a similar pattern to Denmark, but while both countries showed mainly use of monetary assessment and simple tools with some quantification, Poland exhibited a relatively lower prevalence of monetary assessment tools. Again, in these jurisdictions, only a few reports mentioned modelling tools and/or physical assessment, and those that did related mainly to environment and tax policy.
Conversely, the Commission and the UK have much richer reported patterns of tool use: only a handful of cases use no tools. In the Commission, almost all of the cases reported use of simple tools, and just under half reported monetary assessment. There are also more cases of modelling tools being reported (about one in five cases) than in any other jurisdiction. The average length of the appraisal reports was also more than double that of any other jurisdiction. The UK, by contrast, showed greater use particularly of monetary assessment, which is mandatory in the appraisal guidelines. A few cases (16%) mentioned modelling tools; these were mainly economic models in the fields of housing, transport and pensions policy. Some such appraisals are very long (more than 200 pages) and contain much detailed analysis, but the majority are rather brief. Few reports mentioned specific tools for participatory analysis; there was often just a consultation with no additional methods employed for synthesising.
Although many countries have sought to learn from one another and international bodies such as the OECD, there is still no one dominant approach to undertaking IA that is firmly institutionalised in all countries. Rather, each country employs IA in a distinctive way which fits its prevailing political and policy context. It is important therefore not to ‘de-contextualise’ IA, especially when seeking to define and extend ‘best practices’ or increase the use of IA tools, such as cost benefit analysis, scenarios or formal computer-based models. Many – but by no means all – of the 17 IA systems surveyed already harness the analytical power of IA tools to inform their assessment activities. In fact, only 9 of the 17 actively promote their use via the production of guidance to the officials undertaking IAs. On the whole, the use of IA tools in practice is highly differentiated both between the main tool types (simpler tools tend to be more popular than more sophisticated ones) and amongst individual IA systems (tool use is generally higher amongst the older Member States than the newer ones). To summarise, the widespread institutionalisation of IA has not yet led to a concomitant institutionalisation of IA tool use. Indeed, many of the countries studied still appear unconvinced of the basic need to increase tool use across the board.
In general therefore, user needs with respect to IA tools defy simple generalisations. They tend to be specific to particular tools and/or IA systems. Instead of ‘saturating’ users with information on tools, this pattern of use calls for a more targeted and ‘smarter’ deployment of existing as well as improved tools; one which is sensitive to the prevailing context in each country. For the least enthusiastic adopters, it may be ‘smarter’ to focus on making the case for IA tools, whereas more enthusiastic adopters seem to want more detailed information on specific (types of) tools. Test cases were seen to constitute a potentially important method to understand these contextual conditions (and thus couple supply to demand), a task that was addressed in WP 6.
Latterly, the project on Impact Assessment in Emerging and Developing countries (D0.8.3) extended the survey work on IA practices beyond Europe and suggests that there are vast opportunities for improving IA practice and tool use in developing countries.
The achievements of WP1 can be summarised as follows:
• Providing a reality check - massive investments have been made in diffusing IA globally, but big differences exist in national practices.
• Supporting knowledge brokerage, documenting and understanding differences in IA practices as a precondition for altering the status quo.
• Building the foundations of the test case work – ‘smarter’ deployment of assessment tools can reveal the potential of IA (but there are also ‘costs’ – time, resources etc. – to take into account)
• Adapting to changing political priorities – since 2008, new demands have arisen (sustainability; carbon; ecosystems etc). WP1 has responded to these changes in stakeholder needs by working with many different partners (OECD, World Bank, UK DEFRA)
• Developing new research agendas – thinking afresh about the uses that many different assessment tools (inc. but not limited to computer models) are put in / alongside ‘non IA’ venues.
4.1.3.2.2 Knowledge for Decision Making: the LIAISE KIT as community platform and library of contextualized knowledge
The need to contextualize knowledge for specific IAs as well as the need for bridging between different research communities and IA practitioners was guiding for the development of the LIAISE KIT (www.liaise-kit.eu)as a toolbox and science policy interface which provides access to relevant knowledge. The platform
- Is based on extensive review of the user requirements and the practice of IA,
- Builds on and integrates knowledge from previous EU funded projects,
- Has been developed in close interaction with users and a systematic and extensive collection of feedback on various test versions.
- Represents a model of the IA process and the substantive requirements of integrated IA which enables users to describe their research and expertise in the context of IA,
- Provides user roles and rights for self-maintenance with a minimum of centralized requirements for administration.
It has been developed as a web based knowledge and community platform which
1. Provides access to scientific knowledge in the context of Impact Assessment for Sustainable Development and
2. Provides a platform to exchange news and for collaboration on issues related to Impact Assessment
With these functionalities, the LIAISE KIT provides a Science Policy Interface (SPI) that makes scientific knowledge available for the use in IA and facilitates the exchange of researchers and IA practitioners. It does so by describing scientific knowledge in the context of IA for Sustainable Development. Scientific knowledge is typically described and categorized by scientific criteria using disciplinary categories and keywords. Assessing the relevance of knowledge for application in societal decision making would either require technical and disciplinary knowledge on behalf of the user in order to assess the suitability of knowledge for the specific purposes or trusted relationships with scientist or consultant which may act as a knowledge broker. In this situation, it is particularly difficult for innovative tools to be considered for application in IA.
The LIAISE KIT facilitates the process of selection and interaction between researcher and IA practitioners by facilitating a description of IA relevant knowledge using keywords from the sphere of IA, and combining them with the disciplinary/scientific description of knowledge. Thereby, knowledge gets accessible for IA practitioners also beyond their disciplinary skills. Furthermore, by knowledge is being contextualized in the domain of IA. This facilitates the process of knowledge brokerage and the identification of relevant knowledge for the purposes of a specific policy proposal. In order to achieve the functionalities, the platform has been developed as a science policy interface on the basis of extensive user interactions (both IA practitioners as well as IA researcher) and testing. Several versions have been developed and features have been stepwise tested and implemented.
Scientific knowledge which may be used in the context of IA is implemented as different content types. The types of knowledge include (in brackets are current numbers of the respective type):
- Descriptions of models (ca. 100)
- Descriptions of methods (ca. 44)
- Expert profiles (ca. 60)
- Publications (ca. 370)
- Examples of good practices of IA (ca. 130)
- Model applications (ca. 135)
- Projects (ca. 40)
- Datasets (external sources)
These types of knowledge are modelled in different content types and are described by taxonomies. The taxonomies represent the domain of IA and in particular the different procedural steps and substantive requirements for IA. The taxonomies include:
- Impact areas (environmental, social and economic) (ca. 170)
- Process of IA divided in IA activities (ca. 30)
- Economic sectors (NACE)
- Spatial aspects (all countries and scales according NUTS)
- Policy areas (54)
- Policy instruments (11)
By combining keywords from the different taxonomies, the specific context of an IA can be described. For example, a policy proposal can be described as a policy instrument (e.g. regulation) in a policy area (e.g. health policies) at the national level (e.g. in Italy), affecting an industrial sector (e.g. chemical industry). The IA may address specific impact areas which are considered as relevant and can be broken down in specific steps of an IA.
The search engine of the LIAISE KIT is based on facetted search. The content can be searched by combining different taxonomy terms and a drill down of search terms which are relevant for a specific Impact Assessment. Thereby the user can identify different relevant pieces of scientific knowledge for his/her purposes. By relaxing specific requirements, similar items can be identified which may be still relevant for the specific purposes although they do not meet exactly the requirements.
Besides the content types and the taxonomies, a third element of the LIAISE KIT are the user roles and related rights. The following roles have been defined:
- Anonymous users can view most content
- Editors can upload and edit model descriptions, publications and other practices, and expert profiles
- News agents can upload new items (see below)
- Lead editors can edit impact areas or descriptions of methods (see below)
With this roles, it is ensured that IA practitioners may use the toolbox without registering.
The content of the LIAISE KIT is derived from a range of sources:
- Taxonomies are derived from an analysis of IA requirements, e.g. in guidelines for IA
- Analysis of literature, project reports and IA reports on model descriptions, methods, good practices of IA and model applications
- Descriptions of models, their applications, publications, expert profiles are uploaded by model developers and authors.
- Contributions of lead editors on method descriptions and impact areas: Lead editors are volunteers that provide state of the art overview text on certain sections of the LIAISE KIT, namely specific impact areas (such as employment, climate, innovation, etc.), or IA methods (such as cost benefit analysis, focus groups, surveys etc.). The contributions of lead editors are quotable and highly visible within the LIAISE KIT and for external search engines such as google. The contributions are provided in the framework of an easily editable wiki. A large share of the lead editors have been recruited from outside the LIAISE consortium and the platform provides a platform to disseminate expertise and project findings.
- Datasets are harvested from semantic web services. The EEA publishes its datasets via a SPARQL endpoint. The concepts and keywords which are used by the EEA to describe the data are matched with the LIAISE KIT taxonomies. The EEA semantic web services are queried by the LIAISE back office and the results are returned to the front office. Using these semantic web technologies, the LIAISE KIT can be integrated in the “web of data”. The LIAISE KIT content and the related ontologies are made available in standard formats (RDF) to be consumed by other data sources.
The functionalities as a community platform include the following features:
- A news blog informs the users about recent developments within the LIAISE KIT (e.g. new features, experts, models, publications, taxonomy terms, etc.) and in the wider community (e.g. events, calls, etc.).
- Every second month, a selection of news items is assembled in a newsletter (IA bulletin) and mailed to ca. 700 subscribers. The subscriber of the IA Bulletin are IA practitioners, IA researcher and LIAISE KIT user.
- The LIAISE KIT hosts a description of the LIAISE network, including the partner organisations, projects that are conducted within the LIAISE network, the LIAISE PhD network (offspring) and an overview on the LIAISE Lead editor.
- The community function is also facilitated by groupware functions. Closed user groups can be set up and administrators are appointed to manage the participation in these groups. The groups can be used for collaborative work on texts. There is a pre-defined format of a group which provides a template for an Impact Assessment. The template is structure along the steps of an IA and provides links to relevant resources (e.g. good practices, publications, methods) of the LIAISE KIT for the respective step of the process.
User guidance and a help desk is implemented in several levels:
- A manual which is accessible for all user describes the LIAISE KIT and its various functionalities
- A help desk manual provides a documentation for the administration of the LIAISE KIT
- Technical support in using the LIAISE KIT, software updates and backup is offered by Freie Universitaet Berlin. However, the software is designed to keep these efforts at a minimum. Registered users can upload and maintain their data typically without any support.
- Lead editors and experts can be approached for issues related to their fields of expertise. During the development and testing of the LIAISE KIT it turned out that a help desk represented by a single person or a small group does not dispose of the necessary skills to support IA in a meaningful way. A public forum has been tested to discuss IA related questions in a wider audience, but this feature had to be dismissed because of lack of acceptance among IA practitioners to discuss sensitive issues in a wider public. Therefore, the lead editors and closed groups have been established to provide targeted and detailed support for the various aspects of IA.
Quality assurance of the LIAISE KIT content is provided by different means:
- The feature of user feedback in forms of stars and comments as foreseen in early versions of the LIAISE KIT has been dismissed. IA practitioners indicated that they are unwilling to provide feedback, the number of potential users is too small to provide impartial and meaningful responses and the context of use determines quality and hence a judgement cannot be transferred to other contexts.
- A first proxy of quality is the level of detail which is provided in the description of IA models and methods. A reference model has been developed as a framework for a standardized description of IA models. Models are described from different viewpoints of IA, e.g. applicability on different sectors, regions, scales, impact areas, etc. The information would allow the user to assess if a model fits for its specific requirements.
- A second proxy of quality is the actuality of the information. Outdated model descriptions are moved to archives. The user are still able to query in the archives.
- A third proxy of quality are past applications and publications in peer reviewed journal articles. The modelers are invited to add publications and model applications.
- A fourth mechanism to assure quality are the lead editor. Contributions to IA methods are reviewed by the lead editor. A more formal approach for peer review of contributions to the LIAISE KIT has been considered but due to lack of time and resources the implementation is postponed.
Given the importance of user requirements and the specific context of Impact Assessments, the LIAISE KIT provides a platform to facilitate user requirements analysis. A throughout user requirement analysis is vital for developing knowledge which is applicable and relevant in decision making. Throughout the LIAISE project, we have tested several formats and approaches for the analysis of user needs:
• We realise that there is no ‘one size fits all’ approach, as the users of IA tools are inherently diverse and have complex requirements.
• 1:1 interviews typically provide the richest data, but are time consuming and require a substantial buy-in of users.
• Workshops and focus groups compromise in insight, but provide viable options and allow for follow-up with individual users later on.
• Questionnaires and electronic communication facilitate outreach and allow for gathering of input from a wide range of users.
• Challenges in time and availability of both tool developers and users to engage in a dialogue on requirements.
We conclude, that user requirements analysis is a process, not a one-off activity. This has profound implications for future user engagements and has emerged as a key message from the activities which involved direct communications with IA users. It was for instance illustrated by the fact that engaging with an established group of users working towards well-defined common objectives (e.g. regulatory IA in the United Kingdom in the case of Air Pollution Information System -APIS) facilitates communication. In addition to that, the development of clear, concise targets for new/improved IA tools can be established this way.
Working with usability and user requirements experts can aid tool developers (who are often scientific researchers, not software engineers) in eliciting user requirements. This could be a substantive contribution by a community of practice as outlined in the following, bringing together experts from different backgrounds, tool developers and users.
We have identified a need to differentiate between adding features and functionality and improving usability. This helps to determine where mainly technical improvements of the capabilities of tools are required, or how usability improvements could be made e.g. by better documentation or training in the use of tools.
The LIAISE-KIT can contribute to this process by establishing fora and connecting tool developers and users to foster a lasting dialogue, build trust and improve the understanding of user needs.
To summarize, the LIAISE-KIT:
• Provides a framework to describe knowledge in the context of IA
• Enables researcher to upload their publications, models, projects
• Provides a platform for researcher to publish on methods and impact areas
• Is harvesting and contextualising external data sources
• Is a platform for collaborative work on IA
• Provides a platform to post relevant news
• Can contribute to the user engagement process by establishing fora and connecting tool developers and users to foster a lasting dialogue, build trust and improve the understanding of user needs
It is thereby contextualizing knowledge and provides a platform to access innovative tools which is relevant for specified purposes of IA. Its content types (various types of knowledge), taxonomies (structured keywords from the domain of IA), the user roles (enabling and encouraging contributions to the platform) and the backoffice tools which contextualizes external data sources facilitates the science-policy interaction and overcomes path dependencies in the use of models and unrealistic requirements to assess the applicability and relevance of scientific knowledge in a specific context on behalf of IA practitioners.
4.1.3.2.3 Co-Design of Knowledge - Experiences from LIAISE Test Cases(WP6)
LIAISE explored IA tool use and knowledge exchange in six real-world cases to create a realistic understanding of the requirements of knowledge users in relation to possibilities of knowledge production. Testing was inspired by knowledge brokering approach, which challenges the ‘linear’ knowledge exchange by allowing exchange, co-evaluation and joint construction of knowledge with the aim of enriching decision-making. In particular, the test cases were interested in finding out in what circumstances, contexts, certain knowledge brokering approaches are fit for purpose in IA? In addition, the test cases aimed at investigating procedures for interaction between researchers and policy-makers. Thus, from the IA perspective it is important to find out when a step-wise approach of interaction in IA is suitable and when more dynamic and iterative processes are needed? Furthermore, the test cases examined current use versus possible uses of existing impact assessment tools.
Test cases (Table 3) played an important role in LIAISE, because they provided practical information both on tools in practice and user needs. Test cases enabled good relations between researchers and policy makers, and contributed to the main objective of LIAISE NoE to bridge the gap between these actors. Test cases involved several research questions/objectives, which were specified by individual test cases.
LIAISE's Test case package comprised of six cases varying from EU, national and regional level within an EU member state and in China. Interaction between experts and policymakers was core activity in all test cases. Different LIAISE aspects were tested depending on the focus of individual test case: LIAISE-Kit (former Toolbox), IA libraries; communication: IA innovation reporting/ policy briefs/ IA bulletin; Training curricula; Shared Research Agenda; Use of tools & informed use of tools; Understanding interaction between policy & research; and Awareness of LIAISE.
Table 3 Characteristics of the LIAISE test cases
Test Case Main Focus Key aim
Agri adaptation EU-level climate change adaptation in agricultural policy Develop knowledge brokering process enhancing the use of models in IA
Resource efficiency National and EU-level resource efficiency policies (Germany) Test how far knowledge platforms can contribute interaction
Climate policy National level climate and energy policy (Finland) Explore and develop interaction between modellers and policy-makers
Energy policy National level energy policy (Estonia) Explore how policy developers choose quantitative models for IA
Agri farmers Regional level agricultural policy (Greece) Develop best practise interaction process for IA of regional policies
Land use Regional level land-use policy (China) Test participatory tool combining scientific knowledge with stakeholder perceptions
During LIAISE testing process the knowledge brokering was summarised simply as a process of communication between researchers and users. But as there are different kind of policy process with variety of needs for communication and interaction, there are also several different strategies for knowledge brokering in practice. Thus, there is no single ideal form of brokering, but the choice of strategy depends on the context of the case. The knowledge brokering approaches applied and tested in LIAISE test cases were based on Sarah Michaels (2009) categories and are:
• Informing: one-way knowledge dissemination by researchers
• Consulting: provision of knowledge by researchers requested by policy-makers
• Match-making: broker identifies what, by whom and how knowledge can be exchanged in form of value to policy-maker
• Engaging: one actor (usually policy-maker) frames the discussion and knowledge exchange
• Collaborating: actors jointly frame the process of interaction and negotiate substance to address a policy problem
• Building capacity: actors jointly frame the process of interaction and negotiate substance to address multiple dimensions of a policy problem considering what can be learned for future
In reality test cases were versatile and included application of many knowledge brokering strategies.
Lessons learned for co-design of knowledge
Identifying and acknowledging the context of a specific policy case and IA is crucial: based on the literature and TC experiences WP6 developed a typology of different contextual factors affecting the selection and success of the Knowledge Brokering strategies and individual means and forms of activities in IA. Examples of contextual factors relevant for IA process are presented in Figure 1. A clear challenge for most of the test cases was the interaction with policy makers. One reason for this might be that most – if not all – test cases were rather ‘supply-driven’ meaning that they had been initiated and mainly also planned by LIAISE researchers. Test cases were rather forced to carry out their activities during a certain period (the LIAISE testing period) which was necessarily not tied in with the real-world policy processes or policy makers’ needs. In addition, policy officers have established tendering processes and contacts to the other information producers (e.g. consultants and academia). It became evident during testing that knowing and acknowledging the context of a specific policy case and IA is crucial.
Test cases faced several challenges during the testing process. ‘Match-making’ knowledge brokering approach, for example, was proven to be non-sufficient to embed research-driven tool development process in policy area. It, however, opened up a route for communication between researchers and policy-makers. It might also be difficult for researchers to follow in-house routines of policy-making (e.g. there might be delays in interaction due to bureaucratic reasons). ‘Collaboration’ knowledge brokering approach was less successful for jointly framing the tool development exercise because open policy questions did not fit within the limitations of the existing models. For researchers this appeared as ‘lack of demand for a tool or knowledge’. Furthermore, both researchers and policy-makers might have been reluctant to move beyond ‘business as usual’ in the knowledge exchange during IA. Some researchers also felt that linear IA is process it is too rigid for complex and multi-level policy problems and processes.
Figure 1 an overview of the issue with relevant lessons learned from the test cases
Table 4 Barriers for successful knowledge exchange in IA identified in test cases include
Policy-makers: Researchers: General:
• Diverse needs
• Thematic thinking vs. SD
• Ownership of IA – no room for out-house researchers
• Monopoly, non-open models
• Timing of knowledge supply
• Power struggle between policy sectors
• Few knowledge producers (national and regional level)
• No second opinions available (national and regional level) • Supply-driven models and tools
• Disciplinary instead generalist approaches
• Competition between researchers, models and dialogs in science
• Gap in use because of publication process
• Different approaches in science/ methods/ disciplines/ ontologies/ worldviews
• Endangering trust is avoided – results not given until final – lack of testing • Selective use of proof
• Pre-defined agendas
• Business as usual relations between actors
• Organizational changes
• Lack of continuity – project based culture
• Broking between tools and policy questions missing
• Very technical focus of IA and use of evidence
• Lack of openness and flexibility
• Non-focus of motivation and objectives of SPI
Despite of the challenges, the test cases succeeded in building collaborative relationship with the policy-makers during testing. Tools with wide range of application possibilities (e.g. flexibility to accommodate new goals) appeared to be particularly useful and provide an opportunity for interaction. In addition, it appeared that face-to-face meetings with policy-officers and researchers’ active role in promoting a new tool for IA in the scoping phase of the IA are very useful. Engaging active and specialised policy-makers helps researchers to get technical feedback and tailor supply-driven tool development. LIAISE Kit can also serve as a platform for exchange and networking, especially for informing and match-making knowledge brokering. IA researchers can furthermore act as a facilitator in policy-policy interaction by organising seminars and focus group meetings related to inter-ministerial multi-dimensional problem framing and solving (match-making and capacity-building knowledge brokering). It must, however, be emphasised that the success of knowledge brokering, tool application or tool improvement is possible only when the role of the IA knowledge provider or knowledge broker is clearly defined in scoping and planning phase and there is trust and credibility on both sides.
Table 5 Opportunities for future knowledge exchange in IA identified in test cases include
Policy-makers: Researchers: General:
• Many needs
• Support for holistic thinking, trade-offs
• Enhanced use of existing proven tools
• Consistent approach preferred – procedural tools
• Engaging with other policy sectors in complex issues
• More knowledge producers needed (national and regional level)
• Second opinions needed (national and regional level) • Early engaging in model and tool development - increase in use
• Generalist approaches useful in complex problems
• Meta models - interdisciplinary approaches
• Publication of policy briefs on ongoing activities – expectation management
• Openness about modelling (assumptions etc.), increase in trust and credibility
• Researchers’ achieving better broking skills • Comparative use of proof
• Flexible, changing agendas
• New opportunities for boundary organisations and brokers
• Network culture – durable IT or personal ad hoc support
• Matching tools with policy questions
• Co-design of knowledge instead of linear transfer
• Constant (con)testing of knowledge: no ‘orphan’ knowledge
• Open co-tailoring
Collaboration in IA increases the use of IA results in policy-making but requires trust and credibility based on previous cooperation or built during (long) IA process. The IA researchers/knowledge providers must, however, be open about limitations of a tool to increase credibility. Developing a new tool for real-life ongoing policy-process is rather demanding due to time and political constraints. Instead, there might be room for tool tailoring or improvement.
Based on LIASE test cases it appears that iterative co-design of knowledge is enabled by tools that are:
• Already existing
• Easy-to-apply/ transparent
• Time-saving for policy actors
• Procedural but allowing choices, and
• Including an element of co-tailoring relevance by policy-makers’ questions
4.1.3.2.4 The Shared Research Agenda for IA
Survey on scientific tools designed for policy IA
LIAISE conducted an extensive survey on scientific tools designed for policy IA. We analysed the abstracts of 7781 projects funded in the European FPs 6 and 7 provided on the European Cordis website (Podhora et al., 2013 , LIAISE Innovation Report #6, LIAISE Policy Brief #5, D2.2 D2.3). We selected 203 projects that developed, extended, applied and/or tested tools for the IA process. We concentrated on quantitative or qualitative tools (models, participatory tools etc.), their components (e.g. indicators and comprehensive analytic methods) and superior evaluation frameworks (toolboxes and platforms). We structured the analysis of the projects that designed the tools according to (i) 36 European policy areas (European Union, no date), (ii) 35 impact areas outlined in the European IA guidelines (SEC, 2009 , amended by “sustainable development in general”), (iii) the jurisdictional levels (from international/global to local), and seven tool categories (de Ridder et al., 2007 assessment framework , scenario analysis tools, multi-criteria analysis tools, cost-benefit/ cost-effectiveness analysis, accounting tools, physical analysis tools and indicator sets, modeling tools, amended by category “other”). In each of these groups, we counted the number of projects to identify research peaks and gaps. The analysis of the projects funded identified the following key results:
• Policy-relevance of research: A small percentage (less than 3%, equalling 203 projects) of the projects funded in FP6 and 7 provided tools for policy IA. About half of these projects identified provided tools for environmental, agricultural and transport policy areas. Tools designed for these three policy areas were subject of a further in-depth analysis presented in the next points.
• Impact areas and sustainability dimensions: The tools mainly addressed the impact areas corresponding to the policy areas (environmental impact areas for environmental policies, the impact area “land use” for agricultural policies etc.). Social impact areas were generally poorly addressed by the tools. The tools were mainly designed for one to two sustainability dimension(s). They hardly comprised all three sustainability dimensions and sustainable development in general, respectively.
• Jurisdictional levels: The tools were mainly designed for European policies and hardly for other jurisdictional levels (as international or national levels). They were mainly designed for a single jurisdictional level and not for multi-level governance.
• Tool categories: The majority of the projects designed several tools. Most tools had a quantitative character; participatory tools were poorly covered. More than half of these tools could not be categorized according to the current seven integrated assessment categories identified by de Ridder et al. (2007) .
• Terminology challenges: Many tool descriptions did not refer to the policy-relevant terms defined by the European Community (e.g. impact areas as set out in the European IA guidelines, policy areas of the European Union).
LIAISE SRA – methodology, framework, target groups
The setting of research agendas has been an emerging topic especially in the past five years. So far, research agendas have mainly been developed from two different communities and thus from two different angles. They have been either designed by the scientific community (e.g. EPBRS, 2010) or by the policy-making community (e.g. ESFRI, 2008) . So far, no research agenda could be identified that exclusively focussed on policy impact assessment (IA). However, a few authors addressed aspects related to IA (Turnpenny et al., 2009 on policy appraisals and Pope et al. 2013 on various IA instruments) .
The results of the survey clearly identified the need for a Shared Research Agenda for policy IA. Its development was based on seven steps:
1. Research gaps identified for IA related research projects funded in the European FPs 6 and 7
2. Expert workshops on ecosystem services, soil and transport/ innovation in relation to the three leading policy areas
3. Gathering and discussion of research gaps within LIAISE work packages and test cases
4. Presentations and additional discussion rounds
5. Translation of research gaps and results into research questions presented in the LIAISE SRA
6. Comments from and public discussion with scientists and policy-makers at the LIAISE dissemination conference
7. Dissemination
The LIAISE SRA addresses the six target groups (see figure 2), three from the scientific community and three from the policy community, while here the European Commission is used as an example system:
1. Scientific community:
a. no experiences with policy-oriented research
The SRA could assist this group to align their research towards a more policy-oriented focus. This support may also help researchers to increase the policy-relevance of fundamental / basic research.
b. (first) experiences with policy-oriented research
The SRA could support researchers who already are experienced with policy-relevant research to identify, strengthen and discuss possible topics for their upcoming research.
c. political and sustainable development sciences as policy-relevant research per se
The SRA could provide insight for political and sustainable development scientists to better understand open questions the scientists (who are no experienced with policy-relevance) have to address when delivering evidence to policy-making.
2. Policy-making community:
a. research policy-making
The SRA could provide information for the design of research policies, specifically the upcoming research programmes of Horizon 2020 in the fields of IA in general as well as in specific disciplines (e.g. in the exemplary themes soil, ecosystem services and transport/ innovation in relation to IA).
b. European research institutes/ in-house research
The SRA could assist European research institutes that work as policy-related knowledge brokers to better understand the kind of open questions that need to be covered for better providing scientific evidence for the policy process.
c. European DGs
The SRA could provide information about research gaps in IA for the European DGs (e.g. with respect to the current SRA focus particularly to the DGs AGRI, ENV and MOVE).
Figure 2 Target groups of the LIAISE SRA
The LIAISE Shared Research Agenda for Policy Impact Assessment
The SRA identified existing IA tools and outlined research needs based on expert discussions. The SRA thereby addressed all IA spheres: the policy sphere (tool uptake/ use), the scientific sphere (tool provision) and a combination of both spheres to improve the mutual understanding of the individual needs and interests within the spheres. It thereby concentrated on the exemplary themes “IA in general”, “ecosystem services”, “soil” and “transport/ innovation” as a starting point. These themes were selected due to their relation to existing IA relevant research as well as current and future policy requirements. These examples helped to identify strengths, weaknesses and implementation challenges of the present SRA, particularly to provide information for a continuous update of the SRA.
The LIAISE SRA consists of three levels that build upon each other as a pyramid (Figure 3). It focuses on three exemplary themes: IA in general, ecosystem services, soil and transport/ innovation (see below).
Figure 3 Levels of the LIAISE Shared Research Agenda pyramid and scientific target groups
Level 1 addresses the general research scope for policy IA and suggests a general structure for research funding in relation to the science-policy interface and IA. The information can be offered to research funders to design research programmes. The structure comprises the three spheres of the science-policy interface:
- Policy-relevant topics: Policy-relevant themes as policy and IA processes,
- Scientific tools: Assessment methods as provision of tools and data and methodologies, and
- Knowledge transfer: Bridging the science-policy interface.
Level 2 suggests guiding research questions that detail the three sections of level 1 for each of the four exemplary themes of the LIAISE SRA, for example, for IA in general: Topics: How do different scales influence the IA process and results? And Transfer: How can results be presented and communicated in a way that supports their inclusion into the policy process?. Another example of such research questions for soils is Topics: How can the impact of land utilization on soil functions and services better be incorporated into policy-making? Tools: How can harmonization and standardization of classification, management and storage systems for data, tools and indicators serve for the purpose of IA?
Level 3 details the guiding with specified research questions for the four themes. Both levels may assist the funding bodies when specifying their research programme and researchers when outlining their superior research questions (e.g. in internal institutional programming, institutional working groups, large research projects). They provide suggestions for the actual design of new and interdisciplinary research questions.
The next steps of the LIAISE SRA
The continuous update is a central element of the LIAISE SRA. The constantly high level of the quality of the SRA can be reached by linking the updates to the LIAISE KIT as expert, networking and communication platform. It is structured along policy-relevant and scientifically relevant taxonomies as impact areas, IA methods and IA models. Further, it is important that the future version of the SRA reflects the comprehensiveness of sustainable development. The relation to sustainable development relevant in the IA process can also best be addressed by the kit due to the taxonomy of the 35 European impact areas. A SRA lead editors in the KIT will coordinate the updating process. When updating the LIAISE SRA it is important to constantly invite the wider IA community to participate in the updates and to inform the policy-making and scientific community on the results. Thereby, LIAISE shall work hand in hand with well-established (research) networks and associations. This updating concept thereby relevantly contributes to the LIAISE objective of further consolidating the IA research community.
The LIAISE training modules provide support on policy-relevant research to all levels in a scientific career, from early to senior stages with view to policy-relevant and IA related research. The training module B for advanced and senior scientists reflects on the structure of the LIAISE SRA: policy-relevant topics, scientific tools and knowledge transfer. The main objective is to increase the autonomy of the participants to individually check the policy relevance of their research proposals and projects by means of the LIAISE self-test. Further, early career researchers are to become familiar with policy-relevant research with module A to help shaping their expertise and profile.
4.1.3.3 Synthesis: Situational analysis of the gaps
LIAISE conducted an empirical study on the reported tool use in EU impact assessment (Jacob et al. 2013 – published on LIAISE website). The study examines all 805 IA reports on EU level, which were published between 2003 and 2013. In general, it can be observed that scientific knowledge is still underutilized in policy-making processes. For example, numerous scientific models are available which are able to cover a variety of policy issues, different scopes and scales. However, the analysis showed that only a relatively small number of IAs on EU level used those models in the impact analysis. Out of the 805 IA reports that were published in the evaluation period, only 222 (27,6%) reported the use of quantitative models for the assessment of the impacts. Model use has significantly increased over time: from 5% in 2003 to 39% in 2013. However, the analysis also showed that the usage of scientific models still varies considerably among the different policy DGs. For example, in DG CLIMA, 90% of IAs were based on modelling results, while e.g. in DG MARKET and SANCO only a minority of IAs have been reported the use of models.
Figure 4 Reported use of quantitative models in Impact Assessments by the EC
Furthermore, the analysis has revealed that only a small part of the models which are described in the LIAISE KIT have been utilized in IAs. At the time of the analysis, 19 models described in the KIT have been used in EU IA, while 72 have been never utilized. This points to the wider research reservoir that could contribute to future IAs.
To conclude: although the application of models has significantly increased in the last decade, the full potential of scientific knowledge is not yet utilized. The perceived gap between the availability of scientific knowledge and it’s consideration in actual decision-making processes that was the starting point for the LIAISE project, can still be observed. The underlying causes will be described below.
A first explanation for the gap between research and practice is that IA for SD belongs to the category of wicked problems, described in Rittel et al. 1973 . Societal problems that are difficult or impossible to solve due to: incomplete or contradictory knowledge; the many values, interests, opinions and numbers of people involved; the economic consequences and complex relations with other societal challenges. E.g. health-nutrition; nutrition- education-income; income-economy; economy-demography-natural resources. Wicked problems cannot be solved by traditional, linear systems of knowledge production with researchers providing undisputed ‘truth’ to society. Here the role of science is to deliver knowledge and information that users perceive as salient, legitimate and credible . In basic sciences the linear scientific process is the only way forward. Scientific progress does not depend on input from and interaction with policy makers. Wicked problems however require a truly transdisciplinary approach: strong horizontal interaction with external stakeholders in a joint process of innovation and shared learning . The compliance testing work in LIAISE on the reality of policy making confirms this general picture. It revealed a gap between the aims of IA and its actual use. The application of IA is difficult and often at odds with the process of decision making . A systematic review of research-practice interaction around policy appraisal concludes that practitioners and IA researchers are not proceeding their practices and research in parallel but are ‘drifting apart’.
A second cause for the existing gap stems from the different types/cultures of knowledge production within the scientific community as such. Table 6 gives general characteristics as described by Gibbons . The sharp distinction between Mode 1 and 2 of course is a simplification. In the LIAISE NoE many researchers work in both modes, albeit with unequal preferences. There were quite some occasions where differences in the scientific cultures had to be addressed in order to come to a productive collaboration. This in particular applied to the case studies with a mix of modellers from the basic sciences and researchers from policy sciences. Specific IA related gaps in this category were identified in a LIAISE evaluation of nine years of European research funding . E.g. boundaries between disciplines, impact areas and jurisdictions and problems related to geographical scales and time horizons.
Table 6 Mode 1 and Mode 2 knowledge production (Gibbons et al. 1994).
Mode 1 Mode 2
Problems set and solved in the context of the (academic) concerns of the research community Problems set and solved in the context of application
Disciplinary Transdisciplinary
Homogeneous Heterogeneous
Hierarchical, tending to preserve existing forms of organisation Heterarchical, involving more transient forms of organisation
Internal quality control (peer review) Quality control is more socially accountable
A third element of general relevance for the gap is the lack of a common platform for information exchange and discussion about the production and use of IA knowledge across sectorial and disciplinary divides. Such a platform could enable IA knowledge workers to jointly develop a shared research agenda and plan-for-action. Some international communities (e.g. IAIA, TIAS) exists, which operate as loose networking fora to share developments. More in depth confrontation and alignment would enable a more efficient use of resources towards a common goal. Networking mechanisms are not in place to jointly explore the possibilities and engage in each other’s discourses. The lessons learned in our internal interaction processes in the NoE provided useful building material for the vision on a future Community of Practice.
A common platform for knowledge exchange could enable IA knowledge workers to jointly develop a shared research agenda and plan-for-action. Some international communities (e.g. IAIA, TIAS) exists, which operate as loose networking fora to share developments. More in depth confrontation and alignment would enable a more efficient use of resources towards a common goal. Networking mechanisms are not in place to jointly explore the possibilities and engage in each other’s discourses. The lessons learned in our internal interaction processes in the NoE provided useful building material for the vision on a future Community of Practice
Potential Impact:
4.1.4 Potential impacts: main achievements from a societal perspective
4.1.4.1 Intro: the LIAISE approach to generating impacts
The ambition to aim for long-term impacts that go beyond the project objectives was an important feature of the LIAISE approach towards evidence based policy-making. To achieve this, the on-going evaluation by Ecologic (section 4.1.3.1.3) was included in the project design to facilitate internal communication; to take stock of where the project is heading to; to support reflexive and constructive discussion on both project objectives and results; and to decide on possible adjustments. In doing so the basic questions are: 1) are we doing the right things, and 2) are we doing things right? The Impact Chain Approach was used as a method for structuring this discussion with Inputs Activities Outputs Outcomes Impacts:
• Inputs are the resources that are used in an activity. This level of detail is not relevant for the MB discussion at this stage any more (it was during the proposal writing).
• Activities are what you actually do to create the change you want to achieve.
• Outputs are the most immediate results of the activities in the work packages. They create the potential for outcomes and impacts to occur.
• Outcomes describe the true changes that occur as a result of the activities.
• Impact is the vision of a preferred future that underlines why the action is important. It refers to the longer-term change that you hope your action will help create.
Table 7 Examples of applying the Impact Chain Approach to LIAISE
Activity Output Outcome Impact
WP3 (toolbox back office) and WP4 (toolbox front office) Toolbox More used tools & better informed use of tools in IA Better informed policy making
WP2 Science for IA tools Mapping of IA research Increase awareness of IA (Research & Policy) & research funding agencies Set better targets and trigger actionable research on IA
WP5 (durability) Business plan Self-sustainable funding Continuous science/policy interface
WP7 (dissemination & training) Training curricula Users have skills Better informed policy making
LIAISE used this approach by thinking backward along the chain, starting with the desired impacts by asking “what do we want to accomplish”; next to ask “which outcomes and outputs are required” and finally “which activities are needed”. Table 7 gives some examples; in reality more impact chains were needed to ‘funnel’ a number of activities in different WPs towards one single impact.
The following steps have been taken:
• Workshop with the MB (Amsterdam, December 2010). Presentation and adoption of the impact chain approach as the method for the evaluation. Identification and ranking of project outcomes and impacts.
• Annual Meeting (Milan, January 2011). Presentation of the method and results of the MB workshop. Discussion of the structure of the evaluation and the linkages to WPs. Adaptation and further refinement of the ranking of outcomes and impacts.
• Presentation to the Policy Board (Brussels, February 2011). General approval of the proposed approach.
o Creation of a logic model:
o Elaborating on the information gathered from stakeholders.
o Creating causal impact chains by linking outcomes and impacts to outputs,.
o Reviewing and refining the logic model in discussion with LIAISE partners.
o Developing indicators and an evaluation timeline in line with the deliverables schedule.
• Conducting the evaluation and reporting.
4.1.4.2 Charter and vision on a future Community of Practice
For mobilising the available scientific knowledge reservoir on IA for SD to its full potential, it is crucial that the societal stakeholders perceive the contributions from science as salient, legitimate and credible . This requires changes in:
• The ways IA research is programmed and evaluated in interaction with users;
• The ways scientific researchers and knowledge workers in other sectors of societal activity produce and exchange new knowledge and information;
• The interaction between mode 1 and mode 2 researchers;
• The ways knowledge and information are disseminated in order to better harness the existing knowledge reservoir for the needs of IA for SD;
• The interaction between scientific researchers and policy makers in concrete policy cases. Here special attention is needed for Knowledge Brokerage.
This process oriented view is quite different from the model and tool oriented view at the start of the NoE. The change reflects the improved interaction and collaboration between researchers from basic sciences and social/policy sciences. The lessons learned show that not tools as such, but the information supply about tools should be standardized and that there is a strong need for a platform for strategic reflection by IA researchers on their role and tasks in relation to IA for SD. Research on the environmental, economic and social dimensions of SD is not sufficient to provide the policy process with salient, legitimate and credible scientific knowledge. Additionally requires features are:
• An integrative, open and transdisciplinary approach by a Mode 2 scientific process in dialogue with IA practitioners and policy makers;
• Transparency about what can/cannot be expected from science in a specific decision making process, as well as transparency about trade-offs between the different dimensions of SD and related interests/values/stakeholders; between short/long term impacts; and between geographic entities, or jurisdictions;
• Contextualisation of IA knowledge by: a) actively involving expertise in the field of policy analysis and knowledge brokerage in the scientific process; and b) standardized descriptions and quality evaluation of (transdisciplinary) IA methods and tools. This should enable potential users to assess the relevance of the available scientific knowledge and information for their specific use cases.
• Procedures and norms/standards for research programming and funding, planning and execution, dissemination and quality monitoring/evaluation that support 1-3.
These features will be realized by a Community of Practice on IA Research for SD . for IA knowledge workers in research and other sectors of societal activity. It provides the platform, as well as the networks, procedures and services that are needed to upgrade their scientific contributions to the process of IA for SD by:
• Fostering individuals and organizations to learn and be challenged;
• Transcending traditional disciplinary and science – policy boundaries;
• Actualizing a new type of science, based on inter-disciplinarity, transparency and active stakeholder involvement (transdisciplinarity);
• Collecting, assembling and disseminating critical insights for society on the complexity and relevance of long term perspectives on SD;
• Providing an arena and conducive environment for discussion on IA for SD.
-------------------------------------------------------------------------------------------------------------------------
The Charter
Preamble. SD as a guiding principle has been widely accepted by the public as well as private sector and civil society. Ex-ante IA is becoming increasingly important to ensure public support for, and trust in proposed societal changes towards SD, e.g. in terms of green economy, resource efficiency and sustainable and smart solutions. Ex-ante IA has the potential to reveal the short- and long term economic, environmental and social impacts of a policy throughout all policy sectors and levels of decision making. However, to operationally integrate the principle of SD into day to day IA processes, remains a serious challenge for policy makers as well as for researchers. The Charter is a key element in establishing a Community of Practice on IA Research for SD. It represents the shared views of the IA research community on how to move ahead in improving research on IA for SD in a dialogue with stakeholders from outside the scientific community. The Charter addresses the issues that are seen as relevant for IA for SD from the perspective of scientists (scientific, organizational, administrative / financial and cultural / behavioural).
The LIAISE community is aware of principles for researchers developed in related contexts such as the (Lund Declaration, 2009), the (Bellagio Principles, 1996), or the (DeGEval Standards 2008). However, this charter directly addresses the way research is planned and conducted in the specific context of IA research for SD. The Charter expresses an intentional commitment of the partners in the CoP: a general endorsement of the views, principles and guidelines. Our Commitment. The partners in the future CoP are committed to developing solutions that contribute to SD. It is their aim to overcome the gaps between research and societal decision making identified in the previous section of this document by developing and designing knowledge production that is targeted at achieving societal impact.
Hence, it is the overall aim to create knowledge that meets the following criteria:
• Relevancy: We are committed to creating knowledge that is targeted at making an impact for society and societal decision making.
• Credibility: We are committed to creating knowledge that is trustworthy and embedded in societal contexts.
• Legitimacy: We are committed to creating knowledge that takes into account divergent world views of different stakeholders in a balanced way.
To achieve this, the partners are committed to a set of guiding principles, which are reflected in their activities in the field of research on IA for SD:
• Commitment to quality: We are committed to scientific excellence and will utilize the mechanisms of scientific peer reviews to this end. Besides, we will contribute to the development of other quality attributes of knowledge in order to improve the relevancy for society.
• Commitment to transparency of the scientific process: We are committed to clarity about methods and assumptions underlying the research process as we are aware of the risks of offering IA knowledge and tools as black boxes. We are also committed to including the discussion of their limitations and being explicit about entailed normative world views, which we are willing to discuss.
• Commitment to contextualization: We are committed to carefully analyse the context of decision making and societal problems and to adapt scientific inquiry accordingly in order to link research to relevant societal developments and to ensure its relevancy for the decision making process.
• Commitment to dialogue: We are committed to become engaged in dialogues with societal decision makers in order to better understand and respond to societal knowledge needs. We are also committed to dialogues with decision makers and research funders targeted at collaborative research programming. In this way research questions shall be developed across different disciplines and together with societal actors to conduct research that meets societal needs.
• Commitment to considering long-term and integrative perspectives: We are committed to carefully analyse potential trade-offs and synergies between the different dimensions of SD and the short and long-term perspective to enable decision makers to meet the challenge of considering these trade-offs in policy making. This entails openness for interdisciplinary research.
These principles should be considered and applied adequately on a case by case basis. Not every research project and not every IA requires a consideration of the principles of IA research for SD. The principles open up additional options, rather than being meant as prescriptive.
-------------------------------------------------------------------------------------------------------------------------
4.1.4.3 Stakeholder Analysis
Table 8 Stakeholders, stakeholder interests and potential added value LIAISE
Stakeholders
Horizontal Units: units overseeing quality of IA process (e.g. IA units at EC or national level, EC IA Board) Stakeholder interests Potential contributions / added value LIAISE
Networking and discussion forum Meta-data on IA for SD knowledge Repository of methods and models on IA for SD Good practice examples of IA for SD Methods for transdisciplinary research Good practice examples of trans-disciplinary research Quality standards for models Shared Research Agenda
Context: Development of requirements and guidelines for IA for SD
Horizontal Units Quality of IA process X X
Policy Units Represent own policy issues adequately in IA X X X X
Efficient use of resources to conduct the IA X X X
International Organisations Quality of IA process X X X X
Politicians Increase influence on decision-making process X
Consultants Giving policy advice X X X
Context: Design of an IA process
Policy Units Narrowing down options X X
Politicians Transparency of the analysis X
Legitimacy X
Stakeholders in business / civil society Early influence on policy-making X X X
Transparency of the analysis X X X
Legitimacy X X X
Horizontal Units Ensuring quality of IA X
Context: Decision on using models in an IA process
Policy Units Applicability of the model X X X
Insight in / understanding of process and results X X X
Horizontal Units Ensuring quality of IA X
Insight in / understanding of process and results
Consultants Application of their models X
Data availability X
Context: Funding of research and development of models
Researchers Scientific interests X X X X
Data availability X X
Policy relevancy (model improve-ment based on feedback from tests in IA practice) X X X
Research funding agencies Ensuring excellence of research X X X
Ensuring policy relevance of research X X X
Policy units Policy relevant models X X X X
4.1.4.4 Ongoing evaluation
As indicated in the DoW the Ecologic Institute has been commissioned to conduct an ongoing evaluation of LIAISE which has accompanied the project from the start and is complementary to the standard FP7 project evaluation procedures by the Commission that focus on monitoring the project progress in terms of the realisation of the project objectives. Here the focus was on providing timely feedback on the progress towards the expected long-term project impacts. This induced critical reflections that enabled the project actors to consider and if necessary, undertake timely adjustments. A first evaluation report was completed in August 2012. It was followed a few months later by the evaluation of the toolbox. Finally, the second round of the evaluation of LIAISE, which was focused on the post-project entity, took place during the last months of the project. Here - on request of the coordinators – special attention was given to the long-term objective of transforming the NoE to a durable network. Ecologic Institute was therefore asked to focus on critically assessing the positioning of the future LIAISE entity in the current IA landscape. Therefore the evaluation was targeted at the different roles and functions of LIAISE as well as the project as a whole. The final report was published at the end of April 2014. This section summarizes strategies to improve the potential impact of the project and to increase the attractiveness of the future network, not only to users, but also to its members. Section 4.1.3.4.1 describes how the on-going evaluation was used to focus on the envisaged long-term impacts of the project.
The recommendations for the various roles of LIAISE are listed in Table 9. The most important recommendations emerging from the evaluation are probably those regarding the leadership and coordination of the LIAISE community of practice (CoP), as it is the foundation of all other aspects of the network and must therefore be solid.
• Increase the focus on the needs of members of the CoP rather than focus exclusively on user needs. While it is important to position LIAISE as a network and ensure that users find it attractive, the network is nothing without its members and their needs must be understood and taken into consideration. If they are to commit to the CoP, they must to get something in return for their time and work.
• The CoP should work with a long term vision but short-term plans rather than only with two-year plans. This will enable a more regular assessment of progress and create more achievable goals, allowing for results to be visible faster and credibility to be built.
• The LIAISE network needs a strong leadership and a clear distribution of responsibilities. Due to the absence of an institutional structure, it is easy for roles to become unclear. It is important to have a clear person or group of people in charge of setting the agenda and crying foul when there is lack of activity or a lack of coordination.
• LIAISE must rapidly secure commitment for “infrastructure items” such as the online platform and IA Bulletin, as they are not project-funded and need a continuous flow of money to run.
Table 9 recommendations for the various roles of LIAISE
IA knowledge and information hub
Optimize the Google search for the LIAISE website. If LIAISE aims to be a hub for IA knowledge and information, it is important for it to appear in the top results for searches for keywords like “impact assessment” or “IA tools”.
Continue emphasizing the community aspect of the LIAISE website. This is an added value lacking from most alternative offers, and it is more likely to help bridge the gap between policy-makers and researchers.,
Focus on enabling cooperation and synergies rather than theoretical discussions, as journals are seen as the reference for the latter. The hub should rather focus on networking and research collaborations.
Secure funding to run the online platform smoothly as it should always be running and updated regardless of the amount of project grants within the CoP.
Networking and discussion forum
It is crucial to ensure the active participation of policy-makers (European Commission and Member State level) as this is the main added value of the LIAISE forum.
The national units which advise and coordinate national IA work are a promising group of policy makers to connect to.
Considering the existence of alternative offers, LIAISE must strengthen the link to other existing communities/discussion forums (e.g. IAIA and many others)
Internally, LIAISE must strategically foster connections on two levels: experts and heads of partner institutions. Heads of institution are well connected and important to find synergies and common research opportunities. Experts are needed to have more concrete interactions in the discussion forum and help steer the research agenda.
Innovation and testing
Ensure the presence and participation of high-level or well-connected people in the network to involve whole institutions rather than just experts. This will allow for more possible overlaps for projects.
LIAISE coordinators must deepen regular contact with policy-makers and involve them in research programming. This is important not only to attract people to the network, but to ensure that the research emerging from the CoP will actually be policy-relevant.
Tool identification and quality monitoring
It is essential to work on establishing the credibility of LIAISE KIT certification. Modellers will only pay for having their tool certified if they trust it will result in more uses. Therefore users should consider the LIAISE certification as a valid seal of quality.
To bypass the problem described above, creation (if necessary for free) of a critical mass of certifications might be a good idea. It would allow modellers to become familiar with the certification and to trust it, encouraging other modellers to pay for it in the future.
An option might be to pursue endorsement by the European Commission to ensure credibility. It is an authoritative institution which will be trusted by both policy-makers and researchers.
Learning Feedback from LIAISE Schools was generally positive, but there are still some ways in which it could be improved.
Start implementing the designed curricula for policy-makers, as so far only schools for researchers (in particular early career ones) have been held.
Training coordinators should diversify the venues used for dissemination and improve the advertising strategy, as there are some issues getting a sufficient number of participants. Existing databases of training programs and PhD courses should be used.
The exchange between researchers and policy-makers should be encouraged rather than always training them separately. An idea would be to invite some policy-makers to speak at a School for researchers and vice-versa.
Again the national units which oversee and advise national IAs are an important target groups as “training the trainers” would be a very efficient tool of dissemination
Opportunities to include LIAISE topics and trainings into existing training programs should be actively pursued. There are plenty of alternative offers in the market but they do not need to be seen as competition, as LIAISE can often add a certain value to existing programs. Their scopes do not perfectly overlap.
4.1.4.5. Dissemination and Implementation Plan [including business plan]
The CoP will be designed as a Network governed by a Board and with an Executive Committee supported by a small Staff Office. Characteristics:
• The network unites partners with a research focus on IA and with expertise in the fields of the 3 dimensions of SD and/or policy sciences.
• The network is open and inviting to partners committed to developing a shared vision and research agenda by addressing key issues in research on IA for SD such as: evidence based policy making; creation of research networks and infrastructures; creation of synergies in research programming and funding across national boundaries; foresight analysis of emerging needs; strengthening science-policy interfaces, etc.
• Participation in the activities is open to external parties and individuals. Although this Charter is targeted at the research community, the partners in the CoP are strongly aware, that unilateral action in the scientific community will not be sufficient to develop shared strategic views on research on IA for SD. Contributions from practitioners and research funders are essential for a successful implementation of the principles laid out in this Charter. Therefore all stakeholders interested in the subject of IA for SD are strongly invited to actually participate in the operational activities to be organized by the CoP.
• The network actively seeks a dialogue, strategic linkages and synergies with relevant external networks and institutions in, or outside the scientific community, e.g. policy makers/users in the private sector, consultants/desk officers/other IA practitioners, the general public and other stakeholders.
• The rules of governance for the CoP will be described in a Partnership Agreement
Table 10 Roles and functions CoP and its products and services for specific stakeholders
Role and functions Products/services Users/stakeholders for 1-7
IA knowledge and information hub
• Web-platform on IA for SD
• News agent
• Meta-data repository 1. CoP/Network with networking and information services and linkages with external research networks and stakeholders.
2. Meta-data on Knowledge for IA to enable comparison, reflection and evaluation of these models and methods.
3. Procedural principles for IA for SD. An overview of procedural standards to conduct IAs.
4. Quality Guidelines for Models with special attention for policy-relevancy.
5. Guidelines / examples of methods for transdisciplinary research.
6. Shared IA Research Agenda with a focus on knowledge generation at a strategic level to enable researchers addressing policy makers’ needs.
7. Staff exchange of early career researchers. Researchers/students (1, 2, 3, 4, 5, 6, 7)
Policy makers and policy units (1, 3, 4, 6)
IA practitioners: desk officers and consultants (1, 2, 3, 4)
Horizontal units with the task to provide guidelines for IAs and monitor the quality of IAs (1, 2, 3, 4)
Research programmers and evaluators (1, 2, 3, 4, 5, 6)
Funding agencies (1, 4, 5, 6)
Private sector organisa-tions (1, 2, 3, 4, 5)
Networking and discussion forum
• Think tank
• Methodological reflection
• Interaction with policy-makers at EC and MS
Innovation and testing
• Innovation generator, Experimental Lab
• Research programming
Tool identification and quality monitoring
• Evaluating quality of IA and IA tools
• Database with meta-information
Learning
• Reflection on experiments with shared learning
• Training courses
The design of the roles, functions, products and services of the CoP (Table 10) resulted from a stakeholder analysis based on the experiences in the NoE, the many contacts with external stakeholders (institutions and individuals) in workshops, conferences and other meetings and on the discussions with the LIAISE Policy Board. For an effective and efficient performance, collaboration and synergies will be explored with external organisations in similar fields of activity. This can be networks (e.g. PEER), institutions/associations (e.g. IAIA, TIAS, FUTURE EARTH, SDSN), programmes/projects (e.g. FACCE JPI, MACSUR, APRAISE, VISIONSRD4SD) and conferences (e.g. the bi-annual Berlin Conference on the human dimension of global change; and the bi-annual conference Science for the Environment).
A business plan has been developed for the setup of the CoP and its activities. During the project the initial idea to develop LIAISE as a ‘company for IA tools and services’ was abandoned. The growing insights in the ‘gap’ and the discussions with the Policy Board, lead to the conclusion that the CoP needed an ‘activity based’, rather than a ‘company based’ business plan. The general idea is to keep the structural costs to a minimum and to cover the costs of the individual services and activities (e.g. conferences, workshops, training courses) by financial contributions from individual participants and from various sources of funding that are directly related with the specific activities. Priorities for getting the CoP going are: the establishment of a policy forum for exchange between researchers and practitioners in the field of IA for SD; the development of a shared research agenda; the provision of a web-platform providing standardized non-technical and contextualized meta-data to describe IA knowledge and tools; and the development of product and process standards to assess the quality of research on IA for SD.
List of Websites:
website: http://www.liaise-kit.eu
Contact details can be found on the website
4.1.1. Executive Summary
An evidence based approach of policy development and societal decision making is becoming increasingly important, both in the EU and in Member States. The process of integrated impact assessment (IA) potentially provides a framework to compare policy options, it is applicable to all (policy) sector and able to identify potential trade-offs between different impact areas, regional and temporal scales and thereby maximizing the benefits while minimizing unwanted side effects.
Considerable funding in the framework of European research programs has been invested to support IA by means of developing models, methods, datasets, toolboxes and by studying the process of IA. However, both research and the experience of practitioners (principally desk officers) demonstrates that the results of these research projects are not being utilized to its full potential. A better uptake could be expected if the understanding of researcher of policy needs would be improved, the capabilities of tools would be adapted accordingly and awareness of IA practitioners on the potentials of tools developed in research project would be increased.
Additional gaps exists between the different scientific disciplines: Research efforts are fragmented between those that study the functioning of assessment procedures and those that design and update assessment tools. Some research projects are primarily motivated to directly inform IA for SD, but many are primarily motivated by scientific merits.
The Network of Excellence (NoE) LIAISE was funded to explore these gaps, to develop solutions and to give these solutions a structural permanence by developing an organisational setup and a business plan for a future centre of IA excellence that can continue beyond the funding for the NoE.
To achieve this goal, LIAISE activities entailed among others:
• Research on the practice of IA and the use of evidence in policy making throughout European Member States, the OECD and in developing countries
• Review of research projects and their categorization of their contributions in the context of IA for SD
• Development, improvement and testing of tools which take into account the requirements of IA for SD
• Development of a toolbox with IA relevant knowledge as a Science-Policy Interface and including standards to describe knowledge for decision making
• Organising dialogues between IA practitioners and different disciplines
• Development of a network of researchers, research organisations and IA practitioners which are committed to develop jointly knowledge for decision making
• Development and testing of training formats for IA practitioners and researchers
The major achievements of LIAISE are:
• A better understanding and collaboration between researchers from the ‘modelling’ disciplines and researchers from the policy sciences and social sciences. This resulted in a better understanding of the concept and roles of IA methods and tools in the policy process. Research needs to be contextualized in order to become relevant for decision making.
• Tools for ensuring such contextualisation, including techniques of Knowledge Brokerage, the LIAISE KIT, the Shared Research Agenda and a strategic dialogue with IA practitioners.
• The lessons learned provided building material for shaping the future of LIAISE as a Community of Practice on IA research for SD and are reflected in:
• A Charter on IA research for SD: a set of guiding principles for delivering knowledge that is relevant, credible and legitimate.
• A vision for a Community of Practice on IA research for SD and its most important roles and functions: IA knowledge and information hub, Networking and discussion forum, Innovation and testing, Tool identification and quality monitoring, Learning.
• An implementation plan and a business plan to get the CoP going. These plans have been endorsed by all 19 parties: all 15 partners of the NoE consortium and Ecologic, IEEP, IOM, and IZA. Additional partners are expected to join in the future.
Project Context and Objectives:
4.1.2 Project context and objectives
4.1.2.1 The role and position of impact assessment in European policy making
An evidence based approach is becoming increasingly important in policy development and societal decision making. The process of impact assessment (IA) is a central tool in regulatory policies and provides a framework for comparing policy options, while being adaptable to all (policy) sectors. Evidence on the likely impacts of policies enables policymakers to maximize benefits to society and minimize unwanted side-effects. Therefore the analysis should cover the impacts in the targeted policy domain, as well as unintended impacts, side effects and trade-offs in adjacent policy domains. IA systems and procedures have been implemented in the EC and all OECD countries and are spreading in other parts of the world.
Policy making in the European Union follows a complex system of multilevel governance involving interaction between the European policy system and the Member States. The overarching goals for policies at the European level are laid down in comprehensive strategy documents such as the Sustainable Development Strategy, the ‘Lisbon Strategy’ for Growth and Employment and more recently Europe 2020. These strategies recognise that policies aiming at the overarching goal of Sustainable Development (SD) should follow a three-pillar approach targeted at economic, social and environmental objectives. These three pillars of SD are recognised as equally crucial, interconnected, and urgent. All EU institutions (including the Commission, but also the Council and the Parliament) have committed themselves to base major policy proposals on a balanced assessment of the social, environmental and economic dimensions which takes into account the external dimensions of SD.
The instrument of ex-ante IA as developed by the EC aims at collecting evidence on the likely impacts of policies and thereby maximize the benefits to society and minimize unwanted side-effects. A first guideline was published by the Commission in 2003, and updated in 2005 and 2009 after a process of public consultation . It aimed to replace and integrate all sectoral assessments of potential impacts of proposed measures into one global instrument with a balanced consideration of development targets against the three dimensions of SD. The guidelines describe the organisational structure of the process and a basic process: 1) identification and analysis of the policy problem; 2) definition of the policy objectives; 3) development of alternative policy options and a baseline scenario to describe the development without policy intervention: 4) actual analysis of the impacts of the policy options with respect to the three dimensions of SD; 5) comparison of the analysed impacts of the various policy options against the baseline scenario; and 6) outlining a set of recommendations for indicators, monitoring procedures and the ex-post evaluation of the implemented policy.
Since 2003, the role and importance of IA and evidence based policy making has steadily increased within the European institutions. This is indicated inter alia by:
- the setup of an Impact Assessment Board of High level officers, reviewing the quality of IA
- the dedication of JRC as main supporting Directorate to support IA
- the setup of support units in all DGs and in the SecGen
- training of desk officers
- the setup of an Directorate for IA in the European Parliament
- awarding of numerous studies and framework contracts to support IA
- evaluations of the performance of the EC IA system by the Secretariat General and the European Court of auditor
These developments have been echoed in the Member States and beyond: Although IA requirements and practices vary considerably, the LIAISE surveys identified relevant activities and capacity building throughout Europe.
4.1.2.2 Perceived gaps between IA tool availability and actual use in the policy process
European research policies since Framework Program 6 have invested in several large scale projects to develop models, methods and data in support of Impact Assessment. A focus area was in topics related to research for Sustainable Development. Among these projects were MATISSE, SENSOR, SEAMLESS, IQ TOOLS, EVIA and others. Despite of these investments in developing new models and methods in support of IA, both research and the experience of practitioners (principally desk officers) demonstrates that in practice the broad range of potentially relevant tools to support IA is still not being used to its full potential. Compliance testing work on the reality of policy making revealed a gap between the aims of IA and its implementation. The application of IA tools is difficult and often at odds with the process of decision making . The political process is determining to a large degree the applicability of IA tools, not only the features of scientific knowledge. Relevancy of models and methods is determined by the context of use. Therefore, it is important for researchers to understand the context of use. The call text of topic ENV.2009.4.2.1.1 (Network of excellence for IA tools) and the documents provided by the Commission specified the main features of the gaps between scientific IA tools and the actual use in the policy process:
Policy orientation vs. research orientation. Policy-makers tend to look for tools with a proven record of effective use in policy. Policy relevance however is not yet a significant part of the researchers incentive system. From their point of view the development of new conceptual approaches and ever more complex tools is more rewarding than a focus on the usability of existing tools.
Complexity vs. transparency. The tools used must be based - as far as possible - on rigorous analysis while recognising explicitly where value judgements are included. However, many scientific models remain ‘black boxes’. Differences in data sources used or simple misunderstandings about the terminology used may undermine trust.
Maintaining existing investments vs. preparing for the future: promising or successful tools from a policy perspective are not always maintained or further applied by the developers. Due to sometimes limited interactions with policy-makers, researchers do not properly recognise the loss of opportunity in not further maintaining or applying existing tools.
Accessibility vs. applicability: the access to existing assessment tools is often poor. One of the reasons is the lack of structure to link the diverse and ever-changing needs of policy makers with the overwhelming abundance of existing tools on the supply side. Another important reason is the limited access to the data that are needed to apply the tool.
To conclude: high expectations are not being met with respect to the use of IA and the linking of policy assessment tools and procedures to pursue SD in the EU. The improved use of tools for SD requires an in depth understanding of policy needs, tool capabilities and the associated opportunities for (and obstacles to) linking them in the everyday policy process. Current research efforts are fragmented between those that study the functioning of assessment procedures and those that design and update assessment tools. Some of these research efforts seek to directly inform and in turn improve IA for SD, but very many do not.
The NoE LIAISE was funded to explore these gaps both between IA practice and research as well as between different disciplines. Expected impacts as formulated in the call: Increased integration of the IA research community, improved efficiency of IA tools through their integrated and mutually complementary development, structured dialogue between the research community and policy makers about IA tools development, and enhanced use of IA tools in policy processes at EU and Member State level.
4.1.2.3 LIAISE: mission, vision and strategy as formulated in the DOW
The DoW described the mission, vision and strategy of the NoE as follows. The challenge for LIAISE was to bring the above mentioned elements together in a more systematic fashion by: 1) showing decision-makers at different levels of governance and in different jurisdictions that different assessment procedures consist of similar phases and can draw on similar types of tools; 2) stimulating researchers to streamline the rich diversity of tools and communicate the potentials and limitations of individual tools; 3) further developing that framework to enable tool selection and use in close collaboration with IA users and IA researchers; and 4) fostering a deeper understanding of the role of IA (and IA tools) in the different phases of the policy process. The mission (the overarching objective of LIAISE) was to contribute to the renewed EU Sustainable Development Strategy (SDS) by bridging the gaps between science, policy making and implementation, with academically-grounded approaches to IA. The vision for LIAISE was to establish a ‘virtual centre of excellence’ with a leading position in the international IA research community and close institutional linkages with users at European and Member State levels. Together these characteristics should provide the capacity to contribute to the desired impacts as formulated by the Commission.
The vision was translated into the following strategic goals:
• Creating the hub of the NoE, comprising a core group of institutes, whose task will be to initiate and further develop the virtual centre of excellence mentioned in the vision.
• Using the NoE as a platform to initiate a more structured dialogue between the IA research
• community and IA users. This should result in: a) Communication structures, incentives and mechanisms that stimulate external institutions to join LIAISE as associate partners and thus contribute to the development of the virtual centre of excellence; b) A shared IA toolbox reaching out to policy makers and those involved in conducting IAs; and c) A shared research agenda on strategic IA tools and processes and the development of new IA tools;
• Safeguarding the achievements of 1 and 2 for the post NoE period.
Therefore, the key characteristics of LIAISE will be:
• A bridge between the IA user community and the IA research community. This requires a deeper understanding of the role and limitations of IA in policies for SD, the stakeholders involved and the possibilities for an improved use of existing tools by the various categories of stakeholders.
• A means (the shared IA toolbox) to handle the interplay between the socio-economic and
• environmental systems in combination with multi-scale problems in ways that help policy-makers implement IA to contribute to the key objectives of the EU SDS.
• A way of handling the ongoing development of new tools. The ongoing development and
• refinement of the EU SDS will throw up new strategic questions. Progress in the fields of science and technology will also lead to new opportunities for developing IA tools that answer emerging policy needs.
• A close, two way collaboration, interaction and information exchange between and within the IA research community and IA users. This will be absolutely crucial for the success of theproposed NoE.
4.1.2.4 Strategic changes during the execution of the project
During the course of the project three major developments strongly affected its strategic scope and outcomes of the network: 1) the position and role of the Policy Advisory Board; 2) changes in the EC strategies regarding SD; and 3) the changing views in the science-policy-interface on the concept of IA methods and tools. The re-organisation of DG RTD and the frequent changes of project officers (6 in 54 months) added additional complexity to the question how to adequately address these developments.
The call text explicitly requested the installation of a Policy Advisory Board (PAB): The NoE will also establish the appropriate links with policy makers to ensure the use of the shared IA toolbox. In this context, the consortium should also establish a policy-advisory board. In the DoW, this was addressed by installing a PAB that combined the interaction with the IA users as-well-as the scientific supervision of the project. The combination should prevent the risk (as observed in other FP6/7 projects) that working with separate advisory bodies (a policy advisory board and a scientific advisory board) might introduce a science-policy gap in the organisational design of the project as such. The PAB was given a central role and position in the conceptual approach for the long-term integration in the NoE . To enable the PAB to fulfil its supervisory role, an on-going evaluation by an external party (Ecologic) was included in the project design.
After the project was granted to the LIAISE consortium, DG RTD pro-actively engaged in the formation of the PAB with EC representatives from the various Policy DGs and SecGen. The first PAB meeting showed that there were no shared views between DG RTD, SecGen and the policy DGs, despite DG RTD efforts to internally fine-tune the preparation of the call text and the PAB formation. The envisioned PAB members refused to accept any 'advisory' responsibilities for the project and changed the name to Policy Board (PB). Individual members are not representing the views of their organisation, but use their personal capacities, knowledge and experience to critically reflect on the project. The status of the on-going external evaluation by Ecologic changed from a monitoring instrument at the disposal of the PAB to an activity in support of the scientific Management Board.
Subsequently, the members of the policy board were supportive in providing recommendations for the research conducted within the network and provided access to other policy officers e.g. in the context of the test cases. It was emphasised, however, on several occasions that LIAISE would not be considered as any kind of support service for IA complementing or replacing existing structures within the European Commission. LIAISE is considered to play a role in research and research programming, if its members are willing to take part in IA, they are encouraged to apply for tender.
The call text ENV.2009.4.2.1.1 (Network of excellence for Impact Assessment Tools) directly relates the activities of the NoE to the Renewed EU SDS. Indeed, the renewed EU Sustainable Development Strategy (SDS) reaffirms the need to support policy-making by scientifically sound approaches for IA. With the publication of the EU2020 Strategy the prominent position of the SDS as a driver for SD in the EU, was replaced with the EU2020 Flagship Initiatives to support smart, sustainable and inclusive growth. These changed also were reflected in organisational changes at DG RTD. The DG RTD unit that formulated the call and strongly promoted the NOE initiative, ceased to exist. The key persons for LIAISE were no longer connected to the project; the head of unit moved to another DG and the project officer left the Commission. An indifferent attitude towards the project became apparent from EC stakeholders. This could be observed in the meetings of the PB as well as in interactions with Policy DGs and JRC. In the PB meeting of 26 November 2012 the future of LIAISE was discussed. The response by EC representatives as described in the minutes is: The objective of LIAISE NOE is to establish a long lasting Network of Excellence that coordinates research and the envisaged interaction between policy and science in the field of impact assessment. From the responses of the PB to the options for a post LIAISE entity, a clear role for LIAISE cannot be distilled to achieve this objective. The Policy DGs tend to prefer LIAISE competences as part of regular consultancy contracts, JRC indicate that the Commission has sufficient in-house capacities for training and tendering procedures for consultancy. The existing procedures of DG RTD to coordinate research programming with the various DGs are considered as sufficient. Therefore, at this moment reviving existing scientific associations is the best suggested option for a future LIAISE entity. These existing scientific associations are not institute or consortium based, but consist of individual research members. For this no future LIAISE entity is required, LIAISE researchers can sign up individually to these associations and have an impact on their actions.
The statements made in the Policy Board also demonstrate that different actors have developed own support units and institutions for impact assessments during the lifetime of LIAISE and which perceived LIAISE as a potential competitor, respectively questioning the added value of LIAISE in this emerging landscape of actors related to IA.
In their report on period 2 the coordinators asked the Commission’s attention for these alarming developments. ‘In the perception of the coordinators this decline of genuine EC interest in and involvement with the long-term objectives of the NoE, is a serious threat for a successful and durable centre of excellence. The 'business concept for the shared toolbox' which was a key element in the assignment by the EC to LIAISE, can only be maintained in a situation where the EC is prepared to act like a future customer for the shared toolbox and LIAISE expertise. As coordinators we must discuss the EC involvement with DG RTD to agree on a shared strategy for achieving EC commitment in period 3 and the appropriate level of ambition.’ This message was actively taken up by the newly appointed project officer who organised a review by an external expert and a workshop with the LIAISE coordinators and other on-going projects in the field of SD in order to strengthen the coherence and synergies in April 2013. The review suggested to focus LIAISE more on sustainable development in the broader sense and relatively less on impact assessment in the European Union context, and urged LIAISE to think more strategically on the relevance of evidence production in the wider sense of achieving sustainable development. LIAISE achievements in networking, contextualizing tools and models, scientific progress, and method development for test cases were acknowledged, while at the same time stimulating LIAISE to think of a broader group of audiences for its services and knowledge.
Before outlining the follow-up of the review and workshop, it is necessary to briefly describe the third major development that strongly influenced the strategic scope of the project: the changing views in the science-policy-interface on the concept of IA methods and tools. Among the IA researchers the view predominated that IA methods and tools are developed by researchers in the scientific domain, then disseminated into the application domain where they are actually used by IA practitioners and policy-makers. The members of the PB pointed at the fact that this separation between the two domains is artificial, unproductive and not reflecting real life situations where IA methods (and scientific evidence in general) are successfully used in policy processes. From their perspective the scientific content of a method or tool cannot and should not be separated from its policy context. This PB intervention has proven extremely relevant and valuable for the scientific achievements of LIAISE. A major part of the research has been targeted at sharpening the views on the roles and responsibilities of researchers and policy-makers in policy processes, on the process of Knowledge Brokerage and on the question how to support these activities with ‘contextualized knowledge and information’. This not only had significant consequences for the tool development in LIAISE, but also for the business and quality concepts underlying a future centre of excellence. The ambition to deliver contextualized knowledge strongly limits the opportunities to develop and sell standardized IA tools as a ‘product’. Instead the focus turned to activities and processes related to the development of scientific knowledge. Research needs to be sensitive to the context of use and therefore systematically analyse the contexts of potential application and to develop and to describe scientific knowledge accordingly while at the same time meeting scientific criteria. In follow up discussions with the project officer in the summer 2013, it was agreed to focus efforts on a synthesis of main lessons learned of LIAISE for evidence production, to expand the exploration of users to member states, international organizations, and potentially business, and to expand the policy board with participants familiar with impact assessment of research investments.
Based on these experiences and considerations, a strategic focusing of LIAISE has been initiated by the consortium in close collaboration with the project officer. Major steps of this focusing were:
• The annual meeting of the consortium in Tallinn: It was agreed to explore a focusing of LIAISE on IA for Sustainable Development and to position LIAISE strategically addressing the research needs and subsequent research organisation on transdisciplinarity, long-term issues and integrated analysis along all dimensions of sustainable development.
• Selection of mini-research projects as part of an open call for remaining flexible budget focusing on Impact Assessment in international organizations, Research Impact Assessment, Impact Assessment in Emerging and Developing countries and durable concepts for networks of excellence of researchers.
• The conduct of an extensive stakeholder analysis exploring the potential services derived from a LIAISE NoE and the relevant stakeholders and potential customers for these services.
• The focusing of the LIAISE activities on mid- and long term programming of research
• The development of a charter which expresses the commitment of LIAISE researchers in developing research which is applicable for meeting societal needs in the context of sustainable development
• The development of a business plan which defines core products and services for maintaining LIAISE as a network with the specific focus in the future and which coordinates the partner contributions.
The focusing of LIAISE suggests a different positioning. Instead of developing a research institute or an organisation which provides consultancy, training or other services, a network conceived as a community of practice was selected as most appropriate. A community of practice is characterised by its diversity of members in terms of disciplines and professional background but with a shared ambition of solving similar problems.
The focusing of LIAISE as a community of practice, its commitment to a charter and the proposed business plan was well received by the stakeholders, the members of the policy board, and the members of LIAISE. The partner organisations, including the partner that were associated during the funding period expressed their willingness to provide substantial contributions to such a future LIAISE community of practice.
Project Results:
4.1.3 Main S&T results/foregrounds
4.1.3.1 General approach of the LIAISE NoE
The LIAISE consortium unites the multi-disciplinary competences of a core of 15 European institutes from 9 countries that in turn consolidate the expertise from large FP6 projects on IA tool development such as SEAMLESS, SENSOR, MATISSE, Sustainability A Test, IQ Tools and EVIA. It includes expertise from the field of environmental sciences, economics and political sciences. This makes it possible to analyse current policy needs, to link them in innovative ways to the available reservoir of IA knowledge and to test these innovative solutions in targeted, co-designed and co-produced IA test cases. Three additional elements in the project setup complete the general approach:
• The formation of a Policy Board with policy makers and IA practitioners from the EC (policy DGs, DG Research and Innovation, SecGen), Member States, the OECD and NGOs. The Policy Board provides a platform for the interaction at the strategic level between IA researchers and IA users;
• The formation of focus groups with IA researchers and IA practitioners to discuss operational issues and user experiences;
• The on-going evaluation by Ecologic (www.ecologic.eu) a think tank for applied environmental research, policy analysis and consultancy. It was commissioned by LIAISE to assess the progress towards the general project objectives. This evaluation is based on the impact chain approach and is complementary to the standard FP7 project evaluation procedures by the Commission which focus on monitoring the project progress in terms of the realisation of the project objectives. The impact chain approach gives insights in the progress towards the envisaged impacts of the project (the ambitions beyond the realisation of the project objectives, e.g. the contribution to evidence based policy development).
4.1.3.1.1 Project structure: Work Packages and deliverables
Table 1 Work Packages and deliverables
WP0: Management and coordination of LIAISE
Tasks: 1) Overall coordination of LIAISE; 2) Coordination, planning and monitoring system; 3) Set Up of a Policy Advisory Board; 4) In Built Evaluation; 5) Knowledge management activities and associate partners
WP0 Deliverables
1. Policy Advisory Board
2. Gender Action Plan
3. Letter of institutional commitment
4. Operational planning and monitoring system with guidelines 5. Quality Assurance plan
6. Activity and financial reports for every 18 months
7. An updated financial and implementation plan
8. Report on Awareness and Wider Societal Implications
9. Final report
WP1: IA for sustainability: policy needs, assessment procedures and governance contexts
Tasks: 1) Coordination of WP1; 2) Synthesize and meta-analyse existing research on the procedures and everyday practices of IA in the EU and the Member States; 3) Explore the relationship between IA and wider systems of governance for SD development in the EU and its Member States
WP1 Deliverables
1. A meta database of different cases of IA in action, drawing on results from inside and outside LIAISE including previous FP6 investments.
2. Review articles in relevant academic journals and practitioner outlets
3. Summary of user needs and expectations regarding IA processes, IA tools, IA training and WP6 test cases.
4. Review of links between IA and wider systems of governance for SD in a representative sample of juris-dictions and/or policy areas, via an edited book or similar.
WP2: Science for IA Tools and Procedures
Tasks: 1) Coordination of WP2 and framework development; 2) Synthesising emerging knowledge to better analyse policy impacts; 3) Research for IA Tools; translation of knowledge gaps into research questions; 4) Roadmap for continuous interaction between IA community and other research communities; 5) Knowledge use in different assessment venues; 6) Development of IA modules for design of research
WP2 Deliverables
1. Methodological framework for WP2 activities
2. Overview of research groups/networks producing knowledge of relevancy for IA tools and processes
3. Synthesis of research needs for IA tools in research programmes inside and beyond the IA research community
4. Procedural concept to facilitate a continuous uptake of emerging scientific and social scientific knowledge in IA tool and process improvement, beyond the lifetime of the NoE
5. A literature review on the politics and policy of evidence-based policy-making in different assessment venues, with particular reference to assessment tools
6. Modules for research designers to make research programmes and projects compatible with IA knowledge needs
WP3: Shared toolbox: back-office and improvement of selected IA tools
Tasks: 1) Coordination of WP3; 2) Assuring quality by creating and using an IA Tool Reference Model (RM-IAT); 3) Requirements analysis, creation and improvement of the back office tools; 4) Development of the back office IA toolbox;
WP3 Deliverables
1. Reference Model for IA Tools (RM-IAT) including glossary, software quality assurance (SQA) plan, standards for documentation and user requirements engineering and standards for architectural design and distributed processing
2. Analysis of requirements of tools in terms of: applica-tion aim; application domain and scale(s); flexibility;. harmonization and standardization;, usability; user scope; transparency; documentation 3. Conceptual and architectural design of the toolbox back office
4. Improved tools (software delivery and deployment) including documentation and manual
5. Implementation of shared toolbox back office (software delivery and deployment) including documentation and manual
6. New tools (software delivery and deployment) including documentation and manual
WP4: Shared toolbox: front office
Tasks: 1) Coordination of WP4; Conceptualising and designing the toolbox front office; 3) Development of the toolbox front office; 4) Implementation of the toolbox front office; 5) Maintaining the toolbox
WP4 Deliverables
1. Design options for the toolbox – paper to be presented to the Policy Advisory Board
2. Concept for the toolbox, description of (sub)catego-ries, quality criteria and outline for meta-description 3. Design options for the help desk
4. Populated toolbox for the front office with inventories models/tools/actors/good practice/data sources
5. Help desk
WP5: Shared toolbox: governance and post-project durability
Tasks: 1) Coordination of WP5; 2) Establishment of product standards for participation in the shared toolbox; 3) An operating and performance review; 4) Preparation of a Route Map to a Self-governing Entity; 5) Development and implementation of a Business Plan.
WP5 Deliverables
1. Product Standards
2. Submission of a Route Map to a Self-governing Entity
3. Report on the operating and performance review of the Shared toolbox and associated services
4. Business plan with implementation timelines
5. Report on implementation of business plan
WP6: Test cases in sustainability priority areas
Tasks: 1) Coordination of WP6; 2) Test cases: the development of options; 3) Modules for IA support; 4) Case Studies; 5) Analyse the test results with regard to LIAISE activities
WP6 Deliverables
1. Options for test cases
2. Modules for IA support
3. Results from test cases
4. Material of each closed test case for individual WP7 policy briefs
5. Final material on the test cases for WP7 policy-briefs
6. Compendium of test case studies. Form could be an edited book or an open access journal special issue
WP7: Dissemination and Training
Tasks: 1) Coordination of WP7; 2) Web site and materials; 3) Scientific publications and Policy briefs; 4) IA Training activities; 5) Events.
WP7 Deliverables
1. Dissemination Plan
2. Biannual Innovation report on IA
3. Periodical news bulletin
4. Open Access platform
5. Report on the continuous updates of the website
6. LIAISE Curricula for researchers 7. LIAISE Curricula for policy-makers
8. Concept for IA Book series and IA Journal
9. Launch of LIAISE papers series
10. LIAISE Brief Series
11. High level Conference
12. Final workshop
4.1.3.1.2 Stakeholder Analysis
The LIAISE ambition to better utilize the available potential of the research knowledge reservoir in evidence based policy-making requires a better understanding of needs, preferences, working procedures and options for action in both policy making as well as in research. LIAISE conducted a stakeholder analysis to identify relevant actors and their needs with regard to the relevancy of research. The available data collected in other LIAISE activities, provided a useful source of information for a secondary analysis from a stakeholder perspective. Hence, the data were not specifically collected for a stakeholder analysis, but for other purposes: an analysis of IA practices in EU Member States, the development of a research agenda, the testing of LIAISE services, etc. The analysis is based on data from the following sources:
• About 40 LIAISE workshops: test case workshops, agenda workshops, etc.
• Literature analysis: country studies on the design and implementation of IA processes within LIAISE and LIAISE related projects
• Individual contacts and interviews as part of surveys among 170 policy makers in 17 European Member States and more than 200 researchers
• About 30 special sessions at international conferences
• Mapping of about 200 research projects
• Evaluation of model use in EU impact assessments
The empirical basis and its interpretation for the stakeholder analysis have been validated by the work package leaders who were responsible for these activities. The stakeholder analysis found that there are relatively few stakeholders who are concerned with the meta-level of policy impact assessment, for example by issuing guidelines, researching or steering the implementation of IA. Most stakeholders rather consider IA as a means to achieve a goal in their respective thematic policy domain or research field. A wide group of actors is potentially interested in improving the information exchange in the science-policy interface. Their specific interests not only depend on the type of stakeholder, but also on the use context and the level (strategic or operational) at which a certain decision has to be made. We identified four general practices of IA or contexts of use in which LIAISE provides an added value to improving the situation:
• The development of requirements and guidelines for IA for SD (strategic level)
• The design of an IA process (operational level)
• The decision on using models in an IA process (operational level)
• The funding of research and development of models (strategic level)
Although most stakeholders have a special perspective on IA coming from their fields of disciplinary specialization or policy domain, LIAISE was able to identify general user groups in the different practices of IA, who have similar strategic interests and needs. These actors include horizontal and policy units in governments, international organisations, research funding agencies, stakeholders from civil society and business, consultants, but also individual researchers and politicians. To identify actions to better bridge between research and policy making, LIAISE studied the different practices of IA with regard to the occurring (general) stakeholder constellations, their interests, and the possible LIAISE contributions to improving the science-policy interface in each context. The results are summarized in the following Table 8 as part of Section 4.1.3.4. on Potential Impacts.
4.1.3.2 Main achievements from a research perspective
4.1.3.2.1 The Practices of IA in Europe and beyond (WP1)
WP1’s research on the practices of Impact Assessment (IA) in the countries of the EU was based on the assumption that IA should be treated as one venue among a number in which there is potential for evidence and tool use to inform policy formulation. It was also premised on the understanding that many different kinds of learning are possible through IA. Apart from ‘instrumental’ learning, in which objective information is fed to policy makers, who use it to make better decisions in a rational and straightforward way, ‘conceptual’ or ‘double loop’ learning, which challenges underlying assumptions and changes values and norms perhaps gradually, is possible. WP1 therefore championed the need to understand user needs in the broadest sense, i.e. not simply as a list of how existing and new tools could be developed in an instrumental manner, but considering the richer context and role of IA and IA tools and the wider needs of users. This thinking also led to the conclusion that Test Cases were much more than about testing tools (or instrumental learning), but also about attempting to understand the way practitioners and researchers interact to develop new questions and problem framings (‘conceptual’ or ‘double loop’ learning).
WP1 confirmed that by 2008, all 31 OECD countries had either adopted, or were in the process of adopting, a formal system of policy appraisal (OECD 2009). Building on this, D1.3 - A summary of user needs and expectations regarding IA processes, IA tools, IA training and test cases - reported on the most comprehensive survey yet conducted of user needs and expectations with regard to IA systems and tools in 17 European countries. Previous surveys of this kind had been either narrower in focus or conducted in less depth. None focused (as WP1 did) on the experience and insights of those people at national level who determine the strategic direction of IA, e.g. writing guidance documents, controlling quality and offering technical support for IA.
WP1 collected the following types of information:
- On the origins and purposes of IA: When was IA introduced and why? e.g. to reduce administrative burdens, improve the quality of legislation, improve communication or to integrate sustainable development. What is its purpose considered to be in practice?
- On the design of IA: including, is it mandatory or voluntary; how are policies selected for IA and what are the criteria used; at what stage of the policy making process is IA carried out; how is quality controlled and what impacts are included?
Thus, information was gathered not only on how IA was supposed to be conducted according to guidance and other official documents, but on how IAs are conducted in practice: how did implementation differ from the guidance; how much resource devoted; what impacts were considered in practice, and what quality was achieved? Ideas were also sought on how IA could be improved and what barriers it faces in each country.
The survey found a wide variety in the IA systems studied. The systems themselves, their underlying purposes and the tools they use vary both within and between the 17 countries. Many different factors affect the way they are structured and their functioning. These include the availability of resources (skills, time and data with which to conduct an IA) as well as the steps that have been taken to establish quality control mechanisms and institutions. In terms of underlying purpose, sustainable development was perceived as one relatively minor aim of IA. Reducing the costs of regulation appears to be a dominant motivation in most jurisdictions.
Following the mid-term review, an additional analysis examined the types of tools used in a sample of 325 published appraisals from 8 jurisdictions (this time including the Commission), using a detailed framework which includes a novel, 7-fold classification of tool types (see table 2). This created a systematic picture of precisely which tools are actually used in different systems. Again, this significantly extends the existing literature which focuses on a limited number of cases and jurisdictions.
Table 2 Percentage of cases using certain types of tools, over the periods examined. Note: PA = Physical Assessment tools; MA = Monetary Assessment tools; Model = Modelling tools; Other = Scenario, Multi-Criteria Analysis, Stakeholder analysis and other tools
Jurisdiction Number of appraisals
(period covered) Average length of report
(pages) Simple
(% of cases) PA
(% of cases) MA
(% of cases) Model
(% of cases) Other
(% of cases) No tools
(% of cases)
Cyprus 20 (2009-11) 14 0 0 0 0 0 100
Denmark 50 (2006-11) 2.5 68 4 56 12 2 28
European Commission 50 (2008-11) 84 96 4 44 18 8 4
Finland 50 (2009) 2.5 16 0 18 2 4 66
Greece 36 (2010-11) 17 19 0 14 0 0 78
Ireland 49 (2004-10) 13 33 0 45 6 0 39
Poland 20 (2008-10) 7 60 0 40 0 5 30
UK 50 (2007-10) 38 60 0 92 16 10 4
TOTAL 325
These jurisdictions were chosen for several reasons: they represent a spread of well-studied and less well-studied places; all have reasonably accessible appraisal processes and other government documents that could be studied empirically; and they represent both ‘early’ and ‘late’ adopters of appraisal systems (Adelle et al. 2012) .
Some (e.g. Cyprus, Finland, Greece) appear to use hardly any tools; in the cases of Greece and Finland, more than half the appraisals sampled reported no tools. In Finland, use of a standard checklist is reported to some extent, and other methods are occasionally reported, e.g. partial CBA, but on average the reports are extremely brief (less than three pages). In Cyprus, while an appraisal procedure is present, and a standard form has to be filled in which increases the appraisal's length, no specific tools were reported in any of the cases analysed. Other jurisdictions (e.g. Ireland, Poland, and Denmark) still show a large minority of cases using no tools, but there is more evidence of use of some simple and monetary assessment tools, in around half of the cases sampled. Table 2 shows that only 6% of the Irish cases reported use of modelling tools (all related to building regulations). In Denmark, the reports were very brief and mainly revealed use of monetary assessment and simple tools, with some quantification. A few cases (12%) mentioned modelling tools and two – relating mainly to environment and tax legislation - mentioned physical assessment. But while a wider range of tools was reported than in some other jurisdictions, a relatively large proportion (28%) of cases still reported no tools used. Poland exhibited a similar pattern to Denmark, but while both countries showed mainly use of monetary assessment and simple tools with some quantification, Poland exhibited a relatively lower prevalence of monetary assessment tools. Again, in these jurisdictions, only a few reports mentioned modelling tools and/or physical assessment, and those that did related mainly to environment and tax policy.
Conversely, the Commission and the UK have much richer reported patterns of tool use: only a handful of cases use no tools. In the Commission, almost all of the cases reported use of simple tools, and just under half reported monetary assessment. There are also more cases of modelling tools being reported (about one in five cases) than in any other jurisdiction. The average length of the appraisal reports was also more than double that of any other jurisdiction. The UK, by contrast, showed greater use particularly of monetary assessment, which is mandatory in the appraisal guidelines. A few cases (16%) mentioned modelling tools; these were mainly economic models in the fields of housing, transport and pensions policy. Some such appraisals are very long (more than 200 pages) and contain much detailed analysis, but the majority are rather brief. Few reports mentioned specific tools for participatory analysis; there was often just a consultation with no additional methods employed for synthesising.
Although many countries have sought to learn from one another and international bodies such as the OECD, there is still no one dominant approach to undertaking IA that is firmly institutionalised in all countries. Rather, each country employs IA in a distinctive way which fits its prevailing political and policy context. It is important therefore not to ‘de-contextualise’ IA, especially when seeking to define and extend ‘best practices’ or increase the use of IA tools, such as cost benefit analysis, scenarios or formal computer-based models. Many – but by no means all – of the 17 IA systems surveyed already harness the analytical power of IA tools to inform their assessment activities. In fact, only 9 of the 17 actively promote their use via the production of guidance to the officials undertaking IAs. On the whole, the use of IA tools in practice is highly differentiated both between the main tool types (simpler tools tend to be more popular than more sophisticated ones) and amongst individual IA systems (tool use is generally higher amongst the older Member States than the newer ones). To summarise, the widespread institutionalisation of IA has not yet led to a concomitant institutionalisation of IA tool use. Indeed, many of the countries studied still appear unconvinced of the basic need to increase tool use across the board.
In general therefore, user needs with respect to IA tools defy simple generalisations. They tend to be specific to particular tools and/or IA systems. Instead of ‘saturating’ users with information on tools, this pattern of use calls for a more targeted and ‘smarter’ deployment of existing as well as improved tools; one which is sensitive to the prevailing context in each country. For the least enthusiastic adopters, it may be ‘smarter’ to focus on making the case for IA tools, whereas more enthusiastic adopters seem to want more detailed information on specific (types of) tools. Test cases were seen to constitute a potentially important method to understand these contextual conditions (and thus couple supply to demand), a task that was addressed in WP 6.
Latterly, the project on Impact Assessment in Emerging and Developing countries (D0.8.3) extended the survey work on IA practices beyond Europe and suggests that there are vast opportunities for improving IA practice and tool use in developing countries.
The achievements of WP1 can be summarised as follows:
• Providing a reality check - massive investments have been made in diffusing IA globally, but big differences exist in national practices.
• Supporting knowledge brokerage, documenting and understanding differences in IA practices as a precondition for altering the status quo.
• Building the foundations of the test case work – ‘smarter’ deployment of assessment tools can reveal the potential of IA (but there are also ‘costs’ – time, resources etc. – to take into account)
• Adapting to changing political priorities – since 2008, new demands have arisen (sustainability; carbon; ecosystems etc). WP1 has responded to these changes in stakeholder needs by working with many different partners (OECD, World Bank, UK DEFRA)
• Developing new research agendas – thinking afresh about the uses that many different assessment tools (inc. but not limited to computer models) are put in / alongside ‘non IA’ venues.
4.1.3.2.2 Knowledge for Decision Making: the LIAISE KIT as community platform and library of contextualized knowledge
The need to contextualize knowledge for specific IAs as well as the need for bridging between different research communities and IA practitioners was guiding for the development of the LIAISE KIT (www.liaise-kit.eu)as a toolbox and science policy interface which provides access to relevant knowledge. The platform
- Is based on extensive review of the user requirements and the practice of IA,
- Builds on and integrates knowledge from previous EU funded projects,
- Has been developed in close interaction with users and a systematic and extensive collection of feedback on various test versions.
- Represents a model of the IA process and the substantive requirements of integrated IA which enables users to describe their research and expertise in the context of IA,
- Provides user roles and rights for self-maintenance with a minimum of centralized requirements for administration.
It has been developed as a web based knowledge and community platform which
1. Provides access to scientific knowledge in the context of Impact Assessment for Sustainable Development and
2. Provides a platform to exchange news and for collaboration on issues related to Impact Assessment
With these functionalities, the LIAISE KIT provides a Science Policy Interface (SPI) that makes scientific knowledge available for the use in IA and facilitates the exchange of researchers and IA practitioners. It does so by describing scientific knowledge in the context of IA for Sustainable Development. Scientific knowledge is typically described and categorized by scientific criteria using disciplinary categories and keywords. Assessing the relevance of knowledge for application in societal decision making would either require technical and disciplinary knowledge on behalf of the user in order to assess the suitability of knowledge for the specific purposes or trusted relationships with scientist or consultant which may act as a knowledge broker. In this situation, it is particularly difficult for innovative tools to be considered for application in IA.
The LIAISE KIT facilitates the process of selection and interaction between researcher and IA practitioners by facilitating a description of IA relevant knowledge using keywords from the sphere of IA, and combining them with the disciplinary/scientific description of knowledge. Thereby, knowledge gets accessible for IA practitioners also beyond their disciplinary skills. Furthermore, by knowledge is being contextualized in the domain of IA. This facilitates the process of knowledge brokerage and the identification of relevant knowledge for the purposes of a specific policy proposal. In order to achieve the functionalities, the platform has been developed as a science policy interface on the basis of extensive user interactions (both IA practitioners as well as IA researcher) and testing. Several versions have been developed and features have been stepwise tested and implemented.
Scientific knowledge which may be used in the context of IA is implemented as different content types. The types of knowledge include (in brackets are current numbers of the respective type):
- Descriptions of models (ca. 100)
- Descriptions of methods (ca. 44)
- Expert profiles (ca. 60)
- Publications (ca. 370)
- Examples of good practices of IA (ca. 130)
- Model applications (ca. 135)
- Projects (ca. 40)
- Datasets (external sources)
These types of knowledge are modelled in different content types and are described by taxonomies. The taxonomies represent the domain of IA and in particular the different procedural steps and substantive requirements for IA. The taxonomies include:
- Impact areas (environmental, social and economic) (ca. 170)
- Process of IA divided in IA activities (ca. 30)
- Economic sectors (NACE)
- Spatial aspects (all countries and scales according NUTS)
- Policy areas (54)
- Policy instruments (11)
By combining keywords from the different taxonomies, the specific context of an IA can be described. For example, a policy proposal can be described as a policy instrument (e.g. regulation) in a policy area (e.g. health policies) at the national level (e.g. in Italy), affecting an industrial sector (e.g. chemical industry). The IA may address specific impact areas which are considered as relevant and can be broken down in specific steps of an IA.
The search engine of the LIAISE KIT is based on facetted search. The content can be searched by combining different taxonomy terms and a drill down of search terms which are relevant for a specific Impact Assessment. Thereby the user can identify different relevant pieces of scientific knowledge for his/her purposes. By relaxing specific requirements, similar items can be identified which may be still relevant for the specific purposes although they do not meet exactly the requirements.
Besides the content types and the taxonomies, a third element of the LIAISE KIT are the user roles and related rights. The following roles have been defined:
- Anonymous users can view most content
- Editors can upload and edit model descriptions, publications and other practices, and expert profiles
- News agents can upload new items (see below)
- Lead editors can edit impact areas or descriptions of methods (see below)
With this roles, it is ensured that IA practitioners may use the toolbox without registering.
The content of the LIAISE KIT is derived from a range of sources:
- Taxonomies are derived from an analysis of IA requirements, e.g. in guidelines for IA
- Analysis of literature, project reports and IA reports on model descriptions, methods, good practices of IA and model applications
- Descriptions of models, their applications, publications, expert profiles are uploaded by model developers and authors.
- Contributions of lead editors on method descriptions and impact areas: Lead editors are volunteers that provide state of the art overview text on certain sections of the LIAISE KIT, namely specific impact areas (such as employment, climate, innovation, etc.), or IA methods (such as cost benefit analysis, focus groups, surveys etc.). The contributions of lead editors are quotable and highly visible within the LIAISE KIT and for external search engines such as google. The contributions are provided in the framework of an easily editable wiki. A large share of the lead editors have been recruited from outside the LIAISE consortium and the platform provides a platform to disseminate expertise and project findings.
- Datasets are harvested from semantic web services. The EEA publishes its datasets via a SPARQL endpoint. The concepts and keywords which are used by the EEA to describe the data are matched with the LIAISE KIT taxonomies. The EEA semantic web services are queried by the LIAISE back office and the results are returned to the front office. Using these semantic web technologies, the LIAISE KIT can be integrated in the “web of data”. The LIAISE KIT content and the related ontologies are made available in standard formats (RDF) to be consumed by other data sources.
The functionalities as a community platform include the following features:
- A news blog informs the users about recent developments within the LIAISE KIT (e.g. new features, experts, models, publications, taxonomy terms, etc.) and in the wider community (e.g. events, calls, etc.).
- Every second month, a selection of news items is assembled in a newsletter (IA bulletin) and mailed to ca. 700 subscribers. The subscriber of the IA Bulletin are IA practitioners, IA researcher and LIAISE KIT user.
- The LIAISE KIT hosts a description of the LIAISE network, including the partner organisations, projects that are conducted within the LIAISE network, the LIAISE PhD network (offspring) and an overview on the LIAISE Lead editor.
- The community function is also facilitated by groupware functions. Closed user groups can be set up and administrators are appointed to manage the participation in these groups. The groups can be used for collaborative work on texts. There is a pre-defined format of a group which provides a template for an Impact Assessment. The template is structure along the steps of an IA and provides links to relevant resources (e.g. good practices, publications, methods) of the LIAISE KIT for the respective step of the process.
User guidance and a help desk is implemented in several levels:
- A manual which is accessible for all user describes the LIAISE KIT and its various functionalities
- A help desk manual provides a documentation for the administration of the LIAISE KIT
- Technical support in using the LIAISE KIT, software updates and backup is offered by Freie Universitaet Berlin. However, the software is designed to keep these efforts at a minimum. Registered users can upload and maintain their data typically without any support.
- Lead editors and experts can be approached for issues related to their fields of expertise. During the development and testing of the LIAISE KIT it turned out that a help desk represented by a single person or a small group does not dispose of the necessary skills to support IA in a meaningful way. A public forum has been tested to discuss IA related questions in a wider audience, but this feature had to be dismissed because of lack of acceptance among IA practitioners to discuss sensitive issues in a wider public. Therefore, the lead editors and closed groups have been established to provide targeted and detailed support for the various aspects of IA.
Quality assurance of the LIAISE KIT content is provided by different means:
- The feature of user feedback in forms of stars and comments as foreseen in early versions of the LIAISE KIT has been dismissed. IA practitioners indicated that they are unwilling to provide feedback, the number of potential users is too small to provide impartial and meaningful responses and the context of use determines quality and hence a judgement cannot be transferred to other contexts.
- A first proxy of quality is the level of detail which is provided in the description of IA models and methods. A reference model has been developed as a framework for a standardized description of IA models. Models are described from different viewpoints of IA, e.g. applicability on different sectors, regions, scales, impact areas, etc. The information would allow the user to assess if a model fits for its specific requirements.
- A second proxy of quality is the actuality of the information. Outdated model descriptions are moved to archives. The user are still able to query in the archives.
- A third proxy of quality are past applications and publications in peer reviewed journal articles. The modelers are invited to add publications and model applications.
- A fourth mechanism to assure quality are the lead editor. Contributions to IA methods are reviewed by the lead editor. A more formal approach for peer review of contributions to the LIAISE KIT has been considered but due to lack of time and resources the implementation is postponed.
Given the importance of user requirements and the specific context of Impact Assessments, the LIAISE KIT provides a platform to facilitate user requirements analysis. A throughout user requirement analysis is vital for developing knowledge which is applicable and relevant in decision making. Throughout the LIAISE project, we have tested several formats and approaches for the analysis of user needs:
• We realise that there is no ‘one size fits all’ approach, as the users of IA tools are inherently diverse and have complex requirements.
• 1:1 interviews typically provide the richest data, but are time consuming and require a substantial buy-in of users.
• Workshops and focus groups compromise in insight, but provide viable options and allow for follow-up with individual users later on.
• Questionnaires and electronic communication facilitate outreach and allow for gathering of input from a wide range of users.
• Challenges in time and availability of both tool developers and users to engage in a dialogue on requirements.
We conclude, that user requirements analysis is a process, not a one-off activity. This has profound implications for future user engagements and has emerged as a key message from the activities which involved direct communications with IA users. It was for instance illustrated by the fact that engaging with an established group of users working towards well-defined common objectives (e.g. regulatory IA in the United Kingdom in the case of Air Pollution Information System -APIS) facilitates communication. In addition to that, the development of clear, concise targets for new/improved IA tools can be established this way.
Working with usability and user requirements experts can aid tool developers (who are often scientific researchers, not software engineers) in eliciting user requirements. This could be a substantive contribution by a community of practice as outlined in the following, bringing together experts from different backgrounds, tool developers and users.
We have identified a need to differentiate between adding features and functionality and improving usability. This helps to determine where mainly technical improvements of the capabilities of tools are required, or how usability improvements could be made e.g. by better documentation or training in the use of tools.
The LIAISE-KIT can contribute to this process by establishing fora and connecting tool developers and users to foster a lasting dialogue, build trust and improve the understanding of user needs.
To summarize, the LIAISE-KIT:
• Provides a framework to describe knowledge in the context of IA
• Enables researcher to upload their publications, models, projects
• Provides a platform for researcher to publish on methods and impact areas
• Is harvesting and contextualising external data sources
• Is a platform for collaborative work on IA
• Provides a platform to post relevant news
• Can contribute to the user engagement process by establishing fora and connecting tool developers and users to foster a lasting dialogue, build trust and improve the understanding of user needs
It is thereby contextualizing knowledge and provides a platform to access innovative tools which is relevant for specified purposes of IA. Its content types (various types of knowledge), taxonomies (structured keywords from the domain of IA), the user roles (enabling and encouraging contributions to the platform) and the backoffice tools which contextualizes external data sources facilitates the science-policy interaction and overcomes path dependencies in the use of models and unrealistic requirements to assess the applicability and relevance of scientific knowledge in a specific context on behalf of IA practitioners.
4.1.3.2.3 Co-Design of Knowledge - Experiences from LIAISE Test Cases(WP6)
LIAISE explored IA tool use and knowledge exchange in six real-world cases to create a realistic understanding of the requirements of knowledge users in relation to possibilities of knowledge production. Testing was inspired by knowledge brokering approach, which challenges the ‘linear’ knowledge exchange by allowing exchange, co-evaluation and joint construction of knowledge with the aim of enriching decision-making. In particular, the test cases were interested in finding out in what circumstances, contexts, certain knowledge brokering approaches are fit for purpose in IA? In addition, the test cases aimed at investigating procedures for interaction between researchers and policy-makers. Thus, from the IA perspective it is important to find out when a step-wise approach of interaction in IA is suitable and when more dynamic and iterative processes are needed? Furthermore, the test cases examined current use versus possible uses of existing impact assessment tools.
Test cases (Table 3) played an important role in LIAISE, because they provided practical information both on tools in practice and user needs. Test cases enabled good relations between researchers and policy makers, and contributed to the main objective of LIAISE NoE to bridge the gap between these actors. Test cases involved several research questions/objectives, which were specified by individual test cases.
LIAISE's Test case package comprised of six cases varying from EU, national and regional level within an EU member state and in China. Interaction between experts and policymakers was core activity in all test cases. Different LIAISE aspects were tested depending on the focus of individual test case: LIAISE-Kit (former Toolbox), IA libraries; communication: IA innovation reporting/ policy briefs/ IA bulletin; Training curricula; Shared Research Agenda; Use of tools & informed use of tools; Understanding interaction between policy & research; and Awareness of LIAISE.
Table 3 Characteristics of the LIAISE test cases
Test Case Main Focus Key aim
Agri adaptation EU-level climate change adaptation in agricultural policy Develop knowledge brokering process enhancing the use of models in IA
Resource efficiency National and EU-level resource efficiency policies (Germany) Test how far knowledge platforms can contribute interaction
Climate policy National level climate and energy policy (Finland) Explore and develop interaction between modellers and policy-makers
Energy policy National level energy policy (Estonia) Explore how policy developers choose quantitative models for IA
Agri farmers Regional level agricultural policy (Greece) Develop best practise interaction process for IA of regional policies
Land use Regional level land-use policy (China) Test participatory tool combining scientific knowledge with stakeholder perceptions
During LIAISE testing process the knowledge brokering was summarised simply as a process of communication between researchers and users. But as there are different kind of policy process with variety of needs for communication and interaction, there are also several different strategies for knowledge brokering in practice. Thus, there is no single ideal form of brokering, but the choice of strategy depends on the context of the case. The knowledge brokering approaches applied and tested in LIAISE test cases were based on Sarah Michaels (2009) categories and are:
• Informing: one-way knowledge dissemination by researchers
• Consulting: provision of knowledge by researchers requested by policy-makers
• Match-making: broker identifies what, by whom and how knowledge can be exchanged in form of value to policy-maker
• Engaging: one actor (usually policy-maker) frames the discussion and knowledge exchange
• Collaborating: actors jointly frame the process of interaction and negotiate substance to address a policy problem
• Building capacity: actors jointly frame the process of interaction and negotiate substance to address multiple dimensions of a policy problem considering what can be learned for future
In reality test cases were versatile and included application of many knowledge brokering strategies.
Lessons learned for co-design of knowledge
Identifying and acknowledging the context of a specific policy case and IA is crucial: based on the literature and TC experiences WP6 developed a typology of different contextual factors affecting the selection and success of the Knowledge Brokering strategies and individual means and forms of activities in IA. Examples of contextual factors relevant for IA process are presented in Figure 1. A clear challenge for most of the test cases was the interaction with policy makers. One reason for this might be that most – if not all – test cases were rather ‘supply-driven’ meaning that they had been initiated and mainly also planned by LIAISE researchers. Test cases were rather forced to carry out their activities during a certain period (the LIAISE testing period) which was necessarily not tied in with the real-world policy processes or policy makers’ needs. In addition, policy officers have established tendering processes and contacts to the other information producers (e.g. consultants and academia). It became evident during testing that knowing and acknowledging the context of a specific policy case and IA is crucial.
Test cases faced several challenges during the testing process. ‘Match-making’ knowledge brokering approach, for example, was proven to be non-sufficient to embed research-driven tool development process in policy area. It, however, opened up a route for communication between researchers and policy-makers. It might also be difficult for researchers to follow in-house routines of policy-making (e.g. there might be delays in interaction due to bureaucratic reasons). ‘Collaboration’ knowledge brokering approach was less successful for jointly framing the tool development exercise because open policy questions did not fit within the limitations of the existing models. For researchers this appeared as ‘lack of demand for a tool or knowledge’. Furthermore, both researchers and policy-makers might have been reluctant to move beyond ‘business as usual’ in the knowledge exchange during IA. Some researchers also felt that linear IA is process it is too rigid for complex and multi-level policy problems and processes.
Figure 1 an overview of the issue with relevant lessons learned from the test cases
Table 4 Barriers for successful knowledge exchange in IA identified in test cases include
Policy-makers: Researchers: General:
• Diverse needs
• Thematic thinking vs. SD
• Ownership of IA – no room for out-house researchers
• Monopoly, non-open models
• Timing of knowledge supply
• Power struggle between policy sectors
• Few knowledge producers (national and regional level)
• No second opinions available (national and regional level) • Supply-driven models and tools
• Disciplinary instead generalist approaches
• Competition between researchers, models and dialogs in science
• Gap in use because of publication process
• Different approaches in science/ methods/ disciplines/ ontologies/ worldviews
• Endangering trust is avoided – results not given until final – lack of testing • Selective use of proof
• Pre-defined agendas
• Business as usual relations between actors
• Organizational changes
• Lack of continuity – project based culture
• Broking between tools and policy questions missing
• Very technical focus of IA and use of evidence
• Lack of openness and flexibility
• Non-focus of motivation and objectives of SPI
Despite of the challenges, the test cases succeeded in building collaborative relationship with the policy-makers during testing. Tools with wide range of application possibilities (e.g. flexibility to accommodate new goals) appeared to be particularly useful and provide an opportunity for interaction. In addition, it appeared that face-to-face meetings with policy-officers and researchers’ active role in promoting a new tool for IA in the scoping phase of the IA are very useful. Engaging active and specialised policy-makers helps researchers to get technical feedback and tailor supply-driven tool development. LIAISE Kit can also serve as a platform for exchange and networking, especially for informing and match-making knowledge brokering. IA researchers can furthermore act as a facilitator in policy-policy interaction by organising seminars and focus group meetings related to inter-ministerial multi-dimensional problem framing and solving (match-making and capacity-building knowledge brokering). It must, however, be emphasised that the success of knowledge brokering, tool application or tool improvement is possible only when the role of the IA knowledge provider or knowledge broker is clearly defined in scoping and planning phase and there is trust and credibility on both sides.
Table 5 Opportunities for future knowledge exchange in IA identified in test cases include
Policy-makers: Researchers: General:
• Many needs
• Support for holistic thinking, trade-offs
• Enhanced use of existing proven tools
• Consistent approach preferred – procedural tools
• Engaging with other policy sectors in complex issues
• More knowledge producers needed (national and regional level)
• Second opinions needed (national and regional level) • Early engaging in model and tool development - increase in use
• Generalist approaches useful in complex problems
• Meta models - interdisciplinary approaches
• Publication of policy briefs on ongoing activities – expectation management
• Openness about modelling (assumptions etc.), increase in trust and credibility
• Researchers’ achieving better broking skills • Comparative use of proof
• Flexible, changing agendas
• New opportunities for boundary organisations and brokers
• Network culture – durable IT or personal ad hoc support
• Matching tools with policy questions
• Co-design of knowledge instead of linear transfer
• Constant (con)testing of knowledge: no ‘orphan’ knowledge
• Open co-tailoring
Collaboration in IA increases the use of IA results in policy-making but requires trust and credibility based on previous cooperation or built during (long) IA process. The IA researchers/knowledge providers must, however, be open about limitations of a tool to increase credibility. Developing a new tool for real-life ongoing policy-process is rather demanding due to time and political constraints. Instead, there might be room for tool tailoring or improvement.
Based on LIASE test cases it appears that iterative co-design of knowledge is enabled by tools that are:
• Already existing
• Easy-to-apply/ transparent
• Time-saving for policy actors
• Procedural but allowing choices, and
• Including an element of co-tailoring relevance by policy-makers’ questions
4.1.3.2.4 The Shared Research Agenda for IA
Survey on scientific tools designed for policy IA
LIAISE conducted an extensive survey on scientific tools designed for policy IA. We analysed the abstracts of 7781 projects funded in the European FPs 6 and 7 provided on the European Cordis website (Podhora et al., 2013 , LIAISE Innovation Report #6, LIAISE Policy Brief #5, D2.2 D2.3). We selected 203 projects that developed, extended, applied and/or tested tools for the IA process. We concentrated on quantitative or qualitative tools (models, participatory tools etc.), their components (e.g. indicators and comprehensive analytic methods) and superior evaluation frameworks (toolboxes and platforms). We structured the analysis of the projects that designed the tools according to (i) 36 European policy areas (European Union, no date), (ii) 35 impact areas outlined in the European IA guidelines (SEC, 2009 , amended by “sustainable development in general”), (iii) the jurisdictional levels (from international/global to local), and seven tool categories (de Ridder et al., 2007 assessment framework , scenario analysis tools, multi-criteria analysis tools, cost-benefit/ cost-effectiveness analysis, accounting tools, physical analysis tools and indicator sets, modeling tools, amended by category “other”). In each of these groups, we counted the number of projects to identify research peaks and gaps. The analysis of the projects funded identified the following key results:
• Policy-relevance of research: A small percentage (less than 3%, equalling 203 projects) of the projects funded in FP6 and 7 provided tools for policy IA. About half of these projects identified provided tools for environmental, agricultural and transport policy areas. Tools designed for these three policy areas were subject of a further in-depth analysis presented in the next points.
• Impact areas and sustainability dimensions: The tools mainly addressed the impact areas corresponding to the policy areas (environmental impact areas for environmental policies, the impact area “land use” for agricultural policies etc.). Social impact areas were generally poorly addressed by the tools. The tools were mainly designed for one to two sustainability dimension(s). They hardly comprised all three sustainability dimensions and sustainable development in general, respectively.
• Jurisdictional levels: The tools were mainly designed for European policies and hardly for other jurisdictional levels (as international or national levels). They were mainly designed for a single jurisdictional level and not for multi-level governance.
• Tool categories: The majority of the projects designed several tools. Most tools had a quantitative character; participatory tools were poorly covered. More than half of these tools could not be categorized according to the current seven integrated assessment categories identified by de Ridder et al. (2007) .
• Terminology challenges: Many tool descriptions did not refer to the policy-relevant terms defined by the European Community (e.g. impact areas as set out in the European IA guidelines, policy areas of the European Union).
LIAISE SRA – methodology, framework, target groups
The setting of research agendas has been an emerging topic especially in the past five years. So far, research agendas have mainly been developed from two different communities and thus from two different angles. They have been either designed by the scientific community (e.g. EPBRS, 2010) or by the policy-making community (e.g. ESFRI, 2008) . So far, no research agenda could be identified that exclusively focussed on policy impact assessment (IA). However, a few authors addressed aspects related to IA (Turnpenny et al., 2009 on policy appraisals and Pope et al. 2013 on various IA instruments) .
The results of the survey clearly identified the need for a Shared Research Agenda for policy IA. Its development was based on seven steps:
1. Research gaps identified for IA related research projects funded in the European FPs 6 and 7
2. Expert workshops on ecosystem services, soil and transport/ innovation in relation to the three leading policy areas
3. Gathering and discussion of research gaps within LIAISE work packages and test cases
4. Presentations and additional discussion rounds
5. Translation of research gaps and results into research questions presented in the LIAISE SRA
6. Comments from and public discussion with scientists and policy-makers at the LIAISE dissemination conference
7. Dissemination
The LIAISE SRA addresses the six target groups (see figure 2), three from the scientific community and three from the policy community, while here the European Commission is used as an example system:
1. Scientific community:
a. no experiences with policy-oriented research
The SRA could assist this group to align their research towards a more policy-oriented focus. This support may also help researchers to increase the policy-relevance of fundamental / basic research.
b. (first) experiences with policy-oriented research
The SRA could support researchers who already are experienced with policy-relevant research to identify, strengthen and discuss possible topics for their upcoming research.
c. political and sustainable development sciences as policy-relevant research per se
The SRA could provide insight for political and sustainable development scientists to better understand open questions the scientists (who are no experienced with policy-relevance) have to address when delivering evidence to policy-making.
2. Policy-making community:
a. research policy-making
The SRA could provide information for the design of research policies, specifically the upcoming research programmes of Horizon 2020 in the fields of IA in general as well as in specific disciplines (e.g. in the exemplary themes soil, ecosystem services and transport/ innovation in relation to IA).
b. European research institutes/ in-house research
The SRA could assist European research institutes that work as policy-related knowledge brokers to better understand the kind of open questions that need to be covered for better providing scientific evidence for the policy process.
c. European DGs
The SRA could provide information about research gaps in IA for the European DGs (e.g. with respect to the current SRA focus particularly to the DGs AGRI, ENV and MOVE).
Figure 2 Target groups of the LIAISE SRA
The LIAISE Shared Research Agenda for Policy Impact Assessment
The SRA identified existing IA tools and outlined research needs based on expert discussions. The SRA thereby addressed all IA spheres: the policy sphere (tool uptake/ use), the scientific sphere (tool provision) and a combination of both spheres to improve the mutual understanding of the individual needs and interests within the spheres. It thereby concentrated on the exemplary themes “IA in general”, “ecosystem services”, “soil” and “transport/ innovation” as a starting point. These themes were selected due to their relation to existing IA relevant research as well as current and future policy requirements. These examples helped to identify strengths, weaknesses and implementation challenges of the present SRA, particularly to provide information for a continuous update of the SRA.
The LIAISE SRA consists of three levels that build upon each other as a pyramid (Figure 3). It focuses on three exemplary themes: IA in general, ecosystem services, soil and transport/ innovation (see below).
Figure 3 Levels of the LIAISE Shared Research Agenda pyramid and scientific target groups
Level 1 addresses the general research scope for policy IA and suggests a general structure for research funding in relation to the science-policy interface and IA. The information can be offered to research funders to design research programmes. The structure comprises the three spheres of the science-policy interface:
- Policy-relevant topics: Policy-relevant themes as policy and IA processes,
- Scientific tools: Assessment methods as provision of tools and data and methodologies, and
- Knowledge transfer: Bridging the science-policy interface.
Level 2 suggests guiding research questions that detail the three sections of level 1 for each of the four exemplary themes of the LIAISE SRA, for example, for IA in general: Topics: How do different scales influence the IA process and results? And Transfer: How can results be presented and communicated in a way that supports their inclusion into the policy process?. Another example of such research questions for soils is Topics: How can the impact of land utilization on soil functions and services better be incorporated into policy-making? Tools: How can harmonization and standardization of classification, management and storage systems for data, tools and indicators serve for the purpose of IA?
Level 3 details the guiding with specified research questions for the four themes. Both levels may assist the funding bodies when specifying their research programme and researchers when outlining their superior research questions (e.g. in internal institutional programming, institutional working groups, large research projects). They provide suggestions for the actual design of new and interdisciplinary research questions.
The next steps of the LIAISE SRA
The continuous update is a central element of the LIAISE SRA. The constantly high level of the quality of the SRA can be reached by linking the updates to the LIAISE KIT as expert, networking and communication platform. It is structured along policy-relevant and scientifically relevant taxonomies as impact areas, IA methods and IA models. Further, it is important that the future version of the SRA reflects the comprehensiveness of sustainable development. The relation to sustainable development relevant in the IA process can also best be addressed by the kit due to the taxonomy of the 35 European impact areas. A SRA lead editors in the KIT will coordinate the updating process. When updating the LIAISE SRA it is important to constantly invite the wider IA community to participate in the updates and to inform the policy-making and scientific community on the results. Thereby, LIAISE shall work hand in hand with well-established (research) networks and associations. This updating concept thereby relevantly contributes to the LIAISE objective of further consolidating the IA research community.
The LIAISE training modules provide support on policy-relevant research to all levels in a scientific career, from early to senior stages with view to policy-relevant and IA related research. The training module B for advanced and senior scientists reflects on the structure of the LIAISE SRA: policy-relevant topics, scientific tools and knowledge transfer. The main objective is to increase the autonomy of the participants to individually check the policy relevance of their research proposals and projects by means of the LIAISE self-test. Further, early career researchers are to become familiar with policy-relevant research with module A to help shaping their expertise and profile.
4.1.3.3 Synthesis: Situational analysis of the gaps
LIAISE conducted an empirical study on the reported tool use in EU impact assessment (Jacob et al. 2013 – published on LIAISE website). The study examines all 805 IA reports on EU level, which were published between 2003 and 2013. In general, it can be observed that scientific knowledge is still underutilized in policy-making processes. For example, numerous scientific models are available which are able to cover a variety of policy issues, different scopes and scales. However, the analysis showed that only a relatively small number of IAs on EU level used those models in the impact analysis. Out of the 805 IA reports that were published in the evaluation period, only 222 (27,6%) reported the use of quantitative models for the assessment of the impacts. Model use has significantly increased over time: from 5% in 2003 to 39% in 2013. However, the analysis also showed that the usage of scientific models still varies considerably among the different policy DGs. For example, in DG CLIMA, 90% of IAs were based on modelling results, while e.g. in DG MARKET and SANCO only a minority of IAs have been reported the use of models.
Figure 4 Reported use of quantitative models in Impact Assessments by the EC
Furthermore, the analysis has revealed that only a small part of the models which are described in the LIAISE KIT have been utilized in IAs. At the time of the analysis, 19 models described in the KIT have been used in EU IA, while 72 have been never utilized. This points to the wider research reservoir that could contribute to future IAs.
To conclude: although the application of models has significantly increased in the last decade, the full potential of scientific knowledge is not yet utilized. The perceived gap between the availability of scientific knowledge and it’s consideration in actual decision-making processes that was the starting point for the LIAISE project, can still be observed. The underlying causes will be described below.
A first explanation for the gap between research and practice is that IA for SD belongs to the category of wicked problems, described in Rittel et al. 1973 . Societal problems that are difficult or impossible to solve due to: incomplete or contradictory knowledge; the many values, interests, opinions and numbers of people involved; the economic consequences and complex relations with other societal challenges. E.g. health-nutrition; nutrition- education-income; income-economy; economy-demography-natural resources. Wicked problems cannot be solved by traditional, linear systems of knowledge production with researchers providing undisputed ‘truth’ to society. Here the role of science is to deliver knowledge and information that users perceive as salient, legitimate and credible . In basic sciences the linear scientific process is the only way forward. Scientific progress does not depend on input from and interaction with policy makers. Wicked problems however require a truly transdisciplinary approach: strong horizontal interaction with external stakeholders in a joint process of innovation and shared learning . The compliance testing work in LIAISE on the reality of policy making confirms this general picture. It revealed a gap between the aims of IA and its actual use. The application of IA is difficult and often at odds with the process of decision making . A systematic review of research-practice interaction around policy appraisal concludes that practitioners and IA researchers are not proceeding their practices and research in parallel but are ‘drifting apart’.
A second cause for the existing gap stems from the different types/cultures of knowledge production within the scientific community as such. Table 6 gives general characteristics as described by Gibbons . The sharp distinction between Mode 1 and 2 of course is a simplification. In the LIAISE NoE many researchers work in both modes, albeit with unequal preferences. There were quite some occasions where differences in the scientific cultures had to be addressed in order to come to a productive collaboration. This in particular applied to the case studies with a mix of modellers from the basic sciences and researchers from policy sciences. Specific IA related gaps in this category were identified in a LIAISE evaluation of nine years of European research funding . E.g. boundaries between disciplines, impact areas and jurisdictions and problems related to geographical scales and time horizons.
Table 6 Mode 1 and Mode 2 knowledge production (Gibbons et al. 1994).
Mode 1 Mode 2
Problems set and solved in the context of the (academic) concerns of the research community Problems set and solved in the context of application
Disciplinary Transdisciplinary
Homogeneous Heterogeneous
Hierarchical, tending to preserve existing forms of organisation Heterarchical, involving more transient forms of organisation
Internal quality control (peer review) Quality control is more socially accountable
A third element of general relevance for the gap is the lack of a common platform for information exchange and discussion about the production and use of IA knowledge across sectorial and disciplinary divides. Such a platform could enable IA knowledge workers to jointly develop a shared research agenda and plan-for-action. Some international communities (e.g. IAIA, TIAS) exists, which operate as loose networking fora to share developments. More in depth confrontation and alignment would enable a more efficient use of resources towards a common goal. Networking mechanisms are not in place to jointly explore the possibilities and engage in each other’s discourses. The lessons learned in our internal interaction processes in the NoE provided useful building material for the vision on a future Community of Practice.
A common platform for knowledge exchange could enable IA knowledge workers to jointly develop a shared research agenda and plan-for-action. Some international communities (e.g. IAIA, TIAS) exists, which operate as loose networking fora to share developments. More in depth confrontation and alignment would enable a more efficient use of resources towards a common goal. Networking mechanisms are not in place to jointly explore the possibilities and engage in each other’s discourses. The lessons learned in our internal interaction processes in the NoE provided useful building material for the vision on a future Community of Practice
Potential Impact:
4.1.4 Potential impacts: main achievements from a societal perspective
4.1.4.1 Intro: the LIAISE approach to generating impacts
The ambition to aim for long-term impacts that go beyond the project objectives was an important feature of the LIAISE approach towards evidence based policy-making. To achieve this, the on-going evaluation by Ecologic (section 4.1.3.1.3) was included in the project design to facilitate internal communication; to take stock of where the project is heading to; to support reflexive and constructive discussion on both project objectives and results; and to decide on possible adjustments. In doing so the basic questions are: 1) are we doing the right things, and 2) are we doing things right? The Impact Chain Approach was used as a method for structuring this discussion with Inputs Activities Outputs Outcomes Impacts:
• Inputs are the resources that are used in an activity. This level of detail is not relevant for the MB discussion at this stage any more (it was during the proposal writing).
• Activities are what you actually do to create the change you want to achieve.
• Outputs are the most immediate results of the activities in the work packages. They create the potential for outcomes and impacts to occur.
• Outcomes describe the true changes that occur as a result of the activities.
• Impact is the vision of a preferred future that underlines why the action is important. It refers to the longer-term change that you hope your action will help create.
Table 7 Examples of applying the Impact Chain Approach to LIAISE
Activity Output Outcome Impact
WP3 (toolbox back office) and WP4 (toolbox front office) Toolbox More used tools & better informed use of tools in IA Better informed policy making
WP2 Science for IA tools Mapping of IA research Increase awareness of IA (Research & Policy) & research funding agencies Set better targets and trigger actionable research on IA
WP5 (durability) Business plan Self-sustainable funding Continuous science/policy interface
WP7 (dissemination & training) Training curricula Users have skills Better informed policy making
LIAISE used this approach by thinking backward along the chain, starting with the desired impacts by asking “what do we want to accomplish”; next to ask “which outcomes and outputs are required” and finally “which activities are needed”. Table 7 gives some examples; in reality more impact chains were needed to ‘funnel’ a number of activities in different WPs towards one single impact.
The following steps have been taken:
• Workshop with the MB (Amsterdam, December 2010). Presentation and adoption of the impact chain approach as the method for the evaluation. Identification and ranking of project outcomes and impacts.
• Annual Meeting (Milan, January 2011). Presentation of the method and results of the MB workshop. Discussion of the structure of the evaluation and the linkages to WPs. Adaptation and further refinement of the ranking of outcomes and impacts.
• Presentation to the Policy Board (Brussels, February 2011). General approval of the proposed approach.
o Creation of a logic model:
o Elaborating on the information gathered from stakeholders.
o Creating causal impact chains by linking outcomes and impacts to outputs,.
o Reviewing and refining the logic model in discussion with LIAISE partners.
o Developing indicators and an evaluation timeline in line with the deliverables schedule.
• Conducting the evaluation and reporting.
4.1.4.2 Charter and vision on a future Community of Practice
For mobilising the available scientific knowledge reservoir on IA for SD to its full potential, it is crucial that the societal stakeholders perceive the contributions from science as salient, legitimate and credible . This requires changes in:
• The ways IA research is programmed and evaluated in interaction with users;
• The ways scientific researchers and knowledge workers in other sectors of societal activity produce and exchange new knowledge and information;
• The interaction between mode 1 and mode 2 researchers;
• The ways knowledge and information are disseminated in order to better harness the existing knowledge reservoir for the needs of IA for SD;
• The interaction between scientific researchers and policy makers in concrete policy cases. Here special attention is needed for Knowledge Brokerage.
This process oriented view is quite different from the model and tool oriented view at the start of the NoE. The change reflects the improved interaction and collaboration between researchers from basic sciences and social/policy sciences. The lessons learned show that not tools as such, but the information supply about tools should be standardized and that there is a strong need for a platform for strategic reflection by IA researchers on their role and tasks in relation to IA for SD. Research on the environmental, economic and social dimensions of SD is not sufficient to provide the policy process with salient, legitimate and credible scientific knowledge. Additionally requires features are:
• An integrative, open and transdisciplinary approach by a Mode 2 scientific process in dialogue with IA practitioners and policy makers;
• Transparency about what can/cannot be expected from science in a specific decision making process, as well as transparency about trade-offs between the different dimensions of SD and related interests/values/stakeholders; between short/long term impacts; and between geographic entities, or jurisdictions;
• Contextualisation of IA knowledge by: a) actively involving expertise in the field of policy analysis and knowledge brokerage in the scientific process; and b) standardized descriptions and quality evaluation of (transdisciplinary) IA methods and tools. This should enable potential users to assess the relevance of the available scientific knowledge and information for their specific use cases.
• Procedures and norms/standards for research programming and funding, planning and execution, dissemination and quality monitoring/evaluation that support 1-3.
These features will be realized by a Community of Practice on IA Research for SD . for IA knowledge workers in research and other sectors of societal activity. It provides the platform, as well as the networks, procedures and services that are needed to upgrade their scientific contributions to the process of IA for SD by:
• Fostering individuals and organizations to learn and be challenged;
• Transcending traditional disciplinary and science – policy boundaries;
• Actualizing a new type of science, based on inter-disciplinarity, transparency and active stakeholder involvement (transdisciplinarity);
• Collecting, assembling and disseminating critical insights for society on the complexity and relevance of long term perspectives on SD;
• Providing an arena and conducive environment for discussion on IA for SD.
-------------------------------------------------------------------------------------------------------------------------
The Charter
Preamble. SD as a guiding principle has been widely accepted by the public as well as private sector and civil society. Ex-ante IA is becoming increasingly important to ensure public support for, and trust in proposed societal changes towards SD, e.g. in terms of green economy, resource efficiency and sustainable and smart solutions. Ex-ante IA has the potential to reveal the short- and long term economic, environmental and social impacts of a policy throughout all policy sectors and levels of decision making. However, to operationally integrate the principle of SD into day to day IA processes, remains a serious challenge for policy makers as well as for researchers. The Charter is a key element in establishing a Community of Practice on IA Research for SD. It represents the shared views of the IA research community on how to move ahead in improving research on IA for SD in a dialogue with stakeholders from outside the scientific community. The Charter addresses the issues that are seen as relevant for IA for SD from the perspective of scientists (scientific, organizational, administrative / financial and cultural / behavioural).
The LIAISE community is aware of principles for researchers developed in related contexts such as the (Lund Declaration, 2009), the (Bellagio Principles, 1996), or the (DeGEval Standards 2008). However, this charter directly addresses the way research is planned and conducted in the specific context of IA research for SD. The Charter expresses an intentional commitment of the partners in the CoP: a general endorsement of the views, principles and guidelines. Our Commitment. The partners in the future CoP are committed to developing solutions that contribute to SD. It is their aim to overcome the gaps between research and societal decision making identified in the previous section of this document by developing and designing knowledge production that is targeted at achieving societal impact.
Hence, it is the overall aim to create knowledge that meets the following criteria:
• Relevancy: We are committed to creating knowledge that is targeted at making an impact for society and societal decision making.
• Credibility: We are committed to creating knowledge that is trustworthy and embedded in societal contexts.
• Legitimacy: We are committed to creating knowledge that takes into account divergent world views of different stakeholders in a balanced way.
To achieve this, the partners are committed to a set of guiding principles, which are reflected in their activities in the field of research on IA for SD:
• Commitment to quality: We are committed to scientific excellence and will utilize the mechanisms of scientific peer reviews to this end. Besides, we will contribute to the development of other quality attributes of knowledge in order to improve the relevancy for society.
• Commitment to transparency of the scientific process: We are committed to clarity about methods and assumptions underlying the research process as we are aware of the risks of offering IA knowledge and tools as black boxes. We are also committed to including the discussion of their limitations and being explicit about entailed normative world views, which we are willing to discuss.
• Commitment to contextualization: We are committed to carefully analyse the context of decision making and societal problems and to adapt scientific inquiry accordingly in order to link research to relevant societal developments and to ensure its relevancy for the decision making process.
• Commitment to dialogue: We are committed to become engaged in dialogues with societal decision makers in order to better understand and respond to societal knowledge needs. We are also committed to dialogues with decision makers and research funders targeted at collaborative research programming. In this way research questions shall be developed across different disciplines and together with societal actors to conduct research that meets societal needs.
• Commitment to considering long-term and integrative perspectives: We are committed to carefully analyse potential trade-offs and synergies between the different dimensions of SD and the short and long-term perspective to enable decision makers to meet the challenge of considering these trade-offs in policy making. This entails openness for interdisciplinary research.
These principles should be considered and applied adequately on a case by case basis. Not every research project and not every IA requires a consideration of the principles of IA research for SD. The principles open up additional options, rather than being meant as prescriptive.
-------------------------------------------------------------------------------------------------------------------------
4.1.4.3 Stakeholder Analysis
Table 8 Stakeholders, stakeholder interests and potential added value LIAISE
Stakeholders
Horizontal Units: units overseeing quality of IA process (e.g. IA units at EC or national level, EC IA Board) Stakeholder interests Potential contributions / added value LIAISE
Networking and discussion forum Meta-data on IA for SD knowledge Repository of methods and models on IA for SD Good practice examples of IA for SD Methods for transdisciplinary research Good practice examples of trans-disciplinary research Quality standards for models Shared Research Agenda
Context: Development of requirements and guidelines for IA for SD
Horizontal Units Quality of IA process X X
Policy Units Represent own policy issues adequately in IA X X X X
Efficient use of resources to conduct the IA X X X
International Organisations Quality of IA process X X X X
Politicians Increase influence on decision-making process X
Consultants Giving policy advice X X X
Context: Design of an IA process
Policy Units Narrowing down options X X
Politicians Transparency of the analysis X
Legitimacy X
Stakeholders in business / civil society Early influence on policy-making X X X
Transparency of the analysis X X X
Legitimacy X X X
Horizontal Units Ensuring quality of IA X
Context: Decision on using models in an IA process
Policy Units Applicability of the model X X X
Insight in / understanding of process and results X X X
Horizontal Units Ensuring quality of IA X
Insight in / understanding of process and results
Consultants Application of their models X
Data availability X
Context: Funding of research and development of models
Researchers Scientific interests X X X X
Data availability X X
Policy relevancy (model improve-ment based on feedback from tests in IA practice) X X X
Research funding agencies Ensuring excellence of research X X X
Ensuring policy relevance of research X X X
Policy units Policy relevant models X X X X
4.1.4.4 Ongoing evaluation
As indicated in the DoW the Ecologic Institute has been commissioned to conduct an ongoing evaluation of LIAISE which has accompanied the project from the start and is complementary to the standard FP7 project evaluation procedures by the Commission that focus on monitoring the project progress in terms of the realisation of the project objectives. Here the focus was on providing timely feedback on the progress towards the expected long-term project impacts. This induced critical reflections that enabled the project actors to consider and if necessary, undertake timely adjustments. A first evaluation report was completed in August 2012. It was followed a few months later by the evaluation of the toolbox. Finally, the second round of the evaluation of LIAISE, which was focused on the post-project entity, took place during the last months of the project. Here - on request of the coordinators – special attention was given to the long-term objective of transforming the NoE to a durable network. Ecologic Institute was therefore asked to focus on critically assessing the positioning of the future LIAISE entity in the current IA landscape. Therefore the evaluation was targeted at the different roles and functions of LIAISE as well as the project as a whole. The final report was published at the end of April 2014. This section summarizes strategies to improve the potential impact of the project and to increase the attractiveness of the future network, not only to users, but also to its members. Section 4.1.3.4.1 describes how the on-going evaluation was used to focus on the envisaged long-term impacts of the project.
The recommendations for the various roles of LIAISE are listed in Table 9. The most important recommendations emerging from the evaluation are probably those regarding the leadership and coordination of the LIAISE community of practice (CoP), as it is the foundation of all other aspects of the network and must therefore be solid.
• Increase the focus on the needs of members of the CoP rather than focus exclusively on user needs. While it is important to position LIAISE as a network and ensure that users find it attractive, the network is nothing without its members and their needs must be understood and taken into consideration. If they are to commit to the CoP, they must to get something in return for their time and work.
• The CoP should work with a long term vision but short-term plans rather than only with two-year plans. This will enable a more regular assessment of progress and create more achievable goals, allowing for results to be visible faster and credibility to be built.
• The LIAISE network needs a strong leadership and a clear distribution of responsibilities. Due to the absence of an institutional structure, it is easy for roles to become unclear. It is important to have a clear person or group of people in charge of setting the agenda and crying foul when there is lack of activity or a lack of coordination.
• LIAISE must rapidly secure commitment for “infrastructure items” such as the online platform and IA Bulletin, as they are not project-funded and need a continuous flow of money to run.
Table 9 recommendations for the various roles of LIAISE
IA knowledge and information hub
Optimize the Google search for the LIAISE website. If LIAISE aims to be a hub for IA knowledge and information, it is important for it to appear in the top results for searches for keywords like “impact assessment” or “IA tools”.
Continue emphasizing the community aspect of the LIAISE website. This is an added value lacking from most alternative offers, and it is more likely to help bridge the gap between policy-makers and researchers.,
Focus on enabling cooperation and synergies rather than theoretical discussions, as journals are seen as the reference for the latter. The hub should rather focus on networking and research collaborations.
Secure funding to run the online platform smoothly as it should always be running and updated regardless of the amount of project grants within the CoP.
Networking and discussion forum
It is crucial to ensure the active participation of policy-makers (European Commission and Member State level) as this is the main added value of the LIAISE forum.
The national units which advise and coordinate national IA work are a promising group of policy makers to connect to.
Considering the existence of alternative offers, LIAISE must strengthen the link to other existing communities/discussion forums (e.g. IAIA and many others)
Internally, LIAISE must strategically foster connections on two levels: experts and heads of partner institutions. Heads of institution are well connected and important to find synergies and common research opportunities. Experts are needed to have more concrete interactions in the discussion forum and help steer the research agenda.
Innovation and testing
Ensure the presence and participation of high-level or well-connected people in the network to involve whole institutions rather than just experts. This will allow for more possible overlaps for projects.
LIAISE coordinators must deepen regular contact with policy-makers and involve them in research programming. This is important not only to attract people to the network, but to ensure that the research emerging from the CoP will actually be policy-relevant.
Tool identification and quality monitoring
It is essential to work on establishing the credibility of LIAISE KIT certification. Modellers will only pay for having their tool certified if they trust it will result in more uses. Therefore users should consider the LIAISE certification as a valid seal of quality.
To bypass the problem described above, creation (if necessary for free) of a critical mass of certifications might be a good idea. It would allow modellers to become familiar with the certification and to trust it, encouraging other modellers to pay for it in the future.
An option might be to pursue endorsement by the European Commission to ensure credibility. It is an authoritative institution which will be trusted by both policy-makers and researchers.
Learning Feedback from LIAISE Schools was generally positive, but there are still some ways in which it could be improved.
Start implementing the designed curricula for policy-makers, as so far only schools for researchers (in particular early career ones) have been held.
Training coordinators should diversify the venues used for dissemination and improve the advertising strategy, as there are some issues getting a sufficient number of participants. Existing databases of training programs and PhD courses should be used.
The exchange between researchers and policy-makers should be encouraged rather than always training them separately. An idea would be to invite some policy-makers to speak at a School for researchers and vice-versa.
Again the national units which oversee and advise national IAs are an important target groups as “training the trainers” would be a very efficient tool of dissemination
Opportunities to include LIAISE topics and trainings into existing training programs should be actively pursued. There are plenty of alternative offers in the market but they do not need to be seen as competition, as LIAISE can often add a certain value to existing programs. Their scopes do not perfectly overlap.
4.1.4.5. Dissemination and Implementation Plan [including business plan]
The CoP will be designed as a Network governed by a Board and with an Executive Committee supported by a small Staff Office. Characteristics:
• The network unites partners with a research focus on IA and with expertise in the fields of the 3 dimensions of SD and/or policy sciences.
• The network is open and inviting to partners committed to developing a shared vision and research agenda by addressing key issues in research on IA for SD such as: evidence based policy making; creation of research networks and infrastructures; creation of synergies in research programming and funding across national boundaries; foresight analysis of emerging needs; strengthening science-policy interfaces, etc.
• Participation in the activities is open to external parties and individuals. Although this Charter is targeted at the research community, the partners in the CoP are strongly aware, that unilateral action in the scientific community will not be sufficient to develop shared strategic views on research on IA for SD. Contributions from practitioners and research funders are essential for a successful implementation of the principles laid out in this Charter. Therefore all stakeholders interested in the subject of IA for SD are strongly invited to actually participate in the operational activities to be organized by the CoP.
• The network actively seeks a dialogue, strategic linkages and synergies with relevant external networks and institutions in, or outside the scientific community, e.g. policy makers/users in the private sector, consultants/desk officers/other IA practitioners, the general public and other stakeholders.
• The rules of governance for the CoP will be described in a Partnership Agreement
Table 10 Roles and functions CoP and its products and services for specific stakeholders
Role and functions Products/services Users/stakeholders for 1-7
IA knowledge and information hub
• Web-platform on IA for SD
• News agent
• Meta-data repository 1. CoP/Network with networking and information services and linkages with external research networks and stakeholders.
2. Meta-data on Knowledge for IA to enable comparison, reflection and evaluation of these models and methods.
3. Procedural principles for IA for SD. An overview of procedural standards to conduct IAs.
4. Quality Guidelines for Models with special attention for policy-relevancy.
5. Guidelines / examples of methods for transdisciplinary research.
6. Shared IA Research Agenda with a focus on knowledge generation at a strategic level to enable researchers addressing policy makers’ needs.
7. Staff exchange of early career researchers. Researchers/students (1, 2, 3, 4, 5, 6, 7)
Policy makers and policy units (1, 3, 4, 6)
IA practitioners: desk officers and consultants (1, 2, 3, 4)
Horizontal units with the task to provide guidelines for IAs and monitor the quality of IAs (1, 2, 3, 4)
Research programmers and evaluators (1, 2, 3, 4, 5, 6)
Funding agencies (1, 4, 5, 6)
Private sector organisa-tions (1, 2, 3, 4, 5)
Networking and discussion forum
• Think tank
• Methodological reflection
• Interaction with policy-makers at EC and MS
Innovation and testing
• Innovation generator, Experimental Lab
• Research programming
Tool identification and quality monitoring
• Evaluating quality of IA and IA tools
• Database with meta-information
Learning
• Reflection on experiments with shared learning
• Training courses
The design of the roles, functions, products and services of the CoP (Table 10) resulted from a stakeholder analysis based on the experiences in the NoE, the many contacts with external stakeholders (institutions and individuals) in workshops, conferences and other meetings and on the discussions with the LIAISE Policy Board. For an effective and efficient performance, collaboration and synergies will be explored with external organisations in similar fields of activity. This can be networks (e.g. PEER), institutions/associations (e.g. IAIA, TIAS, FUTURE EARTH, SDSN), programmes/projects (e.g. FACCE JPI, MACSUR, APRAISE, VISIONSRD4SD) and conferences (e.g. the bi-annual Berlin Conference on the human dimension of global change; and the bi-annual conference Science for the Environment).
A business plan has been developed for the setup of the CoP and its activities. During the project the initial idea to develop LIAISE as a ‘company for IA tools and services’ was abandoned. The growing insights in the ‘gap’ and the discussions with the Policy Board, lead to the conclusion that the CoP needed an ‘activity based’, rather than a ‘company based’ business plan. The general idea is to keep the structural costs to a minimum and to cover the costs of the individual services and activities (e.g. conferences, workshops, training courses) by financial contributions from individual participants and from various sources of funding that are directly related with the specific activities. Priorities for getting the CoP going are: the establishment of a policy forum for exchange between researchers and practitioners in the field of IA for SD; the development of a shared research agenda; the provision of a web-platform providing standardized non-technical and contextualized meta-data to describe IA knowledge and tools; and the development of product and process standards to assess the quality of research on IA for SD.
List of Websites:
website: http://www.liaise-kit.eu
Contact details can be found on the website
