Skip to main content
European Commission logo print header

Assessment of Policy Impacts on Sustainability in Europe

Final Report Summary - APRAISE (Assessment of Policy Impacts on Sustainability in Europe)

Executive Summary:
APRAISE stands for “Assessment of Policy Interrelationships and Impact on Sustainability in Europe” and aims at improving environmental policymaking in support of the transition towards a sustainable European society. APRAISE evaluates EU environmental policies and their implementation on member state level and compares the intended policy results with the actual policy achievements. Most importantly, APRAISE explains why a policy may perform differently than expected and draws the relevant conclusions to improve future policy initiatives in similar areas. For this analysis, APRAISE focuses on environmental policy areas that are of key importance for a resource-efficient and environment-friendly Europe: energy, climate, agriculture, water, waste, air and biodiversity. For these areas, APRAISE explains how, based on the respective EU directives, different member states have formulated targets, policies to achieve these targets, and policy instruments for implementation of these policies. APRAISE evaluates policy results by asking three questions, henceforth referred to as the APRAISE 3E method:
1. Efficacy: What environmental policy effects were expected/anticipated in the member state in question, taking into account the best knowledge available at the time of policy design (including how policy instruments were expected to achieve these effects)?
2. Effectiveness: What have been the actual effects of the policy instruments?
3. Efficiency: Could the realised effect/impacts have been achieved with fewer resources or could a better effect/impact be achieved with the same resources?

This approach acknowledges that policies and policy instruments are not implemented in a vacuum or under laboratory conditions, but in real ‘policy systems’, e.g. a market-based society. This also implies that the effects of an environmental policy instrument (such as regulatory, economic and information-based instruments) depend on the socioeconomic and governance system within which it is implemented. The APRAISE 3E method allows for an improved understanding of how stakeholders respond to policy instruments and how this influences the implementation of policy instruments and their outcome. As a result, knowledge about environmental policy instruments can be improved, so that the eventual difference between policy expectations (based on efficacy) and actual policy outcomes (effectiveness) can be reduced.

In order to design and assess scenarios about future developments regarding the ‘policy systems’ and possible impacts on policy performance, APRAISE applies quantitative models, which can prioritise either micro or macro-economics, depending on the situation. Moreover, these models can ‘reconstruct’ the past by formulating ‘what if’ scenarios. For example, what would have been the policy effects in the absence of the economic crisis?

The APRAISE 3E method, in combination with the quantified models, has been tested within the context of six case studies carried out in a total of seven EU member states. Subsequently, the method has been improved to help EU and member state policymakers draw conclusions regarding future environmental policies. For a subset of case studies, model scenarios have been developed to anticipate policy effects assuming different economic and political futures.

To complement the conventional approach that focuses on policy targets, APRAISE emphasises the importance of processes in policy design and evaluation. For instance, some of the APRAISE case studies showed that while targets may have been achieved, there could still be inefficiencies in the policy system context or during implementation, which could compromise the achievement of future environmental policy objectives and efforts to reach a resource-efficient economy. APRAISE
therefore recommends that environmental policies should not only focus on targets but also on underlying mechanisms and processes supporting medium- to longer-term environmental objectives.

Project Context and Objectives:
The key objective of the APRAISE 3E method is to enhance policymakers’ capability to design, understand, evaluate, and implement policy instruments aimed at achieving environmental targets within the context of a broader policy and stakeholder system. Ideally, policymakers base their decisions on the best knowledge available to them at a given point in time concerning the policy environment, and the mechanisms through which policy instruments are expected to lead to the desired goals, including assumptions concerning the ways in which stakeholders respond to these instruments. These ‘upfront’ expectations, which can be based on available theory and experiences, possibly supported by models, are referred to in APRAISE as the efficacy of policy instruments. After implementation, the policy (and the associated instruments) can be evaluated by examining its actual effect: the effectiveness and efficiency of a policy.
Policymaking is complex and can be an ad hoc process. The political, economic and social policy context is difficult to control and changes in these contextual factors may influence the effectiveness and efficiency of a policy instrument (see Figure 1). Therefore, policy design and implementation (such as assumed governance procedures) may deviate from what was expected. Finally, stakeholders targeted by the policy instruments are often targeted simultaneously by several other environmental or other policies and policy instruments, which make their behaviours difficult to predict.
This diagram illustrates how the system context for policy instruments, policy implementation aspects and policy interaction through stakeholder behaviour can influence the effect or impact of policy instruments. This could lead to an outcome whereby the actual impact or effect (effectiveness) of policy instruments differs from anticipated, theory-based impacts/effects (efficacy). Please note that policymaking in reality is usually not as linear as shown in the diagram..
Figure 1: Policy making in reality – policy impact/effect influenced by contextual, implementation and policy interaction factors
In virtually all policy instrument design processes, various assumptions are made about the system context factors (context factors in the nation state, sector or policy area) that could negatively or positively affect the operation of policy instruments during the implementation stage. In a few cases, such assumptions are sufficiently understood and made explicit (e.g. expected economic growth or anticipated collaboration between stakeholders) during the design of a policy instrument. However, the assumptions often remain implicit or are unknown and are therefore not properly accounted for in the policy instrument design stage. Therefore, the detailed and specific design of an individual policy instrument includes to a certain extent a reflection on how well the context of an individual policy instrument has been taken into account before the implementation stage. As a result, the achieved policy impact or effect (effectiveness) may, in practice, differ considerably from what policy makers anticipated in the light of their efficacy knowledge.
Deviations between efficacy and effectiveness can also influence the efficiency of policies. In an optimal situation, the intended policy effect is achieved with the lowest costs possible. In reality, however, efficiency may be lower because the goal has been achieved at higher than anticipated costs, or it has not been achieved but costs have been incurred as expected. Opposite conclusions of efficiency can be drawn if the outcomes are better than anticipated. In line with the discussion on efficacy and effectiveness, the extent to which policymakers are able to design an efficient policy partly depends on their ability to correctly anticipate the impact of the context, policy implementation aspects, and stakeholder responses.
One of the key lessons from APRAISE is that a better understanding of contextual, implementation and policy interaction aspects enables policymakers to design more robust policy instruments, which implementation and operation can be adapted to (foreseen) changes in the circumstances (context). The APRAISE 3E method, in combination with the case studies in which it has been applied, offers a key tool that helps to inform policymakers about these aspects and to enhance environmental policymaking. Figure 2 illustrates this learning process.

Figure 2. Illustration of how the APRAISE 3E method helps policymakers make better informed assumptions about the efficacy of policy instruments

2.1.2. How does the APRAISE 3E method work?

The APRAISE 3E method helps policymakers to more systematically assess the anticipated effect(s) of a policy (instrument) during its design stage, by making better-informed assumptions about their contextual, implementation and stakeholder behaviour aspects (including possible interactions with other policy instruments). The method, therefore, helps to close the gap between expected/intended and achieved policy effects and impacts. At the same time, the method is applicable in any member state and lessons from these applications can be relevant for multiple policymaking levels.
The APRAISE 3E method adopts a systems approach to examine an event or a system in a holistic manner by emphasising the relationships and interactions between the system’s elements (i.e. the actors involved and the institutions governing their interrelationships). The APRAISE 3E method assesses the following groups of factors:

System context factors, such as:
• Environmental factors: The original perception of the extent of an environmental stress/pressure that motivated the design of a policy and the choice of policy instruments may change over time, thereby possibly shaping the ability of the policy or instruments to achieve their targets.
• Economic factors: An environmental policy may change the structure of an economy, supporting some sectors and penalising others. At the same time, economic developments (e.g. GDP growth, energy prices, trade conflicts, and climate policies) and institutions involved in policy implementation can influence the effect of an environmental policy.
• Social factors: Social factors include habits, customs and social attitudes. They can either oppose the implementation of a policy and related instrument (e.g. where a target is not fully accepted by society) or support its implementation (e.g. where a policy increases environmental awareness and thereby support the policy).
• Technological factors: When designing a policy, a certain technological capacity may be assumed as a precondition for achieving the policy target(s). During policy implementation, additional technological inventions and innovations may take place (either autonomously or as an unintended impact of the policy) and help achieve the policy target(s).

Policy implementation factors, such as:
• Political & social acceptance: This refers to the political and social response during policy formulation, implementation and evaluation. Key design elements of a policy instrument can generate or ease resistance of target groups in accepting a policy.
• Policy consistency with wider environmental and sustainability goals: This reflects the extent to which an environmental policy objective and a policy instrument are compatible with other policy priorities, including other sustainable development goals.
• Policy coherence: Policy coherence within APRAISE is associated with the public process, i.e. harmonisation, coordination and cooperation procedures across government departments and agencies. Coherence denotes the degree of alignment of incentives to promote sustainability, both vertically across levels of government and horizontally across different actors and issues within a given level of governance. Closely linked to policy coherence is the practical feasibility of implementation (or enforcement) of a policy. This feasibility relates to the applicability of the policy instrument, considering the national institutions (including infrastructure and human resources) and legislative framework.
Policy and stakeholder interactions are assessed in APRAISE at two levels:
1. Interactions at the policy level occur when policy targets or policy instrument design features may influence the operation or outcome of another policy or policy instrument. In the design phase, negative interactions should be avoided or at least kept within acceptable margins. Interactions are examined by assessing design features of a policy instrument such as type, activity scope, timing, etc.
2. Interactions at the stakeholder level relate to the direct and indirect impacts of policy instruments on the behaviour of stakeholders. The APRAISE 3E method uses a ‘system mapping’ tool to analyse how stakeholders compete or collaborate with each other and how this may influence their response to policy instruments. The tool also explores enabling aspects to help stakeholders modify their behaviour in response to a policy instrument.
An overview of the eight steps to be completed under the APRAISE 3E method is shown in Figure 3. After identifying (task 1) and characterising (task 2) policy instruments, the effectiveness and efficiency of these instruments are analysed and compared with the anticipated effects based on efficacy knowledge (task 3). Insights from analysis of the policy system context, policy transposition and implementation process and possible policy interactions (tasks 4-7) can then be used to improve policymakers’ knowledge of the efficacy of policy instruments (task 8).

Project Results:
INTRODUCTION

The core S&T result of the APRAISE project is the APRAISE 3E Method which helps policy makers to systematically assess possible deviations between anticipated and realised environmental policy results. As such, the APRAISE 3E method supports policymakers in making better-informed decisions about the best ways of achieving environmental objectives, in particular about the choice and implementation of relevant policy instruments. An important contribution of the APRAISE 3E method is that it encourages policymakers to focus not only on their own ‘policy silos’ (e.g. climate policy or waste), but also to consider other relevant policy sectors and subsectors. For instance, a policy to support recycling of waste for production of secondary materials could exist alongside a policy to stimulate the utilisation of waste as an energy source. The latter policy could be inconsistent with the first one, because while recycling aims to use waste for production of secondary materials (e.g. new plastics), the energy recovery policy diverts waste away from recycling, using it, instead, as a low-emission energy source.

However, aligning policies so that policy targets and objectives form a consistent package is not the only requirement for the achievement of the intended targets and goals. The APRAISE 3E method shows that, for a policy mix to be coherent, also the policy context, policy design and implementation, as well as the behaviour of targeted stakeholders need to be considered. As shown in the case studies carried out by the APRAISE project, insufficient consideration of these aspects may negatively affect the results of policies and policy instruments.

EVALUATION OF METHODOLOGIES FOR POLICY EVALUATION AND ENVIRONMENTAL IMPACT ASSESSMENTS

APRAISE has assessed new qualitative ex-post and ex-ante policy evaluation methods. As it is difficult, if not impossible, to assess policy effects without taking account of interactions of the assessed environmental policy with other policies interfering with it, APRAISE has dealt with policy performance and policy interaction in a combined method interlinking both issues systematically. This formed the basis for the APRAISE 3E Method.
As a preparation for the APRAISE 3E Method, the APRAISE project conducted a review of ex-post and ex-ante policy evaluation methods of environmental policies to characterize the specific pros and cons of each of these methods and, therewith, enable the combination of the most effective and efficient set of evaluation methods. APRAISE also reviewed, as input to the APRAISE 3E Method, sustainable development indicators, which were indicators developed both at national and supra-national, and sub-national level. Eventually, identified indicators were used as input data for specifying the policy context as well as various aspects of its performance, in the case study analysis.

The APRAISE 3E Method explains factors that affect effectiveness and sustainable development performance in environmental policy areas. For this explanation, three explanatory factors have been identified:
- Possible changes in the economic, political and social context of the policy and its instruments and how these changes have affected the performance of policy instruments.

- Possible deviations between the expected and observed cycle of policy design, implementation and evaluation and their impact on policy instrument performance.

- Possible interactions of the policy (instruments) with other policy (instruments) and how positive or negative interactions may have enhanced or weakened policy effectiveness.

With the method, a theory-based evaluation was applied in order to create a "theoretical" picture of how a policy or an policy instrument was expected to achieve its objectives and targets. Empirically, the actual implementation and operation of policy instruments were assessed, including the role of contextual factors (including environmental, economic, social, and technological factors) and interactions in particular at the stakeholder level. The APRAISE 3E method enables comparison of the operation of policy instruments in theory with the respective empirical facts verified through stakeholder consultation and assess why a policy instrument has operated or failed to do so in a given context. Then, the method can draw conclusions on the efficiency, effectiveness and efficacy of national policy instruments and EU policies. In particular, the deviation between a policy instrument’s effectiveness and the envisaged outcome (i.e. based on efficacy knowledge) could be a result of contextual factors and/or policy interactions. Insights obtained will help to improve policy design, both nationally and at the level of EU.

As the APRAISE 3E Method is based on a qualitative approach for assessing the performance of a given environmental policy instrument, it is rather limited in terms of conducting sensitivity analysis and exploring different policy futures as possible baselines for policy making. Therefore, APRAISE also developed and employed quantitative approaches of modelling scenarios, for which the impact of an environmental policy is tested with respect to meeting the respective sustainability goal(s).
For that, a variety of models were used in APRAISE, which basically served two different purposes. BSAM, together with BALMOREL, models the decision-making process of power suppliers concerning the technologies employed for producing electricity. This combined model also shows how this decision making process is influenced by relevant environmental policies. This approach thus represents the bottom-up perspective to the efficacy and efficiency of environmental policy instruments and provide us with ideas as to how many environmental technology devices are or will be installed and what their environmental effects will be. By contrast, the model GTAP in combination with VTT TIMES models the macro-economic effects arising from environmental policy instruments and the investment in the corresponding technologies induced by them. The effects include the impact of these investments on growth, employment and foreign trade. Since, due to the logic and structure of this type of model, the maximum level of detail is limited to a set of economic sectors, this group of models assumes a "social planner" perspective, looking at, and assessing the relevant policy effects from top down. Since, primarily, changes of the structures within or between economies are modelled, this part focuses more on the social and economic aspects of sustainability.

With respect to the general approach adopted in the APRAISE 3E Method, the basic contribution of the models was the ex-ante view on the effects of the respectively assessed policy instruments with regard to the adoption of the corresponding environmental technologies and the effect of the latter on ecological sustainability and economic performance. In light of efficacy, effectiveness and policy instrument efficiency, the main contribution of modelling was to assess policy (instrument) efficacy and ex-ante estimates of policy (instrument) efficiency. Accordingly, modelling elicits the effects a policy instrument is expected to exert under optimal conditions and the relation of these effects to the incurred costs.

While both bottom-up and top-down approaches are able to model the combined, cumulated effects of several policy instruments, only the former can model policy interaction endogenously – showing how one policy instrument possibly supports or inhibits another one in achieving its intended effect. And even the bottom-up approach is only able to model the effect of policy instruments affecting the development or employment of environmental technologies directly. In APRAISE, however, a broader understanding of interaction was employed, including for instance policy instruments affecting the environmental policy to be assessed, but being employed in other sectors and intended primarily to address completely different issues. Since it was impossible to model such complex interactions quantitatively, quantitative modelling was not used as stand-alone approach in APRAISE. Instead, it was used in combination with the more qualitative APRAISE 3E assessment approach. In such a more complex setting the qualitative parts contributed to the assessment of the diversity and complexity of policy interactions affecting the influence of the policy instrument to be assessed in relative terms, while the quantitative (modelling) parts contributed absolute figures and the basic trends governing them.

APPLYING THE APRAISE 3E METHOD: CASE STUDY ANALYSIS

The APRAISE 3E method has been applied to six environmental policy case studies, each for two EU member states:
• Offshore wind power and protection of marine environment in Estonia and Germany
• Supporting biofuels for transport and interactions with other environmental objectives in Austria and the UK
• Recycling of plastic packaging waste in Germany and The Netherlands
• Sustainable energy buildings in Greece and the Netherlands
• The impact of hydropower generation on river basins in Austria and Slovenia
• Synergies and trade-offs between renewable electricity production and energy efficiency promotion in the built environment in Greece and Slovenia

Below, the application of the APRAISE 3E method with the help of short descriptions of the case studies is presented. The case studies are presented in the order of the eight steps of the APRAISE 3E method.

• Case study 1: Offshore wind power and protection of marine environment in Estonia and Germany

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

In this case study the impact of the policy mix relevant for offshore wind expansion and its interaction with the marine environment is assessed in Germany (DE) and Estonia (EE). The focus of the case study is on the effectiveness of renewable energy policy instruments in both countries, and on the interactions with other policy instruments, in particular those targeting the protection of nature.

Step 2 - Description of member state policy package to implement EU directive(s)
Both countries use a very similar set of policy instruments to support offshore wind power. Germany has implemented the Renewable Energy Act, along with feed-in tariffs, and Estonia has implemented the Electricity Market Act, with a feed-in premium. In the field of environmental protection, both countries have also selected similar key policy instruments: the Environmental Impact Assessment Act and the Federal Nature Conservation Act (DE) and the Environmental Impact Assessment, Environmental Management Systems Act and the Nature Conservation Act (EE).

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments
Both countries are in quite different stages of offshore wind power development. Although offshore wind development in Germany has slower than planned, it appears to be quite successful in Germany. However, in Estonia, the plan of connecting the first offshore plants to the grid in 2016 appears to be unattainable. Environmental protection measures regarding offshore wind farms, on the other hand, have been implemented well in both countries.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions
In both countries, the performance of selected key policy instruments has been significantly impacted by different factors. In Estonia, the motivation to push offshore wind energy and support potential investors and operators decreased, when it turned out that offshore wind power is significantly more expensive than onshore wind and the country was able to achieve its renewable energy target (25%) without offshore wind. Three obstacles to move on with offshore wind investments in Estonia could be highlighted:
1. hesitation about the considerable rise of energy prices for consumers once offshore wind parks would become operational (context factor);
2. lack of an investment plan for power transmission from offshore installations (implementation factor); and
3. lack of a marine spatial plan, which was a precondition for the development of marine offshore wind parks (policy interaction factors).

The lower effectiveness of the policy mix for offshore wind in Germany to reach its ambitious offshore wind targets has been mainly due to the following context and implementation factors: grid access delays due to financial, political and technological bottlenecks as well as capital access restrictions (caused by the financial and economic crisis), and the recently diminishing political ambition level for the expansion of offshore wind and resulting political uncertainties..

Step 8 - Conclusions for policymakers
As the current renewable energy system is not very motivating for offshore wind energy in Estonia, new renewable energy targets and ceilings are necessary. This is also true for transmission networks, which are currently insufficient to support the introduction of offshore wind energy production in Estonia. However, beside these difficulties, the assumed negative interaction between offshore wind expansion and biodiversity policy has been identified as rather minor. A similar conclusion about interactions between offshore wind expansion and biodiversity concerns can be drawn for Germany.

• Case study 2: Supporting biofuels for transport and interactions with other environmental objectives in Austria and the UK

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

Biofuels are intended to reduce GHG emissions in the transport sector. Against this background, the EU has agreed on a target of 10% renewable energy share in the transport sector by 2020. EU member states use different policy instruments to achieve this target, taking into consideration country-specific characteristics and conditions. However, the promotion of biofuels potentially leads to interrelations with aspects related to biodiversity, water body protection and waste reduction. This case study focuses on these interactions in Austria (AT) and in the United Kingdom (UK).

Step 2 - Description of member state policy package to implement EU directives

With respect to Renewable Energy Directive (EU RED), both countries implemented policy instruments for biofuels (8.45% biofuel target in Austria and 4.7% in the UK): Austria preferred a command & control measure (Fuel Decree and Decree Regarding Agricultural Outputs for Biofuels) and the UK a market-based system supported by a quota for key biofuels (Renewable Transport Fuel Obligation (RTFO) and Motor Fuel and Merchant Shipping Regulations (MFMS)). Additionally, the EU RED requires that biofuels also need to guarantee a certain amount of greenhouse gas (GHG) emissions savings and meet various sustainability criteria. Similarly, the Fuel Quality Directive (EU FQD) indicates that biofuels must meet specified environmental standards. Under the Energy Taxation Directive, both countries introduced different tax rates (or reliefs from taxes) for biofuels and fossil fuels at certain points in time. Additionally in Austria, the Decree for Bioethanol Mix defines a partial tax refund for certain ethanol blends. In the UK, also a separate policy instrument for specific types of biofuels has been set at the national level: biodiesel from waste products falls under a broader Environmental Permitting Regulation and require biofuel producers to obtain the necessary permits for biofuel production.

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments

In neither of the countries is the fixed biofuel targets for biofuel supply (see step 1) likely to be met, as in recent years the growth of biofuel shares has slowed down considerably.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions

Regarding policy context factors, the discussion about indirect land use changes (ILUC), connected with availability of land for cultivating biofuel feedstocks, and subsequent limitations to first-generation biofuels (e.g. assumed lower GHG reduction potential) has had a negative impact on the effectiveness and efficiency of the chosen policy instruments in both countries. Other aspects (the perceived technical limitations concerning ethanol or biodiesel fuel mixtures, taxes on biofuels and imported biofuels feedstock) are country specific. Uncertainties surrounding ILUC have also temporarily placed a cap on biofuels in the UK and targets are unlikely to change until major sustainability issues have been addressed. Similarly, fluctuating conditions (e.g. prices for renewable energy certificates, tax incentives) have had negative impacts on the success of biofuels. Concern about energy security is a positive driving force for the UK biofuel sector, which promotes domestic fuel production. Regarding implementation factors, in Austria, the failure to introduce E10 (10% ethanol content) has had negative impacts, whereas the national administrative framework and coordination among institutions have been beneficial.

Step 8 - Conclusions for policymakers

External contextual factors in both Austria and the UK limit the expansion of first generation biofuels and second-generation biofuels are not likely to make a sizeable contribution to meeting 2020 targets, although in the UK there is growing investment and R&D on this. Thus, the fixed biofuel targets in both countries are not likely to be met. The command and control approach connected with tax reliefs in Austria appears to be more effective in meeting national biofuel targets compared to the market based instruments in the UK.

• Case study 3: Recycling of plastic packaging waste in Germany and the Netherlands

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

Around half of EU household plastic waste is still landfilled. This causes both environmental and health problems, as well as non-recovery of the material and energy contained in plastic waste. This case study focuses on the management of household plastic packaging waste in Germany (DE) and the Netherlands (NL).

Step 2 - Description of member state policy package to implement EU directives

Based on relevant EU directives (Table 3), the Netherlands introduced a Packaging Decision in 2006. It contained a 42% recycling target (by 2012) for plastic packaging material for products supplied on the Dutch market. In Germany, since 1999 at least 60% of plastic packaging materials have to be recovered, of which 60% have to be recycled (i.e. 36% recycling of total plastic packaging materials). In both countries suppliers/producers are released from their take-back and recovery obligation. While in the Dutch case municipalities play a key role in this release, in Germany the Duales System Deutschland (DSD) was established by the industry, which operates parallel to the public waste management services. To discourage the use of plastics for packaging, the Netherlands applied a packaging tax.

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments

Recycling reports in both countries turned out to be better than expected: 48% in the Netherlands (2012; target 42%) and 49% in Germany (2010; target 36%). However, German stakeholders indicated that the minimum recycling quota of 36% is probably under ambitious. Regarding the Netherlands, it remains unclear whether the 48% reported recycling has actually been achieved. Stakeholder consultation indicated that this percentage only reports “collection and preparation of plastic waste for recycling”. After collection, recycling companies (mainly German) can still decide to choose another option for recovery (e. g. thermal recovery or incineration), e.g. for cost reasons.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions

In both countries, achieving plastic recycling targets has been supported by technological progress and availability of appropriate technologies. Moreover, in Germany high oil prices (before 2008) gave an incentive to producers to switch to recycled plastics, whereas in the Netherlands increasing environmental awareness supported separating plastics from household waste for recycling. Negative impacts were caused by unfavourable economic development (especially in NL as the Packaging Decision was implemented during the economic crisis after 2008) and the increasing use of composite materials (plastics, glue, colours, etc.).
In Germany, negative interactions occurred between recycling policies and policies supporting waste incineration and supporting the use of plastic waste as an energy source. This reduces incentives for recycling. For Dutch plastic waste, it is likely that a similar negative interaction has occurred as most of the collected plastics in the Netherlands is transported for recycling to Germany. In the Netherlands, plastic waste collection has furthermore been supported by local tax differentiation, which creates an incentives for increasing separating plastics from household waste.

Step 8 - Conclusions for policymakers

Due to the different sets of policy instruments used in both countries to transpose the Waste Directive the impact of contextual factors on achieving recycling goals has been quite different. Both countries have probably had relatively easy recycling targets, which provided few incentives to accelerate recycling. Moreover, in both countries monitoring is insufficient to make clear which part of the plastic waste collected is recycled or otherwise used.

• Case Study 4: Sustainable energy buildings in Greece and the Netherlands

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

Buildings account for 42% of final EU energy demand and for 35% of total EU GHG emissions. Existing policy initiatives oriented towards the sustainability of the building sector have largely targeted energy efficiency. The focus of this case study is on whether policies promoting energy efficiency interventions in the building sector (insulation, air conditioners, heat pumps etc.) could have a negative impact on the environment due to the higher generation of particular waste streams. The case study looks at Greece and the Netherlands.

Target Sets minimum requirements on the energy performance of new and existing buildings undergoing major renovation Provides indicative energy saving targets and obligations regarding energy savings and energy efficient procurement Established an energy consumption labelling scheme for, a.o. white goods, lights and cars Provides for the creation of collection schemes where consumers return their WEEE free of charge

Step 2 - Description of member state policy package to implement EU directives

In Greece, two key market-based policy instruments were implemented to promote energy savings in the built environment (‘Energy Savings in Households (ESH)’ and ‘Energy Efficiency Programme’, for Municipalities). Furthermore, three regulatory instruments were implemented to establish minimum requirements for energy efficiency in buildings and to promote the use of energy-efficient electrical equipment. In the Netherlands, numerous policy instruments have been introduced in the form of mandatory regulations, subsidies and programs to improve the energy efficiency of the built environment. E.g. a value-added tax (VAT) reduction was offered for investments in building insulation, while a regulatory instrument was introduced in the form of Energy Performance Certificates (EPC) for buildings. Producer responsibility was extended for both countries to recycling and recovery of electrical and electronic equipment.

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments

With respect to energy savings enhancement, in Greece, the effectiveness of the analysed policy instruments appears to be quite low. Nevertheless, the interim targets set for 2010 (5,1 TWh) have been met, which has been largely due to the impact of the economic recession on final energy consumption. The ambitious ‘Energy Efficiency’ Programme (with 30-40% energy saving targets) did not match expectations in terms of participation levels or progress in the implementation of energy efficiency projects.

In the Netherlands, despite the fact that policy instruments have been in place for a longer period, the market uptake of energy efficiency measures in the built environment shows mixed results For instance, the absence of sanctions under the Energy Performance of Buildings Directive (EPBD) has resulted in less than 20% of homes sold with an energy label. The reduced VAT rate for home insulation work programme has led to improvements in the existing buildings although there is no concrete study on the effects of this policy instrument, as it runs parallel with several other policy instruments and regulations.

Producer responsibility for waste of electrical and electronic equipment has been implemented in both countries with quite diverse effects. In Greece, the policy instrument is estimated to be effective in terms of waste collection targets achieved. In the Netherlands, electrical and electronic equipment collection target of the Waste of Electrical and Electronic Equipment Directive (WEEE) (4 kg/capita/year) was already reached in 2006.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions

The economic recession has caused a decrease in energy efficiency investments in Greece, but also in the Netherlands. This is mainly due to the high equity capital required for such interventions and the long payback periods. In addition, investments were negatively affected by, e.g. negative sentiment of the public media in combination with the frequent changes and revisions of EPCs in the Netherlands and insufficient information and limited consumer awareness in Greece. In both countries, technology for energy efficiency in buildings is available, as well as skilled professionals and consultants. Nevertheless, in the Netherlands a major bottleneck in the issuing of EPCs was the lack of standard training for assessors, which has also delayed the implementation of the EPBD. In Greece, lack of clarity of building ownership in urban environments in Greece often negatively affected the performance of the policy instruments.
Regarding policy implementation, in Greece, municipalities often could not generate the required 30% of co-funding to receive 70% funding support from the Energy Efficiency Programme. In addition, low technical capability of the municipalities’ staff and strict evaluation criteria from the banks during the loan approval process delayed project uptake and implementation. In the Netherlands, lack of transparency, reliability, accuracy of the prescribed methodology and frequent changes in the policy framework all contributed to the delayed implementation of the EPBD.

In Greece, recent national policy instruments providing financial incentives to end-users for energy efficiency investments in the building sector have interacted with each other, even though they target different sectors in the built environment. Within the policy mix there are overlaps and inconsistencies between policy instruments. A possible positive interaction could take place between policy instruments providing financial incentives and those with mandatory requirements (such as the EPBD). Such an integrated scheme guarantees that a target is achieved, while it also allows some degree of flexibility with its voluntary element. However, inconsistencies in the policy framework, as observed in the Netherlands, can create serious barriers to achieving the targets even if the linked financial incentives and mandatory requirements have similar objectives and targeted stakeholder groups.

Step 8 - Conclusions for policymakers

In Greece, and to a lesser extent in the Netherlands, the broader unfavourable investment environment post-crisis has negatively affected the performance of policies. At the same time, fuel poverty and escalating energy costs, along with the constrained construction activity supported energy efficiency upgrade efforts in buildings. The Greek experience has shown that delays in processing energy efficiency proposals can strongly impede policy success, while the Dutch example of mandatory (but not enforced) EPCs in buildings has not created a large-scale adoption by house owners of these labels. In both countries, there have been inconsistencies and overlaps between policy instruments targeting similar energy efficiency goals. In some cases, voluntary and mandatory measures can coexist well, whereby stakeholders have the freedom to adopt energy-efficiency options provided that they comply with the energy-efficiency requirements.

• Case Study 5: The impact of hydropower generation on river basins in Austria and Slovenia

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

This case study examines the performance of the national environmental policy mix regarding hydropower decision making in Austria (AT) and Slovenia (SI). It focuses on possible interactions between nature/water protection and renewable energy expansion targets. The case study considers small- and mid-sized hydropower in Austria (capacity ≤20MW) and small hydropower in Slovenia (capacity ≤ 10MW), with specific examples in each country.

Step 2 - Description of member state policy package to implement EU directive(s)

The key policy instruments for hydropower permission are the National Water Act (AT) and the Act on Waters (SI), both of which implement the EU WFD and pursue the water protection target. EU RED is implemented in both countries with the Green Electricity Act (AT) and the Energy Act (SI). These instruments pursue the expansion of renewables via subsidy support. Additionally, both countries have policy instruments for achieving nature protection.

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments

In both countries, the achievement of EU WFD targets concerning hydropower generation is not on track. Frequently, exemptions are allowed and, currently, only a minor part of water bodies have demonstrated improved performance. With their current expansion track, both Austria and Slovenia face problems to meet interim or 2020 hydropower expansion targets. In Slovenia, despite recent hydropower investment, current expansion level towards for targeted projections is generally too low.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions

In both countries, there have been interactions between the policy instruments, such as frequent exemptions from the EU WFD target for granting hydropower permissions. This ‘hydropower conflict’ is reinforced by a range of contextual factors. In Austria, there has been both a strong political focus on hydropower expansion (esp. in Styria) and an increased public awareness of biodiversity. This has complicated coordination across institutions when implementing EU WFD and EU RED, resulting in misinterpretations about EU WFD implementation. This has already resulted in EU infringement procedures regarding the permission of a specific hydropower example in Austria.
In Slovenia, a ‘hydropower conflict’ was avoided due to long approval procedures and a heavy administrative burden regarding hydropower permissions. Implementation of EU WFD and nature conservation legislation is supported by increasing public awareness of biodiversity and a decreasing motivation to invest in hydropower. Moreover, due to the financial crisis since 2008, less funding has been available for hydropower investments. In Austria, also existing national property rights have been a problem for the implementation of the EU WFD due to long duration of permits. Finally, low electricity prices currently slow down the development of hydropower expansion in both countries.

Step 8 - Conclusions for policymakers

The ‘hydropower conflict’ (i.e. exemptions from the EU WFD to enable hydropower permissions) is more significant in Austria than in Slovenia. In Slovenia, nature (water) protection is on a better track, although the overall desired outcome is not being achieved. More EU level guidance on how to handle possible (negative) policy interactions at the national level could avoid possible (‘hydro’) conflicts. This would also create more certainty for investors who, in both countries, could currently become the subject of ex post prosecution of offenses against EU legislation.

• Case study 6: Synergies and trade-offs between renewable electricity production and energy efficiency promotion in the built environment in Greece and Slovenia

Step 1 - Identification of the environmental policy area or sector and relevant EU directives

The focus of this case study is on the interaction between policy instruments aimed at promoting electricity produced from renewables (RES-E) with policy instruments aimed at increasing energy efficiency in the buildings sector. For Slovenia, the main focus is on Solar Photovoltaic plants (solar PV) in the built environment and for Greece the focus is on promoting RES both in the electricity production (i.e. RES-E) and end-use in built environment (i.e. RES-E and RES-heating and cooling applications).

Step 2 - Description of member state policy package to implement EU directive(s)

Both countries implemented Feed In Tarriffs (FiT) as their support scheme for RES-E promotion, but use different policy instruments for the promotion of energy savings. In Greece, energy efficiency policy instruments are grants and subsidies, supporting the uptake of efficient end-use interventions in municipalities and households respectively. In Slovenia, the policy mix for promoting energy end-use savings is largely based on regulations rather than market-based schemes.

Step 3 - Expected (efficacy) and achieved (effectiveness) policy effects of policy instruments

Both countries are on track to meet their RES targets in 2020. In Greece, installed RES capacity nearly doubled between 2011 and 2013, which is mainly due to the exponential solar PV growth (growth of other RES technologies has been negligible). Also in Slovenia, the new FiTs of 2009 especially stimulated solar PV. As a result, RES support costs increased in both countries. At the same time, the energy efficiency market in both countries is still relatively new and underdeveloped. Amongst others, this is due to the attractiveness of FiT for solar PV systems in buildings, which has overshadowed investments in energy end-use reductions in buildings.

Steps 4, 5, 6, 7 - Impact of economic, environmental, social and political developments; policy instruments design and implementation cycle; policy instrument interactions

In both countries, RES and energy saving promotion have been affected by different factors. A first reason is that the phase out of FiTs for solar PVs, due to sharply declining equipment costs, has been slower in Greece and Slovenia than in other member states. Also, the phase out was much slower than the steep drop in PV module costs. This seriously reduced the efficiency of FiT schemes and raised concerns about their long-term financial feasibility. Interestingly, observed efficiency of FiT schemes in Greece and Slovenia has been similar.

Secondly, it was concluded that the energy and climate policy instrument mix in Greece and Slovenia is largely consistent, both at the policy and stakeholder level. In Greece, policy coherence issues mainly relate to the large number of government institutes with a role in the evaluation and authorization of RES projects, which negatively affects uniformity of policy implementation. In Slovenia, lack of coordination and standardisation of procedures is identified across the five different grid operators that are responsible for connecting PV plants to the grid.

Step 8 - Conclusions for policymakers

For enhanced effectiveness and efficiency of the Greek and the Slovenian RES support Fit schemes, the case study analysis has made clear that there is a need for a long-term, more visionary and adaptive policy design with reformulated targets and a more coherent policy strategy.

COMPLEMENTARITY OF QUANTITATIVE TOOLS AND APRAISE 3E METHOD

As demonstrated by the case studies, the APRAISE 3E method helps to understand past and current system contexts for policy instruments. This helps to explain why an observed policy effect (effectiveness) deviates from the anticipated effect (efficacy). However, the APRAISE 3E method is less suitable for making predictions about the future and for formulating scenarios for policy context factors under different assumptions. Therefore, APRAISE has analysed how quantitative modelling tools can translate the lessons from the APRAISE 3E method into recommendations for future policymaking with the help of consistent scenarios.
Ideally, a set of appropriate quantitative tools is selected and scenarios developed after the problem and context are defined. This allows the application of methods on a case-by-case basis, fitting the nature and scope of the policy instruments and their system context. In order to test this, two models have been applied in the APRAISE case studies:

• the Global Trade Analysis Project – GTAP, which is a global general equilibrium model contributing mostly to understanding influences from outside the system (e.g. policies in other countries and other sectors), and
• the Business Strategy Assessment Model – BSAM, which focuses specifically on national-level power sectors, such as the Greek electricity market covered by the APRAISE case study analysis, and which supports microeconomic modelling of economic actors’ interests and decision-making process.

The models have been used to formulate scenarios for policy context factors under different assumptions, both with respect to the future (e.g. ‘what will happen if?’) and the past (e.g. ‘what if there had not been an economic recession?’). With these scenarios, quantitative estimates can be made of the influence of system-context factors (e.g. GDP development, technological progress) on the effect of policy instruments. Such estimates can be made with respect to both immediate and long-term effects (persistence) of policy instruments on the targets and other sectors (including unintended effects). For example, a change in the external conditions may render an initially successful policy measure non-influential or even detrimental.
Below, the application of GTAP to the German-Dutch case study on plastics recycling is illustrated with a particular focus on the effect of a packaging tax. Next, with help of BSAM, an ex post analysis was conducted of the effectiveness and efficiency of the financial and regulatory framework for support of renewable energy-based electricity production in Greece.

• Model-based scenario on impacts of packaging tax in the Netherlands and Germany
With the help of the model GTAP, the impact of a packaging tax on the production and supply of food products has been analysed for the Netherlands and Germany. In both countries, this sector is the largest user of plastic packaging material. For this analysis, GTAP has prepared two scenarios for each country: in one scenario, it is assumed that the packaging tax is implemented only at the national level, while the second scenario assumes that all EU member states use a packaging tax in their packaging prevention and recycling policies.

From these scenarios it is concluded that, in the Netherlands, a national packaging tax has little impacts on domestic food production, as more food products are exported (especially in the short run). This shows that a packaging tax in the Netherlands favours exports of domestic products, as in other countries in this scenario these products are not subject to such a tax. At the same time, domestic consumption of food products decreases as these become more expensive due to the tax. This reduction in consumption is mainly reflected by reduced imports of food products, due to the limited response by foreign suppliers to a Dutch packaging tax. For instance, a multinational supplier is unlikely to change its packaging strategy on the basis of a tax introduced in one country (especially when that country has a relatively small market, such as the Netherlands). The reduced consumption and imports also seem to confirm that the packaging tax is almost entirely passed on to consumer prices. Finally, the comparison between a national Dutch packaging tax and EU-wide packaging taxation shows that a ‘plastic leakage’ through increased exports as in the case of a national tax only would have been considerably reduced with a coordinated EU policy.

The GTAP model simulation shows that a German packaging tax would have a strongly negative impact on food industry production figures (especially in the short run). This observation can be explained by the fact that the German domestic market is much more dominant in terms of food product demand than in the Netherlands, where a relatively large share of food is exported. As a result, a German packaging tax is less easily ‘leaked’ through exports, so that the impact on domestic consumption is relatively strong and production figures respond to that.

• BSAM approach for the Greek wholesale electricity market
To complement the findings of the case study on renewable energy in Greece (see section 2.3.6) with the help of BSAM economic benefits have been calculated of reduced natural gas imports due to the support of wind power generation (wind power replacing gas-based power production).. BSAM has also been used to calculate corresponding CO2 emission reductions from avoided fossil-fuel combustion.

Another functionality of BSAM is to calculate the expected profitability of renewables-based electricity (RES-E) investments and relate this to the demand for these investments. An advantage of this is that it helps to discover influences on the effectiveness of a policy instrument that are otherwise difficult to observe. For example, an unobservable influence could be how people perceive policy-induced uncertainty and delays caused by permitting and grid-connection procedures. These perceptions are difficult to measure but they can have a direct influence on the effectiveness of a policy instrument (such as a financial incentive to support RES-E).

As an example of this, with BSAM, it has been observed that although the expected profitability of rooftop PV installations remained constant in Greece during 2012-2013, significantly fewer installations were installed. This discontinuity may be explained by the fact that in 2013 a discussion began about the need to tax the revenues from rooftop PV installations (with retroactive levying). It is likely that these events increased people’s perceived risk of investing in rooftop PV and reduced their willingness to invest.
Potential Impact:
APPLICABILITY OF APRAISE 3E METHOD TO POLICY IMPACT ASSESSMENTS AND ENVIRONMENTAL TARGET SETTING

Examining unanticipated effects of policy instruments

The APRAISE 3E method focuses on the degree to which a policy has reached its intended or anticipated effects, and seeks to explain the success/failure of the policy, notably by examining the interactions of the policy with selected key policies on the one hand and the general policy context on the other. In addition, in the case studies also unintended or unanticipated effects were identified, to the extent that they were seen to influence the degree of effectiveness.

Examples of unintended effects included those that increased transparency, e.g. through sustainability certification and new, more transparent electricity billing. This greater transparency does not necessarily always operate in favour of the intended policy objectives, as demonstrated, for instance, by the increased resistance against wind power projects as the subsidies were made visible in consumer bills. There were also examples of deleterious effects from the lack of transparency, i.e. the channelling of packaging waste to incineration went undetected because of the absence of appropriate monitoring systems.

In some of the APRAISE case studies it became clear that unanticipated effects of the analysed policies were related to the stages that precede target-setting. Sometimes policy targets appeared as excessively easy to reach (e.g. Estonian wind power policy), while at other times the problem definition was inadequate, leading to effects contrary to intended policy objectives. The global modelling exercises suggested that policy targets were sometimes established without due regard to international trade, such as in the area of biofuel and packaging waste policies. These issues concern:
1. the extent to which the policy in question adequately addresses the fundamental societal problem,
2. the processes of target-setting and policy formulation (e.g. stakeholder participation, policy controversies, negotiations, and power relations). This would allow for the identification and analysis of neglected or underrated policy goals (e.g. exclusion of relevant policy actors and/or asymmetries of power between the actors involved).
3. the impacts of a policy on problem-solving in other areas (e.g. through learning, networking, and enhanced reflexivity in policymaking).

The examples from the APRAISE case studies show that a more systematic analysis of such ‘upstream’ issues would improve the understanding of the unanticipated effects of the policies under analysis. They also highlight the importance of analysing the policy process, not only as a determinant of the effectiveness and efficiency of the policy in question, but also as a source of policy impact.

The APRAISE 3E method also allows the identification of unanticipated changes in stakeholder behaviour in some case studies. Examples included resistance against wind power projects as a result of changes in knowledge, and the development of new waste management technologies in reaction to packaging waste regulations. In the case studies, a number of issues relating to the equity of policy outcomes were identified, assessed primarily in terms of burden-sharing and distribution of benefits from the analysed policies, e.g. from renewable energy support measures.

As part of future efforts to further improve the APRAISE 3E method, the unanticipated effects could be usefully addressed in a more systematic manner. This would entail distinguishing between:
• first-, second-, and third-order effects;
• effects within the target area (e.g. effects of feed-in tariffs on offshore wind power development), and in other areas (e.g. on biodiversity or on the economy); and
• desirable and undesirable effects (with the caveat that the judgement of desirability varies across stakeholder groups and according to the adopted perspective).

A strength of the APRAISE 3E method lies in its ability to address the unanticipated effects stemming from the interaction between the policies that the analyst has identified – on the basis of past theoretical and empirical evidence – as crucial for the achievement of the intended policy objectives. Based on the APRAISE lessons, steps for enhancing the APRAISE 3E method for an improved assessment of unanticipated effects include the following:
1. Identify and distinguish between the anticipated and unanticipated effects of the policy. This implies a distinction between effects anticipated by the policymakers (e.g. in documents justifying the adopted policy) and those stemming from the evaluator’s knowledge (e.g. from literature, interviews and modelling exercises).
2. Explore the implicit assumptions that underlie policymakers’ expectations concerning both anticipated and unanticipated effects (identification of the intervention theories). Focus the empirical analysis of policy implementation on the assumptions that are crucial for the realisation of expectations.
3. Examine the actual impacts of a policy, with an attempt to identify unanticipated effects, also beyond those that the policy interaction analysis would reveal (i.e. involving policies and policy areas other than those included in the policy interaction model – which usually only covers two policies).
4. Evaluate, in close collaboration with the involved stakeholders, the acceptability and desirability of the various unanticipated effects.
5. Include the analysis of unanticipated effects in the formal processes of policy/regulatory impact assessments at the EU and member state levels.

APRAISE 3E method as a tool for policy impact assessments

Policy evaluation is an important stage of a policy cycle as it helps to analyse the implementation and outcomes of a policy. At the EU level, guidelines have been established for assessments of economic, social and environmental impacts of legislative proposals formulated by the European Commission. These assessments are referred to as policy impact assessment or regulatory impact assessment.

At the level of member states, policy impact assessments are usually combined with strategic environmental assessments (SEAs), which assess environmental impacts of sectoral plans and programmes. SEAs are required for plans and programmes in the areas of agriculture, forestry, fisheries, energy, industry, transport, waste/water management, telecommunications, tourism, spatial planning and land use, which set the framework for future development consent of projects listed in the EIA Directive (2014) or which have been determined to require an assessment under the Habitats Directive (1992). The SEA Directive regulates the process, content and quality criteria of these environmental assessment.

The EU guidelines suggest that a policy impact assessment should analyse, among others:
• the effectiveness of a policy in relation to its objectives, whether policy implementation has been efficient (in terms of financial and other resources used for the achieved result),
• whether the policy and its results and impacts are consistent with overarching EU objectives, strategies and priorities, and
• whether and how the policy interacts with other policies.

Policy impact assessments can be both qualitative and quantitative, depending on the nature and significance of the impacts. In practice, qualitative assessments, which are conducted by or on behalf of policymakers, are often supported by quantitative model-based analyses. The EU guidelines for policy impact assessments do not explicitly recommend the engagement of stakeholders in impact assessments.

As the APRAISE 3E method is specifically designed for explaining why observed policy effects differ from policymakers’ anticipations and targets and may have been achieved more or less efficiently, the method could be an added valued to policy impact assessments. First, the APRAISE 3E method could form part of regulatory impact assessments, with respect to both (horizontal) consistency of a policy with other policy areas and (vertical) consistency between EU-level objectives and member state-level policies and goals. Second, a value added of the APRAISE 3E method to most policy impact assessments could be its analysis of the policy context and the impacts of policy interactions at the stakeholder level. As an illustration, Figure 10 in the APRAISE synthesis report (http://apraise.org/sites/default/files/apraise_synthesis_document_2.pdf) shows all steps of the EU Impact Assessment Guidelines and checks how the APRAISE 3E method could contribute to each step. It is suggested that the APRAISE 3E Method could contribute to the European Commission policy/regulatory impact assessment (IA) in several ways.

Impact on environmental target setting - Building further on the APRAISE 3E method

While the APRAISE 3E method starts from the conventional goal-achievement model of evaluation, using as the main performance criterion the degree to which the intended policy objectives have been achieved, it avoids two major weaknesses of such an evaluation approach: it integrates an analysis of costs (through the efficiency analysis) and considers various unanticipated impacts that a pure goal-achievement evaluation would exclude. However, the 3E method obviously leaves out or pays little attention to a number of aspects.

First, the APRAISE 3E method could integrate a more systematic analysis of the policy phases that precede target-setting, exploring the relevance of the policy objectives, the policy formulation processes (such as stakeholder participation, policy controversies, negotiations, and power relations) and constitutive effectiveness. The latter refers to the impacts of a policy on problem-solving in other areas, e.g. through learning, networking, and enhanced reflexivity in policymaking. This would also allow the exploration of problem-solving effectiveness, in other words, the relation between the ultimate policy outcomes and the societal problem to be solved.

Second, participation of relevant actors in the policy implementation could be examined systematically as a key element of process effectiveness. On the one hand, participation may, depending on the specific circumstances and the way in which participation is organised, improve policy effectiveness and efficiency, while on the other hand it constitutes a policy performance criterion on its own. Such criteria can be described as ‘democracy-related criteria’, which also include equity, transparency, and legitimacy of the policy processes.

Third, as for the policy outcomes, the APRAISE 3E method adequately covers the intended outcomes in terms of changes in actor behaviour, including the unanticipated changes. It also covers certain dimensions of equity relating to policy outcomes (esp. burden-sharing and the distribution of policy benefits). To integrate the whole spectrum of equity-related criteria, the APRAISE 3E method could be further extended to the analysis of equity in policy design and implementation processes.
Finally, experience from policy evaluation has highlighted the importance of the policy evaluation process itself as a source of influence. The APRAISE case study analysis shows that including the relevant policy actors in the implementation of the APRAISE 3E method is crucial for its success but this also constitutes a key challenge. This is all the more important given that policies can have multiple, sometimes mutually contradictory and poorly defined objectives. The prioritisation and weighting between such competing objectives as evaluation criteria is a fundamentally political process, and therefore requires the inclusion of the relevant stakeholders. Such inclusiveness would also help to ensure that a maximum of relevant perspectives are considered in the evaluation.

DISSEMINATION ACTIVITIES AND EXPLOITATION OF RESULTS

In the context of APRAISE project various illustrations have been prepared. Apart from the logo and the graphic guidelines already elaborated from M2, illustrations and diagrams have been developed and disseminated through a number of communication means, such as 8 newsletters (M18, M20, M22, M27, M31, M32, M34, M36), leaflets (M33), several announcements, web-site and social media posts etc. Furthermore, the APRAISE web-site contains a wide variety of photos and pictures from the APRAISE Workshops (M25, M32), the APRAISE Summer School (M35) and the APRAISE Final Conference (M36). All this material is available and may be reused by all stakeholders and interested parties.
During the second APRAISE project period, APRAISE presence in the social media (Facebook, LinkedIn, Twitter), already established as of M3, has been further strengthened and established through the constant update with new related information. The APRAISE Facebook account counts 100 likes, while each post reaches an average of 60 people. The APRAISE Twitter account has 20 followers, 65 tweets, while more than 40 people are being followed. Finally, the APRAISE LinkedIn account has 50 members. Moreover, the APRAISE has fully exploited the MyEUROPA platform, also creating an account (M32) and further communicating the project, its news and events to the wide variety of MyEUROPA members. APRAISE utilised the CLIMATE-L and ENERGY-L knowledge sharing tool and its wide stakeholders mailing list by posting related to APRAISE activities announcements, so as to further communicate APRAISE messages to policy makers and practitioners involved mainly in sustainable energy policy. Also, the APRAISE Flickr account is continuously being enhanced with all the photos from the APRAISE Workshops (M25, M32), events (APRAISE Summer School – M35) and the APRAISE Final Conference (M36).

More than 5 publications have been made during APRAISE implementation, while several publications are either under considerations on finding and selecting the appropriate journal for publication or are already in progress. Particularly, these publications include the following.
• Publication entitled “Supporting Decision Makers in Assessing Environmental Policy Instruments” and was elaborated by Charikleia Karakosta, Vangelis Marinakis and John Psarras (NTUA) in the Renewable Energy: Economics, Emerging Technologies and Global Practices, Andreas Poullikkas (Ed). Nova Science Publishers, Inc., ISBN: 978-1-62618-231-8, pp. 15-24 (M16).
• Publication of an abstract entitled “Assessment of Policy Interrelationships and Impacts on Sustainability in Europe - Empirical Analysis of the Energy & Climate Package” by Spyridaki Niki – Artemis (UPRC) in the Book of Abstracts of the 2nd International Symposium and 24th National Conference on Operational Research, 26-28 September 2013, Athens, Greece (M25).
• Submission of a research paper entitled “An ex-post policy assessment of the Energy Performance of Buildings Regulation in Greece & The Netherlands – A cross country comparison” by Spyridaki, N.A. Ioannou, A., Oikonomou, V., Flamos, A., and Szendrei, K., in the Energy Policy Journal and currently under the “Under review” status (M35)
• Abstract and paper publication entitled “Expanding the policy theory behind the Climate and Energy Package in Greece” by Spyridaki N.A. Ioannou, A. and Flamos A. (UPRC) within the framework of the 2014 IEPPEC - International Energy Policies & Programmes Evaluation Conference, 9-11 September 2014, Berlin, Germany (M36).
• Abstract and paper publication entitled “Efficacy and the 3 E method: a new concept in environmental policy assessment” by Tuerk A., Fruhmann C., Frieden D., Spyridaki N.A. van der Gaast W., Sartorius C., Prizlan B. and Gubina A” within the framework of the 2014 IEPPEC - International Energy Policies & Programmes Evaluation Conference, 9-11 September 2014, Berlin, Germany (M36).
• Submission of a research paper entitled “APRAISE 3E method for consistent and coherent environmental policies” by Lieu Jenny, Spyridaki, N.A Tuerk Andreas, van Der Gaast Wytze in the Journal of Environmental Policy and Planning and currently under the “Under review” status (M36).
• Research paper publication entitled “A paper trail of evaluation approaches to energy & climate policy interactions” by Spyridaki, N.-A. and Flamos, A (UPRC) in the Renewable and Sustainable Energy Reviews, 2014, Vol 40, pp 1090-1107. (In progress) (December 2014)

In addition, the APRAISE Special Issue “Assessment of Policy Interrelationships and Impacts on Sustainability EC FP7 APRAISE” will be published in the International Journal of Mitigation and Adaptation Strategies for Global Change, a peer-reviewed, inter-disciplinary international platform for disseminating results of research related to and/or relevant for the management in the energy sector, by Springer. Main objective is to ensure a better knowledge and understanding of the key issues related to the efficacy of energy policy instruments and their respective interrelationships (synergies & trade-offs) with other sustainable development policies. The important dates have been set and along with the respective call for papers, comprising the coverage themes, will be announced in the International Journal of Mitigation and Adaptation Strategies for Global Change:
• 10 January 2015: Submission Deadline (early submission recommended, referee process starts once the paper received).
• 10 April 2015: Notification of status & acceptance of paper.
• 20 June 2015: Revised manuscripts.
• 10 September 2015: Final version of paper.

Within the above framework, the APRAISE partners have prepared several research papers in order to be submitted in the APRAISE Special Issue, further contributing in the establishment of APRAISE as a reference project on energy policy issues. These publications include among others the following:
• “Assessing the efficacy, effectiveness and efficiency of energy and environmental policies: A theory based approach (Cornerstones of the 3E method)” by Lieu Jenny, Spyridaki, N.A. Tuerk Andreas, Oikonomou Vlassis.
• “The impacts of socio-economic context developments, policy design and implementation and policy interactions when assessing the effectiveness and efficiency of environmental policies” by Lieu Jenny, Spyridaki, N.A. Tuerk Andreas, Gubina Andrej, Fruhmann Claudia, Peterson Kaja.
• “Formulating consistent and coherent environmental and climate policy packages” by Lieu Jenny, Spyridaki, N.A..
• “Integrated impact assessment of EU Directives towards Sustainable Development” by Tuerk Andreas, Lehtonen Markku.
• “Unanticipated effects – explanatory factors or policy criteria in their own right?” by Lehtonen Markku, van Der Gaast Wytze.
• “Possible interactions between EU Emissions Trading Scheme and EU and Member State policies and policy instruments for renewable energy and efficiency support” by Mulder Arnold, Jepma Catrinus.
• “Assessment of APRAISE 3E Method as Decision Support Tool – comparative analysis with other DSTs” by Peterson Kaja.
• “Application of quantitative methods to complement the APRAISE 3E method” by Niemi Janne, Papadelis Sotiris, Sartorius Christian, van Der Gaast Wytze.

Moreover, during the period from M18 until M36, the APRAISE consortium has participated in various external conferences, succeeding to reach approximately 500 stakeholders and interested parties, by communicating informational material, by presenting the project, as well as by engaging in extended networking. In summary, APRAISE consortium participated in the following events:
• Annual LIAISE project meeting, 26th of March 2013, Tallinn, Estonia (M18).
• 2nd Scientific Workshop “Methodological approaches for the research on stability in multi criteria decision making issues”, 28th of June 2013, Chania, Greece (M21).
• Smart and Sustainable Cities Conference “From modeling to Practice”, 18th of July 2013, Athens, Greece (M22).
• BETTER “Bringing Europe and Third Countries closer together through renewable Energies" Meeting, 20th of September 2013, Athens, Greece (M24).
• 2nd International Symposium and 24th National Conference on Operational Research, 26 – 28th of September 2013, Athens, Greece (M24).
• 3rd BETTER Stakeholder Consultation Workshop, 25th of February 2014, Vienna, Austria (M29).
• ENSPOL - Energy Saving Policies and Energy Efficiency Obligation Scheme, 14th – 15th of April 2014, Brussels, Belgium (M31).
• 1st POLIMP Stakeholder Workshop “Exploring technology options for the EU transition to a low-carbon economy: The role of public acceptance to enhance predictability of low carbon technology investments”, 25th of April 2014, Brussels, Belgium (M31).
• 3rd European Environmental Evaluators Network Forum “Linking evaluation findings to enhancing sustainability”, 28th – 29th of April 2014, Helsinki, Finland (M31).
• 53rd Meeting of the Euro Working Group on Commodities & Financial Modelling (EWGCFM) and 2nd International Conference of the Research Center for Energy Management (RCEM), 22nd – 24th of May 2014, Chania, Creta (M32).
• 2014 IEPPEC - International Energy Policies & Programmes Evaluation Conference, 9th – 11th of September 2014, Berlin, Germany (M36).

Furthermore, even after M36 and the project completion, APRAISE consortium had the opportunity to disseminate the APRAISE results in the following external events.
• Observatoire Méditerranéen de l’Energie (OME) Renewable Energy Committee Meeting, 7th of October 2014, Paris, France.
• BETTER European Consultation Workshop, 7th – 8th of October 2014, Paris, France.
• 7th International Scientific Conference on Energy and Climate Change, 8th – 10th October 2014, Athens, Greece.
• 2nd POLIMP Stakeholders Workshop “Climate & Finance: Financing Renewable Energy for Europe”, 15th October 2014, London, UK.

During the second APRAISE project period, six APRAISE Newsletters were published. The 2nd APRAISE Newsletter was developed in M18, containing short information on the project, explaining the APRAISE progress through illustrations and diagrams, providing brief material on the 2nd APRAISE Meeting, while announcing the strong APRAISE presence in the Social Media and the forthcoming activities within the framework of the project. The 2nd APRAISE Newsletter was electronically disseminated to 850 stakeholders and interested parties, while it had 543 downloads from the APRAISE Interactive Website.

The 3rd APRAISE Newsletter was launched in M20 presenting the methodology developed to assist the explanation of various issues for a range of environmental policy case studies, depicting case studies for the countries of Greece, Austria and the Netherlands in the areas of renewable energy, energy efficiency and resource efficiency and announcing the 1st APRAISE Video. The 3rd APRAISE Newsletter was communicated via email to 850 stakeholders and interested parties, while achieved 716 downloads from the APRAISE Website.

The 4th APRAISE Newsletter was issued in M27 containing brief information on the project itself, and referring to the APRAISE case studies’ results on the Netherlands & Germany, Greece & the Netherlands, Austria & UK, Austria & Slovenia. It provided direct links to the extended versions of the produced results for all interested stakeholders and general public, while also providing information on the results of the 1st APRAISE International Stakeholders’ Workshop and announcing the organization and implementation of the 2nd APRAISE International Stakeholders’ Workshop. The 4th APRAISE Newsletter was also electronically disseminated to 850 stakeholders and interested parties, while it was downloaded 356 times through the APRAISE Website.

The 5th APRAISE Newsletter was implemented in M32 communicating the APRAISE Policy Workshop and introducing all interested parties, to the implementation of the APRAISE Summer School. The 5th APRAISE Newsletter was electronically disseminated to more than 900 related stakeholders and was downloaded 251 times from the APRAISE Interactive Website.

The 6th APRAISE Newsletter was developed and communicated in M34 providing information on the successful implementation of the APRAISE Policy Workshop, communicating the organization of the APRAISE Final Conference and the development of the APRAISE Special Issue. Finally, it entailed more detailed information on the implementation of the APRAISE Summer School. The 6th APRAISE Newsletter was disseminated by email to more than 900 stakeholders, while it has 52 downloads in the APRAISE Website.

The 7th APRAISE Newsletter was issued in M36 right after the implementation of the APRAISE Final Conference. It included information on the implementation of the APRAISE Final Conference, while it announced the implementation of the APRAISE Synthesis Document, summarizing the results obtained from the three year operation of APRAISE. Finally, the 7th APRAISE Newsletter contained information on the implementation and launch of the APRAISE Policy Instrument Database, as well as the elaboration of the 2nd APRAISE Video. The 7th APRAISE Newsletter was distributed to more than 1000 stakeholders and other interested parties, while it was downloaded 27 from the APRAISE Website.

At the same time, JIN reserved some space for promoting APRAISE activities in particular issues of its free quarterly magazine Joint Implementation Quarterly (JIQ). JIQ has a wide audience among energy and climate specialists in policy making and the business community. Within the APRAISE duration, six APRAISE articles were published in the JIQ magazine (only 3 articles where foreseen according to the DoW in JIQ issues). In particular, during the period of M18-M36 the following articles were issued:
• Joint Implementation Quarterly – July 2013 (M22). Article entitled “Effectiveness and Efficacy of Environmental Policies – APRAISE project updates and next steps. The article has 615 downloads in the APRAISE Interactive Website.
• Joint Implementation Quarterly – April 2014 (M31). Article entitled “APRAISE Policy Workshop – 23 May 2014”. The article has 224 downloads in the APRAISE Interactive Website.
• Joint Implementation Quarterly – July 2014 (M34). Article entitled “APRAISE – Policy Contexts Matter for Effective Policies”. The article has 128 downloads in the APRAISE Interactive Website.
• Joint Implementation Quarterly – October 2014 (after project). Article entitled “Understanding Policy Contexts and Stakeholder Behaviour for Consistent and Coherent Environmental Policies - A synthesis of results from the APRAISE project”.

Furthermore, the APRAISE project was disseminated through the electronic communication of a Christmas Card (M27) by JIN, which contained direct links to the APRAISE project.
Additionally, APRAISE was disseminated through articles in CEPS Newsletters. In particular, CEPS Newsletter issue of October 2014 included an article entitled “Improving environmental policy making” regarding APRAISE.
The 1st APRAISE Factsheet was released in CORDIS in M19 providing information regarding the project acronym, objectives, project summary, project beneficiaries, EU funding, etc. as well as the contact details of the project coordinator. The 2nd APRAISE Factsheet has been developed and uploaded in CORDIS in M29, including useful information on the challenge the project attempts to confront, its main objectives, the methodology followed, as well as the expected results. It also contains information on the APRAISE partners. Additionally, APRAISE “Results in Brief” document has been prepared by the CORDIS science editors with the contribution of UPRC and NTUA (M28), further contributing to the dissemination of knowledge from European research activities and supporting the exploitation of EU-funded research results in general and APRAISE results in particular.

Furthermore, APRAISE has organised two Workshops, in order to achieve the maximum stakeholder engagement, as well as information provision and promotion of the APRAISE results:
• 1st APRAISE International Stakeholders Workshop, on the 30th of October 2013, in Brussels, Belgium (M25).
• APRAISE Policy Workshop, on the 23rd of May 2014, in Brussels, Belgium (M32).
An article on APRAISE was published in the European Office of Cyprus (EOC) monthly magazine “European News”, issue 79, November 2013 (M26). The article, which was issued in both Greek and English languages, provided brief information regarding the project itself, its goals, as well as the main issues that the developed methodology attempts to address and the case studies in which the methodology was applied, therefore enhancing the visibility of APRAISE, while serving and facilitating the results’ dissemination process.

Collection of information and feedback from all partners, regarding D6.3 “Summary of results of dissemination activities”, which had initiated before M18, has been completed and the respective deliverable has been submitted in M30 as scheduled.
The APRAISE results have been integrated into the academic curriculum of NTUA and UNIPI. A presentation dedicated to the APRAISE 3E methodology and implemented case studies was included the postgraduate course “Energy Production and Management” of the NTUA(M32). In addition, it is the partners’ intention to further integrate the academic curricula with the APRAISE results, even after the project implementation. In particular the lectures to be enhanced include:
• “Energy Management and Environmental Policy” undergraduate course, in the School of Electrical and Computer Engineering of the National Technical University of Athens;
• “Management of Energy Resources” postgraduate course, in the Postgraduate Programme “Technoeconomic Systems” that is led by the School of Electrical and Computer Engineering of NTUA in cooperation with the Department of Industrial Management and Technology of the University of Piraeus (UNIPI);
• “Technoeconomics of Energy Systems” undergraduate course, in the School of Electrical and Computer Engineering, in the Department of Industrial Management and Technology of the University of Piraeus (UNIPI).

Five posters have been implemented within the framework of the APRAISE operation during the period from M18 to M36. One poster was implemented to disseminate the APRAISE Policy Workshop (M32) providing relevant information on the date and location of the event, two posters were elaborated for the dissemination of the APRAISE Summer School (M35), while two more posters were developed for the communication of the APRAISE Final Conference (M36).
The 2nd APRAISE Leaflet has been developed in M33 in order to disseminate the results of the APRAISE Case Studies. A completely new format was elaborated, while brief, yet detailed information was provided. Particularly, apart from including information on the project partners and means of communication, it offers the highlights of the case studies results. The 2nd APRAISE Leaflet has been disseminated to more than 150 interested parties.
The APRAISE Summer School has been implemented on 25–29 August 2014, in Ljubljana, Slovenia by the University of Ljubljana (M35). Main objective was to teach students about policy making for a better environment, familiarizing them with the experience, ideas and knowledge of the APRAISE project and teaching them on assessing environmental policy instruments individually and in conjunction with each other, focusing on enhanced effectiveness and efficiency of these instruments. Therefore, the Summer School programme included an integrated set of lectures, field trips, student projects and workshops, as well as familiarization with Slovenia. The dissemination of the APRAISE Summer School comprised the implementation of Newsletters (M32, M34), announcements in the APRAISE Interactive Website and social media, in the partners’ websites, and invitations. Furthermore, within the framework of the APRAISE Summer School, the APRAISE Summer School Synthesis Document was developed (M35) and disseminated to all the participating students, as well as in the APRAISE Interactive Website. The particular document assisted participating students to understand and become familiar with the APRAISE 3E method and the applications in the case studies.

The APRAISE Final Conference “What Role for Targets in EU Climate and Energy Policy? – Lessons from the APRAISE Project for Improved Environmental Policy Making” was successfully implemented on 24 September 2014, in Brussels, Belgium (M36). Main objective of the APRAISE Final Conference was to draw on the results of the APRAISE project to assess how past experience in environmental policy making can contribute to more informed decisions for the future. Additionally, the event highlighted that climate and energy policies are not formulated within an isolated policy context, but rather implemented in existing (market) systems, while drawing on the APRAISE case studies in the area of renewable energy and energy efficiency policies, providing for a better understanding of the mechanisms of environmental policy. More than 230 stakeholders expressed direct interest (through e-mail) in the event, with over 100 participants at the conference; more than 500 leaflets and synthesis documents were distributed to interested parties. The high importance of the event was underlined through the strong, undertaken dissemination activities, such as invitation letters, announcements in the Interactive website, the social media and the partners’ websites, communication of 2 newsletters and the implementation of two posters.

In addition, D6.4 Conclusions from the Final Conference has been delivered in M36, as planned and is expected to reach more than 1,000 report downloads within the following months.
The 1st APRAISE Policy Brief “Accounting for Unanticipated Effects of Environmental Policy Making” has been developed in M36, by CEPS with the cooperation of all project partners and was available before the APRAISE final conference. The 1st APRAISE Policy Brief provides means towards the extension of the APRAISE 3E Method, as the method could benefit if the unanticipated effects which occurred during the implementation of the case studies, were examined more systematically. Its dissemination included a Press release, which was sent out after the APRAISE official duration (October 2014) in order to further enhance the project’s sustainability, announcements in the APRAISE Interactive Website and social media, as well as an announcement to the ENERGY–L stakeholders list, which was also implemented after the APRAISE official duration. Even after M36, the consortium, willing to further underline the importance of the APRAISE results, is preparing the 2nd APRAISE Policy Brief, which is going to be communicated through various dissemination activities.

The 2nd APRAISE Video was developed in M36 and is available at both the APRAISE Website and YouTube. It presents the results obtained from the implemented case studies, therefore further expanding the dissemination and communication targets.
The APRAISE Synthesis Document, major outcome of WP5, was elaborated in M36 and received strong dissemination activities. These included Newsletters (M36), announcements in the APRAISE Interactive Website, as well as a Press Release sent out after the APRAISE official duration, in order to further enhance the dissemination process and the project sustainability (October 2014).

An APRAISE Press Release was sent out to more than 1000 stakeholders, in order to disseminate the APRAISE Policy Brief and the APRAISE Synthesis Document (major outcome of WP5) (M36). The Press release was sent in October 2014 (after the APRAISE official duration) providing direct links and further contributing to the project’s visibility and sustainability.
An interview was implemented, in October 2014, to APRAISE partners, regarding the APRAISE project and its results, as well as its Final Conference. The interview was announced in the Energy Valley website, further promoting APRAISE to relevant stakeholders.

Two APRAISE Announcements to the CLIMATE-l and ENERGY–L stakeholders list will be sent out, in order to disseminate the implementation of the APRAISE Policy Brief (M36) and the APRAISE Synthesis Document (M36) respectively. The Announcements, which will be sent electronically, via ENERGY –L, in October - November 2014, after the APRAISE official duration will proudly, communicate the APRAISE Policy Brief and the APRAISE Synthesis Documents providing direct links.

List of Websites:
http://apraise.org

Coordinator contact:
JIN
Laan Corpus den Hoorn 300
9728 JT Groningen
+31 50 5248430
e-mail: jin@jiqweb.org