Final Report Summary - REPOPA (REsearch into POlicy to enhance Physical Activity)
Executive Summary:
The EC funded REPOPA project found practical ways to support policymakers in their use of research evidence in developing physical activity policies. Simulation policy game, locally tailored policymaking interventions, indicators to assess policies’ evidence-base as well as national internet platforms are examples of the REPOPA products.
REPOPA stands for REsearch into POlicy to enhance Physical Activity; a five year project (October 2011-September 2016, www.repopa.eu). The project was funded by European Union, 7th Framework Program. Researchers from six EU countries of Denmark, Finland, Italy, the Netherlands, Romania, UK, plus Canada were project partners.
Why do we need integration of research and policymaking? Researchers, European Commission, World Health Organization and others provide recommendations to policymakers to take physical activity seriously and integrate it in everyday lives of people. This is more easily said than done! Policymakers juggle with numerous pressures, points of views and resources. Research evidence presented in scientific publications is not the primary driver of their work; instead, political pressures, local priorities, values and resources influence their decisions. REPOPA researchers took the challenge to work hand-in-hand with real-world policymakers to learn how best to co-create policies so that research evidence is taken into account. In other words, REPOPA used the evidence-informed policymaking approach, meaning that research evidence is NOT the primary driver of decision making; instead local contexts, resources and needs are taken into account as well.
REPOPA found several issues researchers can help policymakers with: available and applicable research when needed and getting rid of academic jargon. More should be done to build procedures to gather research knowledge and to have meetings with researchers. Games are nowadays an attractive way to learn: simulation policy game proved to increase mutual understanding between policymakers and researchers; it also showed the importance of organizational and leadership collaboration across sectors. The practical outcome here is a model for policy games which is already applied outside REPOPA.
REPOPA researchers also joined real world physical activity policy groups and helped them use research evidence. These locally tailored interventions helped policymakers to take into account needs and values of the people whom the policies concern. They also triggered politicians to request for more research evidence. Here the practical outcomes are best practice interventions in three countries to co-create evidence-informed policies locally.
REPOPA was not only an academic project; it also developed the findings into usable tools such as indicators for evidence-informed policymaking. These indicators function as a checklist in policymaking to assess how well the policy in question is actually evidence-informed. The project also built a web-based umbrella platform with country platforms for evidence-informed policymaking in physical activity. These platforms bring together different actors and sectors in the countries to network and collaborate in combining research evidence and real-life policymaking. REPOPA used physical activity as the theme in its work; however, the lessons learned and tools developed can be applied to other fields, especially in the area of healthy living.
Project Context and Objectives:
The aim
The Research into Policy to enhance Physical Activity project (REPOPA; www.repopa.eu) had the main aim to integrate scientific research knowledge (=research evidence), expert know-how and the real world policymaking processes. This was done to increase synergy, build sustainable structures and best practices for future health promotion and disease prevention among Europeans.
There is strong evidence on health benefits of regular physical activity and how different physical environments influence active living (Reis et al 2016). Using research evidence can also help to build effective policies. However, there is still a need to get more knowledge on how to integrate research evidence in health enhancing physical activity (HEPA) policies. HEPA policies can vary from direct physical activity policies to those creating environments, which support physical activity.
Evidence-informed instead of evidence-based
When the REPOPA proposal was originally written in 2010, health promotion literature was debating if the clinical medicine term ‘evidence-based’, mostly dealing with individual patients, would fit the health promotion work, since the latter works on equal footing with local stakeholders with their priorities and values (e.g. Aro et al 2008). Evidence-informed policymaking has been suggested for the health promotion (and public health field), since in that work ‘research evidence only informs’ the decisions. This means that research evidence is NOT the primary driver of decision making in health promotion; instead adaptations to local contexts, resources and needs are needed and used (Brownson et al 2009, Ciliska et al,, Graham et al. 2006, Bowen, Zwi 2005). REPOPA took the term ‘evidence-informed’ to describe the approach in its integration of scientific research, expert know-how and real world policymaking (Aro et al 2016).
In line with the above, the theoretical frameworks guiding the REPOPA work have been those up-to-date in 2010 for this kind of work. In the course of the project, further theoretical development has happened, especially in more political science field (Borra’s 2011, Delborne et al 2013, Fischer et al 2014, Wehrens et al 2011) and especially in network analysis (de Leeuw et al 2014). These newer frameworks have informed e.g. article writing of the REPOPA findings. REPOPA had an objective to build a new framework to guide the development and use of indicators to assess if and how evidence-informed decisions or policies are. This new REPOPA indicator framework builds both on previous frameworks and on the work completed throughout REPOPA but especially in the work package No4 dealing with indicator development via Delphi process.
One of the central frameworks in REPOPA has been Evidence-informed Public Health (adjusted from Satterfield et al 2009, Figure 1, Annex) describing how best evidence-informed decisions happen when the best available research evidence, characteristics of the local population and context, including local priorities, resources, know-how and values, all are taken into account, and further, when all this happens respecting the cultural and environmental context. The other frameworks include Knowledge-to Action Cycle (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012), Figure 2, Annex), Stages of Knowledge Use (Knott, Wildawsky 1980, Landry et al 2003) with its seven steps of knowledge use have informed REPOPA, Systems Approach (Best, Holmes 2010) showing that policymaking is seldom a linear process; instead several environmental and social issues make the process more ‘messy’, even circular. Also Organizational Change theories (e.g. Weiner 2009) helped especially building the interventions in work package No2 on policy gaming to increase cross-sectoral collaboration. Though not an actual framework, Nuffield Council (2007) contribution on public health ethics related to Stewardship approach, has been very central for the whole REPOPA, but especially so for work package No3, which carried out locally tailored interventions on real world policies.
Why programmatic/phased research and what it meant
REPOPA applied so called programmatic or phased research design. This means that the whole research process with its protocols for interventions in successive work packages was NOT planned in detail beforehand (Aro et al 2016). On the opposite, each research phase and specific interventions were tailored to the local needs and knowledge gathered; first, knowledge gathered at the baseline, meaning beginning of the project, and then the knowledge of each phase informed the successive phases. This differs from the traditional, controlled, pre-agreed research design in the medical field, where designs, methods and interventions are registered (and often published) before starting the research project. In the evidence-informed work, interventions and tools are tailored to the context; thus context is not controlled for; it is an essential part of the interventions.
Programmatic research design was chosen since it enables the research process to be sensitive to each phase and context; thus all actions were tailored to the situation to gain maximal fit and maximize feasibility. The programmatic design also increased opportunities for REPOPA to create research synergies (Simonsen et al 2010) that added value to the project. On the other hand, this meant that there was limited time between different project phases to plan the next steps; for the researchers involved this meant that they were in parallel involved in wrapping up one phase and building already the next one (Hämäläinen, Aro 2015).
The REPOPA Consortium
The REPOPA Consortium brought together scientific researchers and experts from different disciplines, policymakers and other stakeholders from different societal sectors and from a variety of countries. Some of the REPOPA countries had longer history of working across sectors in health promotion, such as Finland, UK and the Netherlands; some countries had recently started this work (Denmark, Italy); in some, rather top-down traditional centralized system prevailed (Romania). Canada was very likely the most advanced in its explicit knowledge-transfer culture especially in the health care area; Canada was given the evaluator role in the REPOPA project. The structure of the Consortium per se enabled knowledge transfer between the partners; it also greatly added to understanding the importance and influence of context in the REPOPA research and tool development.
Objectives of REPOPA
The objectives of REPOPA were built so that in the course of five years in the varying contexts of six European countries we could a) analyze first the use of evidence in physical activity policies, b) then test different interventions to enhance evidence-informed policymaking, and c) based on these, to develop both tools for later use by policymakers and researchers such as indicators to assess if and how policies are evidence-informed, and policy briefs. So the idea was to deliver a full package of knowledge and practical tools to apply the knowledge (Aro et al 2016).
1st phase
In the first project year, work package No1 had the objectives to explore and analyze the ways evidence was being used in formulating the local, regional and national policies to enhance physical activity (Figure 3, Annex). Further, the objective was to identify the needs of the policymakers for the use of the evidence (facilitators and barriers) and whether these needs were being met by the research community. The third objective was to evaluate ways of delivering and communicating the evidence to the policymakers; and the fourth, to develop both a framework and indicators for the evidence informed policy development.
2nd phase
The 2nd phase of the project developed two sets of interventions to be tested each in three real world policy cases in three countries. One of the sets (work package No2) had the objectives to develop and use a policy game to study a) organizational and process aspects in cross-sector physical activity policymaking, more specifically, in leadership between organizations; b) organizational networks; c) communication and organizational change processes; and d) facilitators and barriers in evidence informed policy making.
The other set of interventions (work package No3) had the objective to study if the Stewardship approach increased the level of evidence-informed policymaking in selected, real-world policy cases. Stewardship approach was based on the needs, context and stakeholder analysis, and the aim was to increase integration of research evidence with other kinds of evidence in close academia-policy maker collaboration. Further, these interventions also aimed at finding ways to identify and effectively engage specific stakeholders in physical activity policymaking.
3rd phase
Building on work packages No 1-3, the 3rd phase of REPOPA (work package No4) had the objective to draw together the REPOPA results, developed methods, frameworks, indicators and best practices in the integration of research evidence and policymaking in physical activity. Further, the aim was to test, validate and negotiate the results among the stakeholders in REPOPA countries and beyond. Finally there was the aim to develop the output into guidance resources and evidence briefs.
Non-research objectives throughout the project period
Parallel with the four research work packages, REPOPA also had a clearly stated objective (work package No5) to monitor and evaluate the project working process, content, scientific products and added value of each work component as well as the synergies among work package component. Additionally, one aim was to apply a framework to analyze the impact of the project in terms of Reach, Effectiveness, Adoption, Implementation and Maintenance of the project activities (Green, Glasgow 2006). Also parallel with the other work packages a dissemination strategy was built with (work package No6) a detailed external communication plan aiming at ensuring the best visibility of the REPOPA activities and outcomes at EU level, targeting stakeholders and wider public, raising interest and awareness of REPOPA activities and results. Further, it had objectives to establish an umbrella platform to support country-specific evidence-informed policymaking platforms created in the course of the project, and also to develop a European advocacy plan and support producing policy briefs to enhance evidence-informed physical activity policymaking. Finally, the administrative objective (work package No7) was the Consortium management and overall coordination.
Project Results:
REPOPA results phase 1
The first year results in the REPOPA project confirmed that research evidence was not systematically or explicitly used in physical activity policymaking; it was other kinds of information and priorities, which were more central. In many cases it was not possible to identify the original research sources as they were mixed with other kinds of evidence, such as experiences, values, political statements, and ambitions. Interestingly, in this study, a sample of local-level policies seemed to use research evidence more explicitly than national-level policies. Examples of other kinds of information used in the HEPA policies were good practices, information from media (newspapers, internet), information about society (economics, traditions), political statements and community activity (values, inspirations and citizens´ opinions). (Hämäläinen et al 2015).
How was the data gathered: The data was gathered by carrying qualitative document and policy analysis using thematic contents analysis of 21 HEPA policies in six EU countries of REPOPA (Table 1, Annex) and by interviewing policymakers and experts (n=86) who had been involved in the preparation of the HEPA policy documents analyzed (Hämäläinen et al, 2013).
Barriers: The main barriers for the use of research evidence in HEPA policymaking were: traditions, certain bureaucratic and organizational factors as well as the nature of policy making processes and political interests, such as pursuit of voters and a value base, research not being available or applicable when needed, academic language being difficult, and no procedures to gather research knowledge or meet researchers. Furthermore, there was lack of time, of financial and human resources, lack of competences and of joint criteria for the use of research evidence in the policy processes (formulation, development, implementation, monitoring and evaluation). As a conclusion, the insufficient collaboration between researchers and policymakers in general was seen as a barrier for the use of research evidence in policymaking. (Goor van de et al, re-submitted).
Facilitators: Main facilitators were: support from administrative departments to use research evidence and an organizational culture that supported the use of research evidence. Other facilitators were achievable and relevant research which is also easy to apply in practice. Furthermore, an important facilitator was direct or institutional relationships between policymakers and researchers. The research bodies may function as knowledge brokers through open forums (such as seminars) or through direct relationships. However, it is crucial to find a common language between policymakers and researchers. (Goor van de et al, re-submitted).
Cross sector cooperation: Working across sectors happened by stakeholder involvement, governance structures and coordination structures and processes. Results provide insights into key governance structures and coordination mechanism as well as processes in HEPA policies in six European countries. Findings show that there are many different structures and ways of working across sectors and also mechanisms for involving stakeholders in HEPA policies. (Hämäläinen et al 2016a).
Italian team analyzed a meta-policy on health promotion with the aim of understanding the use of evidence and the knowledge conversion process in a multi-actor policy trajectory, studying the different sources of evidence and their interaction (Castellani et al 2016).
Case studies were performed to get deeper understanding of external stakeholders´ involvement in the (Danish) local municipal policymaking contexts (e.g. Eklund Karlsson et al 2016) and in youth involvement in Danish physical activity policymaking (Eklund Karlsson et al, in preparation). The first study concluded that politicians had substantial influence on the involvement of external stakeholders, allowing only few to contribute in a closed policymaking process. Main involvement was through personal contacts or regular hearings. The second study concluded that mechanisms to facilitate youth participation in policymaking in the case study communities were lacking. Thus, the Danish goal of improved participatory policy making at the local level was not met by the study communities, especially regarding youth participation.
Manuscripts are under revision e.g. on comparing Danish municipalities in the use of research evidence (Jacobsen et al, under revision).
Initial indicators for evidence-informed policymaking
The policy analysis and stakeholder interviews carried out by REPOPA helped to produce initial indicators, guided by frameworks, for evidence-informed policymaking. The indicator development was completed later in the project by work package No4. Please see the chapter for the suggested indicator framework of the REPOPA project.
Ethics clearance procedure of the project
The EC required ethics clearance according to each partner country rules and regulations and to have it done before data collection started. As REPOPA with its policy research aim, belongs to mostly social science field, the country-specific practices varied a lot. Finland and UK required clearance for the whole international project and for all its sub-projects. However, as the home country Denmark of the coordinating institute, SDU, did not have any procedures for this, ad hoc system needed to be created. An ethics road map for the Consortium was developed by the coordinating team; this road map provided guidance built upon international ethics guidance such as Helsinki Declaration (World Medical Association 2013) and EC draft guidance for social science research http://ec.europa.eu/research/participants/data/ref/fp7/89867/social-sciences-humanities_en.pdf
None of the EU countries carrying out the document analysis required ethics clearance, but for the stakeholder interviews it was necessary both in Finland and UK. Clearance was also needed in Canada for evaluation activities and in UK for the Delphi study for indicator development. Other countries needed to deliver an official certificate that no ethical clearance was needed; in some cases this proved to be hard to get since there was no existing agency for this kind of certificate.
The ethics clearance process took over half a year from the active data gathering of the first project phase. The Consortium published a scientific peer-reviewed paper on this experience (Edwards et al 2013).
Summary
The first REPOPA phase added to the knowledge base of evidence-informed policymaking by analyzing the evidence actually used in a variety of HEPA policies across six EU countries and facilitators and barriers to the use of evidence. This information can benefit future policy development. In the REPOPA project the know-how of the first project phase fed into the successive phases such as interventions and indicator development.
REPOPA results phase 2
Based on the first year findings, the project went further to develop two kinds of interventions to learn how policymakers could be helped in finding and using research evidence.
Policy games to enhance use of evidence in policy making
Work package No2 developed and tested an innovative intervention for HEPA policymaking. It developed a policy simulation game to stimulate use of evidence in local policymaking, especially to enhance collaboration between organizations. A simulation game that resembles the real world policy-making context offers a safe environment to experience and learn about rather complex cross-sector collaboration and use of evidence.
What is a policy (simulation) game?
A simulation game is an experimental approach stemming from organizational sciences where it has been used in organizational change contexts (Duke, Geurts 2004, Geurts et al 2007). A simulation game is built on a profound systems analysis (Spitters et al, under review). This analysis visualizes all relevant organizations involved, their roles and positions and most important, their main drive to be involved in this policymaking context. The main stakeholders can be selected from the systems analysis; these are the “models” for the main roles in the game; they are invited to play the game. So the game resembles the real world context but is also placed in a fictive municipality. It is a one day experience of practicing cross-sector policymaking.
How was the data gathered?
The policy game was carried out in The Netherlands, Denmark and Romania. The teams in each country designed the game, chose real policy cases, did systems analysis and prepared the performance. Policy games were adapted to national and local contexts to fit the organizational actors and structures.
The game was a one day activity in each of the three countries. In total 18-20 participants per country, resembling real world colleagues, participated. They played eight to nine different roles in the teams of two to three persons in the game, based on the main organizations involved in HEPA policy making in the specific municipality. The teams were asked to construct a municipal HEPA policy implementation plan filled with as many as possible HEPA interventions, organized with and supported by other stakeholders and based on (some type of) evidence (see Figure 4, Annex).
Measurements of the experiences from the participants were done by observations of the game and through online questionnaires on three points in time: before the game, directly after and at six months follow up.
Results of the policy games
Results of four aspects of collaboration in organizational networks are presented: of cross-sector collaboration, organizational change, leadership aspects and use of evidence. (Goor van de et al 2015).
Learning experiences of the game for participants showed that cross-sector collaboration requires purposeful communication, coordination and direction. Further, the results showed that game boosts collaboration in real world context, which this experience is useful to daily work, and that game brought positive attitude towards use of knowledge and expertise.
In relation to organizational network, there were increased insights in organizational network and local policy context and increased commitment to and understanding of policy process, increased insights in positions and roles of organizations and sectors, and increased insight in the potential of the organization’s own contribution in evidence-informed policymaking.
In relation to leadership aspects, it was learned that clear directions and both formal and informal leadership promoted cross-sector collaboration, increased awareness of the importance of finding a common language, showing vision, inspiring and involving others. Potential for change in intentions in the games both in Denmark and The Netherlands appeared to be more limited than in Romania in terms of promoting trust, finding common language, communicating visioning and creating an environment open to all opinions.
In relation to use of evidence, game participants recognized barriers in the use of evidence in local policymaking: stakeholders were not very acquainted with the use of evidence in policymaking, it was not clear how to apply theoretical or research-based knowledge in daily practice, how to weigh different sources of evidence, or what is most important, there was lack of time and lack of resources.
Summary:
Simulation policy game proved to increase mutual understanding between policymakers and different organizations involved. It also helped to see how important organizational and leadership support is for using research evidence especially in cross-sector collaboration. It also increased insights into HEPA policy processes, including organizations’ networks and stakeholder roles. There was a change of attitudes and behavior (intentions) towards collaboration, towards leadership aspects and use of knowledge in a positive way. The policy game also helped to show the participants the benefits of the use of expertise and various sources of evidence, for example from organizations in the local network; this stimulates the process of integration of evidence into the local policymaking context.
What was also learned was that a similar policy (simulation) game can be built and executed within three different EU countries. However, the potential of a simulation game appeared to differ across countries, mainly depending on differences in administrative structure and policy context.
Stewardship-based, locally tailored interventions
Why Stewardship interventions?
Stewardship approach (Nuffield Council 2007) in health interventions is considered an ethical public health approach. Stewardship approach gives the government a leading role; it is based on liberal thinking, which emphasizes guidance, enables structures, services, equity, and avoids coercion. This is in line with the Nordic welfare model and it fits well physical activity policymaking in Western Europe.
In practice, Stewardship approach is based on the needs, context and stakeholder analysis. In REPOPA the approach was used to run interventions with the aim to increase integration of research evidence with other kinds of evidence in close academia-policy maker collaboration. Further, these interventions also aimed to find ways to identify and effectively engage specific stakeholder groups in physical activity policymaking. In addition, the Reach; Effectiveness; Adoption; Implementation; Maintenance (RE-AIM) evaluation framework by Glasgow et al (2006) was adapted and used for evaluating the interventions.
How was the data gathered?
The six selected policy case studied dealt with HEPA policies. Each participating country team of Denmark, Italy and the Netherlands selected two policy cases based on a set of common criteria. In Denmark the interventions aimed at increasing cross-sector collaboration and strategic policymaking. In the Netherlands, the interventions aimed at strengthening neighborhood sports and physical activity planning and developing physical activity for those 65+. In Italy the interventions aimed at knowledge and research translation as well as information exchange related to Olympics of students and to healthy roads Pedibus activities.
Intervention contents were tailored to the context meaning that they had a common goal of increasing evidence-informed policymaking but different means of reaching it. Standardization thus happened for the common framework, function and process, but the intervention contents varied according to the local needs (Hawe et al 2004). Intervention contents varied from workshops to training sessions, networking, knowledge exchange and consultation (Table 2, Annex). The intervention length varied from six to 18 months, depending on the policy process in question. The instruments to measure the intervention effect included a set of common questions developed for the study based on the needs assessment done in the beginning of the interventions (Bertram et al 2016), on the use of research knowledge, stakeholder knowledge, needs and values of the policy target groups. The initial REPOPA indicators were included in the core instrument questions used in all intervention settings. In addition, process evaluation was carried out in all settings.
Results of the Stewardship interventions: The interventions increased the level of evidence-informed policymaking in the selected, real-world policy case studies in Denmark, Italy and the Netherlands. Especially, there was an increase in the access, requests and use of research knowledge, in using stakeholder knowledge as well as needs and values of the policy target groups. There was also a decrease in barriers for using research knowledge in policymaking (Aro et al 2015b).
Even if the type and size of the intervention studies did not allow quantitative analysis, there was a clear indication that different stakeholder groups became more aware (Valente et al 2015), more active in searching for knowledge and in networking, and more open to values of others. However, as the 12 months follow-up results showed, it was a challenge to provide sustainable results with not so intensive and often sporadic intervention activities placed in the busy normal working life of the participants). All in all, the initial results post-intervention and also the increased networking and appreciation of the local needs and values, provide a good starting point for further interventions, boosters, research and implementation programs. These kinds of needs-based interventions as well as collaborative policymaking of policymakers and researchers seem to be an effective way to enhance evidence-informed policymaking.
Summary: Stewardship-based interventions, which were tailored to local policy needs, proved to be useful for policymakers to learn to take the needs and values of the policy target groups into account. Also politicians learned to request for more research evidence. However, sustainability of the results needs to be supported by continued collaboration or booster sessions between researchers and policymakers. The instruments (questionnaires) developed to measure the outcome of the interventions very likely would also serve as general tools outside intervention evaluations to measure the developments and trends in evidence-informed policymaking.
REPOPA results 3rd phase
Developing indicators for evidence-informed policymaking
The aim of this phase was to draw together the results, methods, frameworks, indicators and best practices in the knowledge translation from research to policymaking in physical activity developed in the previous project phases. In this phase the idea was to test, validate and negotiate these results and products among the stakeholders from the participating countries and among representatives from other EU countries and to formulate into guidance resources and evidence briefs.
How was the data gathered?
The Delphi technique (Gupta, Clarke 1996, Linstone, Turoff 2002, Okoli, Pawlowski 2004, Fletcher, Marchildon was used; it is a qualitative analysis methodology that aims to extract the collective knowledge of a group of experts in the study of a complex system. The Delphi process was implemented in three stages: two 2014, Boulkedid et al. 2011, Syed et al 2009, Bolger, Wright 2011, Rowe, Wright 2011) internet-based Delphi rounds at international level and a third face-to-face Delphi stage at national level. The process included an internal consultation among the REPOPA researchers and literature analysis, summarizing findings from the previous phases, and identifying draft indicators related to evidence-informed policymaking as well as organizing preliminary indicators in thematic sets. The aim was to develop indicators which would be SMART (Specific, Measurable, Attainable, Relevant, Time-bound) allowing them to be used to (quantitatively) measure the level of policies being evidence-informed.
The two internet-based Delphi rounds involved 76 experts from researchers, policymakers and politicians in the field of health, physical activity and other sectors, coming from the six European REPOPA countries and international organizations, to evaluate and improve the thematic sets of indicators proposed by the REPOPA researchers (Figure 5, Annex). In the first round they were asked to answer an online questionnaire to rate the relevance and feasibility of the draft indicators, to comment their ratings and propose further indicators to be included. In the second round they were asked to answer again an online questionnaire to rate relevance and feasibility of those draft indicators, which had been evaluated in the first round but which had not reached consensus on high relevance and feasibility. Summary of the results gathered in the first round in terms of ratings and comments was provided in order to make the rating easy. In this round the participants were also asked to evaluate new draft indicators developed on the basis of their own suggestions.
The third Delphi stage, which was a face-to-face round, consisted of six national conferences, which took place in the six European REPOPA countries between January and March 2016. The aim was to contextualize at national and local level the indicators that had been evaluated as highly relevant and feasible in the internet-based Delphi rounds. National conferences were organized in a synchronized way following a common methodology, involving researchers and policymakers in the field of health, physical activity and across sectors, partly selected from the panel of the previous Delphi rounds. Before the national conferences, the participants in each country assigned the indicators into policy phases using an online questionnaire; this was done to see when in policymaking cycle each indicator was useful and relevant. At the conferences, the participants performed a SWOT analysis on the indicators (finding their Strengths, Weaknesses, Opportunities and Threats) and identified their barriers and facilitators; they also suggested recommendations to overcome the possible obstacles in using the indicators and foster evidence-informed policymaking. The used referral frameworks were models describing the evolving science and science-policy relationships and theories of knowledge production and exchange, especially the Knowledge-to-Action cycle (Straus, Holroyd-Leduc et al 2008, Graham et al 2006).
Results of the Delphi study to develop indicators for evidence-informed policymaking
The Delphi study produced a list of 25 measurable indicators and 8 further indicator themes/complex indicators, which are not (yet) measurable. The 25 indicators belong to the following four sets: Human resources – Competences and Networking, Documentation - Retrieval/Production, Communication and Participation, and Monitoring and Evaluation (Table 3, Annex). The eight indicator themes/complex indicators point to relevant and important issues in evidence-informed policymaking, but they need further development as indicators.
Fact sheets and policy briefs for policymakers
This REPOPA phase also produced fact sheets of the national conference output for each country. In addition, evidence briefs were developed for each country on the basis of the national conference results and tested for feasibility and usability among participants of the conferences. For the overall policy brief of the Delphi study, please see www.repopa.eu
Guidance resources
The results of the national conferences were also used to formulate a checklist to guide policymakers in using the indicators for evidence-informed policymaking. The checklist is a part of the guidance resources for policymakers and stakeholders on REPOPA indicators and their use. Guidance resources were structured as follows: 1. Short introduction (on evidence-informed policymaking and why the indicators were developed); 2. Development and contextualization of the measurable indicators as well as the final list of REPOPA indicators; 3. Short instructions on how to use the indicators and the checklist to guide the use the indicators; 4. Glossary, final notes.
Summary: Using Delphi process, the REPOPA project developed the findings into usable tools such as indicators for evidence-informed policymaking. These 25 measurable indicators can be used as a checklist in policymaking to assess if and how well the policy in question is actually evidence-informed. Further, evidence briefs were developed to summarize the evidence gathered for each of the six country settings of indicator development.
Dissemination in the REPOPA project
The dissemination goal of REPOPA was both to spread news about what had been done and developed in the project and to build sustainable structures to enhance evidence-informed policymaking so that it would continue after the project lifetime as well. The work started by developing a dissemination strategy with a detailed external communication plan that aimed at ensuring the best visibility of the REPOPA activities and outcomes at EU level, targeting stakeholders and wider public, raising interest and awareness of REPOPA activities and results. To be able to do this comprehensively, a stakeholder mapping was done in the beginning of the project. The stakeholder mapping was done in all partner countries, at a small scale, to identify in the first six months of the project, stakeholders, areas for future improvement, how can we transpose our findings to other health-related fields and indicators for dissemination. One of the findings reveals the importance of in-country platform as a way to translate findings to other health-related fields: “Create legitimate centers that can ensure a clear translation of research results in evidence for policymaking”.
A special task for dissemination in REPOPA was establishing a web-based umbrella platform to support country-level evidence-informed policymaking platforms created in the course of the project (See photo 1 in the Annex, http://repopa.eu/content/eipm-umbrella-platform). Each partner could either search for in-country work group or launch a forum that had an interest in evidence-informed policymaking on local, regional or national level. An example is Finland, where a common platform for HEPA policy discussion was established by the working group in conjunction with others that allowed access to information on physical activity within the Ministry of Education and Culture. Other examples come from e.g. Denmark, where the platform took the form of an interest group ”Research, Practice and Policy – better collaboration and knowledge integration for Public Health” as part of the National Public Health Association. The other countries had a more local approach. For example, in Romania, a Work Group Public Health and Work Group Sports and Community was established at the end of 2013 as part of the Cluj Management and Planning Group, a task force created for the development of the new Cluj-Napoca Development Strategy for 2014-2020. Moreover, the Romanian partner has continued its work as part of the platform by joining national and international networks in 2016. The Romanian partner has applied and has received HEPA Europe temporary membership confirmation. The European network for the promotion of health-enhancing physical activity, until confirmation by the network at its next annual meeting (September 2016), being the first Romanian institution member of the HEPA Europe Network. In Italy, CNR has started to build the Italian National Platform in 2013, involving stakeholders, researchers and policy makers in the field of public health and cross-sector areas at a local and national level, plus a few experts from international organizations. The platform includes research managers of universities and research centers, administrators of health organizations (for instance, local hospitals and the National Health Institute) and political institutions (for instance, the Lazio Regional Council and the Ministry of Health); sector associations (some examples: Associazione italiana di epidemiologia, Associazione Italiana di Economia Sanitaria, Associazione Nazionale dei Medici delle Direzioni Ospedaliere, Cittadini Reattivi, the Italian National Olympic Committee, the European Network for Sport Science Education and Employement, the European Association of Hospital Managers); sector magazines, like Epidemiologia e Prevenzione, L'ospedale, European Journal of Sport Science.
The REPOPA website features of the platform include also rather detailed description of each country context and the status of the evidence-informed policymaking in each context. The platforms are supposed to increase sustainability e.g. in case of staff members of institutions involved change.
Dissemination in the Consortium was planned to happen systematically; for this reason the project developed working protocols and templates including internal and external communication guide, internal deliverables and activities guidelines, publication guidelines and continuously updated publication plans, and internal dissemination reporting template. These templates and guidelines can function as tools also beyond the REPOPA project. (Chereches et al 2016 submitted).
Channels for dissemination
Together with the platform launch, REPOPA has launched its twitter account, which is integrated in the platform as well. The role of the twitter account is to open discussions with other researchers, practitioners, policymakers and actors interested in the REPOPA work and outcomes. Below are some highlights of the twitter account in the last month of the project (September, 2016). Our September 2016 tweets had 1,475 tweet impressions and gathered 131 profile visits, adding eight new followers to our page. Most of our tweets focused on evidence- informed policymaking, sharing our own knowledge, for example: “Policymakers can benefit from support by researchers - #SDU #nhprc2016 http://bit.ly/2dabE7z “ or acknowledging existing knowledge and information disseminated by others, for example: “How can we use evidence to strengthen policy? Introducing the Evidence-Based Policymaking Collaborative: http:/urbn.is/2cDBOD9 #whatworks”. The account will remain active after the project ends, supported by UBB and SDU staff.
Newsletters: twice a year the REPOPA newsletter informed internal and external audiences about the project progress and outcomes. Leaflets and posters: yearly the project leaflets and posters where used as important dissemination tools at meetings, conferences, and workshops. Over 600 leaflets were disseminated.
Conference presentations: a total of 92 abstracts were presented at national, European and international conferences, out of which 35 were published in peer-reviewed international journals. Workshops: each year, REPOPA has organized at least one workshop at the European Public Health Association Conference (EPH).
Publications of peer-reviewed articles and chapters: a total of 16 peer-reviewed articles and chapters and 29 peer-reviewed abstracts were published based on the results of the REPOPA research and dissemination work; a number of publications are under revision. The REPOPA Consortium plans to continue collaborating in publishing valuable data. Popular journals: a total of 30 articles were published in national popular journals, institutional newsletters and international ones as well.
Networks: the REPOPA Consortium has joined networks like icSPORTS, European Implementation Collaborative Network (http://www.implementation.eu/) and Research to Action Network (http://www.researchtoaction.org/) HEPA Europe Network (http://www.euro.who.int/en/health-topics/disease-prevention/physical-activity/activities/hepa-europe).
Evidence briefs: Print and online version of evidence briefs were developed to disseminate research work of the REPOPA project – a total of four evidence briefs were developed for different work packages of REPOPA.
Evidence Brief no. 1: Evidence Informed Policy through Simulation Games – A practice in collaboration; based on the work package No22 simulation game work. REPOPA investigated if a policy game intervention - based on needs, context and systems analysis – could be used to improve communication and collaboration between stakeholders in a cross- and multi-sector health policy making process. In which way does a policy game enhance the level of evidence-informed HEPA policy making and lead to a more efficient and successful policy? Some results are presented in the evidence brief along with the steps we used in simulation game.
Evidence Brief no. 2: Recommendations for Locally Tailored Interventions – Knowledge integration in physical activity policymaking; based on the work package No3 Stewardship-approach work. Policymakers often have a hard time to find and apply research evidence in their real life contexts, which are loaded with different stakeholders, priorities and values. So far there has been rather little research on interventions to support policymaking in balancing these different kinds of evidence and information. REPOPA took this challenge and developed and implemented locally tailored interventions. The evidence brief goes through the steps of the tailored-stewardship intervention.
Evidence Brief no. 3: Recommendations for Using Evidence in Real World Physical Activity Policy Making; based on the work package No4 Delphi study work. Evidence-informed policies take into account both research evidence and contextual resources, priorities and values. There is a need for tools to assess how well policies are evidence-informed. Based on our Delphi study we developed and validated indicators to do this and to offer guidance for the integration of evidence in policymaking. Recommendations are detailed in the evidence brief.
Evidence Brief no. 4: Recommendations for Identifying Optimal Evaluation Processes for Funded Research Projects; based on the work package No5 internal and external evaluation research work. Research funding agencies need to demonstrate the health and policy impact of their research funds using optimally-designed evaluation processes. Some controversy and debate exist about the value of external project evaluations, and expectations and requirements for project teams for process evaluations. Our experience evaluating a multi-country research project offers one effective approach and is presented in the evidence brief.
Animations: So far three animations were developed to better describe the interventions implemented in the project REPOPA as lay summary versions of the work. The animations will be used for dissemination and teaching. For the animations, please visit www.repopa.eu
In parallel to specific work package No6 dissemination activities, a major role was played by the research work packages in disseminating and exploiting the outputs of the research work, which has a stronger impact in research knowledge and project sustainability.
Final REPOPA Symposium in Brussels
The culmination point of the REPOPA project was the final, invitational Symposium organized in Brussels the 8th of September 2016. This symposium discussed the output and experiences of the project and continued to brainstorm on the issues found most problematic in REPOPA and thus needing further work. http://www.repopa.eu/news/researchers-can-support-policymakers-their-work.
This brainstorming was done using the world café method in which groups of experts going around discuss the themes dedicated for each table. All in all we had 20+ external experts participating in the Symposium; they came from the six REPOPA countries representing ministries, sport organizations, universities and from the EU and international organizations. The themes of the world café tables included the following: facilitators and barriers as well as competences needed in evidence-informed policymaking, contextual factors and implementation challenges in developing evidence-informed policies. See photo 3 from the world café in the Annex.
E-survey in the end of the project
The e-survey was developed together with the partners in the last year of the project. It aimed to test the developed evidence-informed policymaking indicators as well as synthesize information provided from the whole project period, including the invitational final symposium September 8th 2016 in Brussels, Belgium and the 7th conference of HEPA Europe, 28-30 September 2016, Belfast, Northern Ireland, United Kingdom. The survey aimed to gather knowledge to improve the validity and reliability of the indicators, and also to test if items in the survey were clear and understandable, relevant and pertinent to their organization. The development of the e-survey was covered by the ethics exemption approval nr. 31/62 from 16.10.2012 of the Ethics Committee of Babes-Bolyai University from Cluj-Napoca, Romania – the coordinator of work package No6 – Dissemination. The application of the e-survey includes an informed consent, as well. The questions are meant to be answered from the perspective of organization/institute of the respondents and aiming for testing the usability of this instrument as regards the readiness for evidence-informed policymaking practices. The e-survey is a 21 items online survey that takes about five to ten minutes. The UBB project team has invited stakeholders to take part in this online survey.
Summary: REPOPA dissemination both spread the output and news from the project and built an umbrella platform with country-level platforms to create sustainability for evidence-informed policymaking in the REPOPA countries and beyond (Chereches et al 2016 submitted).
Project process and impact evaluation
REPOPA Consortium wanted to work transparently and to learn from its working process; for this purpose, a special work package (No5) was established and given to a Canadian partner to lead. The Canadian partner was the only non-European partner in the project. The partner had had long experience in evidence-informed policy work and real-world collaboration in projects.
Evaluation tasks were to monitor and evaluate the working process, content, scientific product and added value and synergies of project parts, dissemination activities plus the potential impact of the project as whole. In addition, the aim was to apply the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation and Maintenance, (Glasgow et al 2006) to analyze impact of project activities in the participating countries/context and in relation to development in policymaking more widely.
How was the evaluation done?
A mixed methods approach was used for both the formative process evaluation and the final summative outcome evaluation. The formative monitoring and evaluation strategy was participatory, consensus-orientated and utilization-focused. The evaluation rounds were done in annual cycles (2013-2016); action recommendations were given to the Consortium to stimulate Consortium discussion and action planning to build on project strengths and opportunities, and address implementation challenges. For the project working processes, annual interviews were conducted with work package and country teams; topics included Consortium diversity, stakeholder engagement, REPOPA knowledge products and dissemination products, and priorities for upcoming work. Also document reviews of periodic reports, final reports, internal six months reports and minutes of different Consortium meetings, were conducted annually to examine project and work package progress, management, and implementation challenges. The Consortium collaboration survey was conducted annually. The survey assessed five domains: communication, collaboration, knowledge translation, project management, and evaluation. The junior researcher competency self-assessment survey was delivered annually to all Consortium members who were trainees. They were asked to rate their level of improvement in the previous year, on 27 competencies and skills related to research methods, knowledge translation, evaluation, research writing, project management and communications, and mentorship. The social network survey examined internal scientific contacts among project team members and external contacts with stakeholders. Network maps were made using NetDraw (Borgatti, 2002) to help visualize the REPOPA networks.
The final project evaluation was conducted using an external summative approach. It drew on the four years of data collected during annual monitoring to assess project working process, synergies, dissemination, and impact. For synergies among work package components and value added to the project, annual interviews with work package and country teams were used; in addition to publications, research, innovation, and challenges to achieving synergies. For dissemination, national platforms and the web-based umbrella platform developed by the project were examined, and project dissemination approaches were reviewed. For scientific and policy relevance, evaluation was done on final work package reports, frameworks, interview schedules and publications; and gender and context were used as additional criteria.
Results on scientific relevance
The project work packages provide considerable value added, with a number of key strengths: rigorous methods (including case studies), good heterogeneity in types of stakeholders and sectors involved and rich descriptions of context that informed several processes (e.g. context mapping) developed uniquely for each work package.
Work package No1 yielded insights into cross-sectoral policy making that may inform both policy makers and researchers. Findings highlighted some critical considerations in facilitating cross-sectoral policy development approaches and yielded methodological advances in the meta-analysis of cross-sectoral policies. Both work package No2 and No3used a context-informed adaptation of their intervention approaches. They provide strong illustrations of how to address the tension of standardizing form versus function in interventions. The use of actual (rather than hypothetical cases) strengthens the applicability of their findings. Work package No4 developed a list of measureable indicators for evidence-informed policymaking, refined with input from stakeholders across six European countries by means of a Delphi process. Processes used in both conceptualizing and refining indicators were robust.
Limitations: Although not a requirement of the Description of Work, the lack of any costing data or specific documentation of human resource requirements makes it difficult to assess the feasibility of adapting and using the interventions in other settings. The gender considerations and analysis as described in the project documents were also met, but a more explicit gender analysis framework could have brought more in-depth information. It is likely that team members were working with “early adopters” and some of the findings may only be transferable to other municipalities that are willing to engage (or already have relationships) with external researchers. Work package No1 only looked at the phases of agenda setting and policy development. Work package No4 provides some promising indicators for evidence-informed policymaking; those which were least developed were in a sub-category of “complex indicators”. These warrant further development given the nature of inter-sectoral decision-making (Edwards et al 2016, submitted).
Results on project working processes
The REPOPA project accomplished the ambitious scope of work outlined in the Description of Work despite several implementation challenges. Collaboration networks that developed within and across work package teams helped Consortium members accomplish their ambitious scope of work (Figure 6, Annex). Between 2013 and 2016, there were improvements in team collaboration in the areas of communication, collaboration, knowledge translation and evaluation. Involving junior researchers in REPOPA proved mutually beneficial for both the project and these trainees. A total of 12 junior researchers were linked to the project at different points in time. Across all project years, junior researchers most often reported self-improvements in a variety of research methods and competencies; they least often identified improvements in either knowledge translation or evaluation skills. Improvements in writing skills were most notable in the final year of REPOPA. Country and work package teams developed and extended their networks with policy stakeholders as the project progressed. There were shifts noted from single to multiple sector engagement and from the involvement of municipal to national and international decision-makers. External connections became deeper and more deliberate although not necessarily more numerous. In the first year, researchers reported that the primary benefit of working with external stakeholders was understanding the policy context. In the last two years, they described their external stakeholders as primarily assisting with the dissemination of research findings and providing access to decision-makers.
Results on Synergies among Work Package Components and Added Value for the Project
During interviews, Consortium members stated their commitment to maximize the added value of a multi-country initiative and its multi-disciplinary team. The structure of work packages, planning meetings and the commitment of work package leads helped to facilitate the cross-fertilization of ideas, approaches and findings. Policy makers’ engagement as intervention participants in all phases of the project was seen by the team as an important success. While the involvement by Consortium members in multiple work packages contributed positively, it also carried a cost, since it meant that a number of team members faced multiple and significant competing priorities. When members were pulled in too many directions, manuscript writing efforts were most often deferred.
Results: Dissemination
By the end of the project, all countries had established national platforms though to varying extents. Teams were starting to use national platforms to disseminate REPOPA findings. Plans to continue the web-based umbrella platform beyond the end of the project may provide a useful resource to support national platform efforts. Because country teams involved in work packages No 2, 3 and 4 had to prioritize research activities for much of the project duration, dissemination tools developed by dissemination work package were underused.
REPOPA used a variety of dissemination strategies to reach researchers, policy stakeholders, and the general public. As a multi-country project, teams had to find the right balance between producing dissemination materials in English versus tailoring language and content to the needs of national policymakers. Stakeholder engagement is essential to intervention success and creates expectations from external participants (and related time commitments for the project team) that go beyond the effort required for more traditional dissemination efforts (e.g. conference presentations and publications).
Summary
Participatory, utilization-oriented process evaluation seemed to strengthen project implementation and science. It very likely increased utility, uptake, and ownership of evaluation findings and related recommendations. Projects that deliberately bring together multiple countries, contexts, and interventions may deliver outputs with stronger scientific and policy relevance. They also face particular challenges in tailoring interventions and developing targeted dissemination strategies for a variety of policy stakeholders. An optimal evaluation design may need to involve a combination of participatory process evaluation that supports iterative planning and decision-making, along with an outcome evaluation to assess impact and the added value of synergies. Imbedding an evaluation team into a Consortium is an effective tool for internal and external evaluation. (Edwards et al 2016, submitted)
Towards an integrated REPOPA framework and indicators for EIPM
Several suggestions have been presented as facilitators and barriers of evidence-informed policymaking but there have been very few attempts to develop a generic and integrated framework to coherently describe the areas and issues to be included in such a framework for evidence-informed policymaking Building on the published literature, findings and lessons learned in different work packages REPOPA project drafted a conceptual framework to guide the development of indicators to assess if and how evidence-informed policymaking is The work started by mapping the theories and frameworks published in the field and operationalizing them for evidence-informed policymaking process using a practical scenario of developing physical activity policy in a school setting (Bertram et al 2015).
Definitions
The framework is a conceptual map or structure linking indicators to theories, goals and planning of policymaking. It is a theme-oriented approach reflecting the key challenges in evidence-informed policymaking. (Eurostat report, http://ec.europa.eu/eurostat/web/sdi/indicator-framework accessed 211116)
The indicator is a variable selected and defined to measure progress towards an objective (Litman 2011).
Policymaking process is evidence-informed when different stakeholders, including researchers, as active participants create their salient and feasible policies by taking into account contextual needs, priorities, values, resources and skills, pay attention to equity and ethics as well as evaluation and accountability of the policy- all this in the wider cultural and social context (Satterfield et al 2009, Edwards 2012).
The process of developing REPOPA framework and indicators
Contribution of work package No1 to the REPOPA framework and indicator development
The work package No1 policy analysis and interview results confirmed the relevance of the preliminary REPOPA framework and indicators, which we drafted in the Description of Work. The framework for the intersection of public health decision making (Satterfield et al 2009; Figure 1, Annex) which describes the overlapping circles of best available research evidence, resources, including practitioner/policy maker expertize and contextual characteristics such as population, its needs, values and preferences - all happening in the environmental and organizational context, seemed to be well-fitting for the REPOPA work. In line with this, a health promotion evidence paradigm (Aro et al 2008, Skovgaard et al 2008, Edwards and Di Ruggiero 2011) that helps to assess salient contextual factors and challenges for policy implementation, was supported by the work package No 1 results.
The Knowledge-to-Action Cycle (Figure 2 Annex), (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012), described in the REPOPA Description of Work, also turned out to provide a viable model for the analysis of and later interventions on evidence-informed policymaking. Building on the research knowledge synthesis, providing locally relevant knowledge and assessing facilitators and barriers of knowledge integration and working towards tailored interventions were all enforced by the results. Further, in the course of REPOPA, Edwards (2012) developed a revised version for CIHR’s (Canadian Institutes of Health Research) Institute of Population and Public Health (IPPH) of Graham et al.’s (2006) original Knowledge-to-Action Framework. The revised version includes scalability of knowledge, inter-sectoral collaboration, implementation systems, economic analyses, and equity and ethics considerations. These elements (except implementation systems due to the scope of work package No1) were also covered in the work package No1 results; especially cross-sector collaboration (Hämäläinen et al 2015; and equity (Hämäläinen et al 2016b, in press), and were relevant for further research such as REPOPA interventions in work packages No2 and No3. In addition, as systems approach (Best, Holmes 2010) shortly described in the REPOPA Description of Work suggests, European physical activity policymaking as studied in REPOPA, seems to benefit from the non-linear, complex systems perspective, which allows unpredictable changes but also helps to better understand policy processes.
Further, Stages of Knowledge Use (Knott, Wildawsky 1980, Landry et al 2006) framework was usable in the REPOPA work; this framework describes six stages of knowledge use seen as a process, which includes reception, cognition, discussion, reference, effort and influence. Elements of these e.g. influence, was one of the central questions to study in work package No3. Weiner’ framework (2009) of organizational readiness for change functioned well especially in work package No2 policy games. Phases of policymaking (Walt et al 2008) guided centrally the work done in work package No4 when mapping the evidence-informed policymaking indicators into policy phases in different countries.
The aim of developing preliminary REPOPA indicators based on the work package No1 results using policy analysis and stakeholder interviews, was to suggest a usable method for assessing the process of developing and implementing policies rather than establishing the status of policies. Hence, the proposed initial indicators from work package No1 identified possible areas for improvement in integrating research evidence into real-life policymaking. They also provided input for the other work packages of REPOPA; the main further work was done in work package No4.
The list below gives the set of initial indicators formulated in the end of the 1st research phase of REPOPA, work package No1; in the subsequent work packages they were further re-formulated, tested, validated and contextualized.
1) Is the best available research evidence taken into account?
• If so, is it based on an evidence hierarchy (systematic review, evidence synthesis, article synopses, randomized controlled trials, cohort studies, case-control studies, etc.): yes/no; please specify
2) Is research evidence integrated with contextual needs?
• Yes/no. If yes, what kind of contextual needs e.g. feasibility, political priorities, resources, population characteristics etc. are taken into account?
3) Are relevant stakeholders involved?
• Yes/no. If yes, which stakeholders and how are stakeholders involved? And to what degree is the research community represented?
4) Have all relevant sectors been identified and involved in the policy development and implementation?
• Yes/no. If yes, which sectors?
5) Are equity issues taken into account in the policy making process?
• Yes/no. If yes, how is equity taken into account? (E.g. inclusion of those whom the policy concerns, consideration of possible special needs for vulnerable or ethnic minority groups in terms of access to physical activity facilities)
6) Is policy evaluation and/or policy impact assessment in place?
• Evaluation: outcome, process, context evaluation. Yes/no: please specify
• Impact: health impact, environmental impact, social impact. Yes/no: please specify.
7) Is a systems approach taken into account, meaning that policy development can be non-linear, with inter-dependences between levels, triggered by e.g. media events, negotiated interests and communication? Yes/no, please specify.
8) Are supporting structures (evidence platforms, working groups, and networks) and knowledge dissemination of evidence-informed policy making in place/built/encouraged? Yes/no. Please specify.
Contribution of work packages No2 and No3 to the REPOPA framework and indicator development
In the work packages No2 and No3 the programmatic research approach chosen for REPOPA was most clearly implemented. In these the systems analysis of the intervention contexts and needs assessment of the stakeholders were the basis for salient and relevant interventions – in line with Satterfield et al (2009) evidence-informed policymaking and Knowledge to Action Cycle (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012) Figure 2 Annex) frameworks. Further, especially in the work package No3 using Stewardship approach (Nuffield Council 2007) to guide the intervention development, the inclusive, equity emphasis was explicit. Using these frameworks these work packages contributed to eliciting potential indicators for evidence-informed policymaking.
Work package No2 policy game interventions in Denmark, The Netherlands and Romania aimed to find out the ways in which policy games can increase the use of research evidence and other evidence in policymaking (Goor van de et al 2015). The intervention results provided further issues to build draft indicators for evidence-informed policy making such as leadership, communication and collaboration within organizational network, including stakeholders, importance of cross-sector collaboration for HEPA policy, where to find useful knowledge, increasing use of knowledge and expertise of stakeholders.
The work package No3 case studies analyzed in Denmark, The Netherlands and Italy aimed to investigate if the Stewardship approach increases the level of evidence-informed policymaking (Aro et al 2015, Bertram et al 2016). The results supported issues emerging from work package No1 and No2 and provided new inputs for the formulation of indicators in especially two areas: Human resources – Skills and competences to promote evidence-informed policymaking and networking, highlighting the relevance of involving stakeholders, vulnerable groups and target groups as active partners in the policy design and implementation; and Communication and participation, proposing indicators related to if and how to gather knowledge from stakeholders and target groups and highlighting, in this way, the relevance of regularly informing researchers and stakeholders about the policy development and implementation process.
Contribution of work package No4 to the REPOPA framework and indicator development
Within work package No4, further analysis of literature on indicators, on their typology and development criteria was done (Bertram et al., 2015). Building on the results of the previous phases of the REPOPA project, as well as on the internal consultation round, a three-round Delphi study was performed to formulate and validate a list of indicators for evidence-informed policymaking (Valente et al, 2016 submitted). First, two rounds of international surveys were run; these produced 25 measurable indicators organized in four thematic areas of human resources, communication and participation, documentation, retrieval and production, and monitoring and evaluation. In addition eight so called ‘complex indicators’ were identified, which could not (yet) be made measurable. After that each of the six REPOPA EU countries arranged national conferences to further contextualize and validate the indicators developed in the internet rounds.
REPOPA framework: operationalization of the Satterfield framework
REPOPA empirically developed, validated and contextualized indicators to measure the central characteristics of the Satterfield et al (2009) (Figure 1, Annex) framework of the intersection of decision/policymaking. REPOPA was able to operationalize otherwise abstract characteristics or knowledge sources in the Satterfield framework.
The Satterfield circle of population needs, values and preferences was empirically confirmed in its importance in work package No3 intervention results (Aro et al 2015, Bertram et al, in preparation). It was found that e.g. in addition to having certain stakeholder groups involved in decision making, also organizational procedures are important for sustain evidence-informed policymaking. Population needs, values and preferences were further operationalized in REPOPA indicator set ‘Communication and Participation’.
The Satterfield circle of best available research evidence is covered by REPOPA indicator set ‘Documentation’. This information was initially produced in work package No1, and it was further confirmed in work packages No2, No3 and No4.
The Satterfield circle Resources, including practitioner expertize is covered by the REPOPA indicator set ‘Human resources’. This knowledge was empirically produced from work packages No2 and No3.
The Satterfield background circle (Figure 1, Annex) Environmental and organizational context was a central descriptor of larger, overall context influencing evidence-informed policymaking e.g. in work package No2 and No3. However, in the Delphi study, indicator candidates related to Environmental and Organizational context were not deemed measurable; they were called ‘complex indicators’ and included issues such as cross-sector collaboration, organizational culture and leadership in evidence-informed policymaking, accountability for policies, involving researchers early in the policy process, commitment of policymakers to enhance citizen participation, requests from policymakers for research evidence and evaluation of policy results.
Using the EU Sustainable Development classification of indicators to headline, operational, explanatory indicators; the latter can be further divided into contextual indicators, indicators under development and indicators to be developed; the current REPOPA indicators are at best explanatory; and the complex indicators could be contextual indicators or indicators under development http://ec.europa.eu/eurostat/web/sdi/indicator-framework.
REPOPA indicator set Monitoring and Evaluation is not directly operationalizing any of the Satterfield et al (2009) circles. On one hand, though deemed measurable, these validated indicators are also related to the category of ‘complex indicators’ (not measurable at this stage), among which there is ‘evaluation of policy results’. On the other hand, Monitoring and Evaluation could be better used to operationalize a phase of another central framework guiding REPOPA work- that of Knowledge to Action Cycle (Straus, Holroyd-Leduc, 2008. Graham et al 2006), where there is a phase of evaluation.
The Satterfield intersection of best available research evidence, population characteristics etc., and resources etc., which is labeled as decision making, depicts evidence-informed policymaking in REPOPA (Figure 1, Annex). This means that when all three small circles overlap and they overlap in the context of policymaking, policymaking can be said to be evidence-informed. Research evidence is taken into account, but so are contextual issues such as priorities, values, resources and so on. This Satterfield et al (2009) descriptive way of presenting the indicator framework does not yet tell about how well/to what extent the overall policymaking is evidence-informed; unless one could consider the size of the overlapping area telling this- this again depends also on the availability of each kind of knowledge and input (best available research evidence).
Added value of the REPOPA framework and indicators
The added value of the REPOPA framework and indicator work is the first step towards operationalization of the concepts and measurability of the evidence-informed policymaking indicators. The REPOPA framework and indicators cover those aspects of evidence-informed policymaking in physical activity area, which were seen important based on the previous literature and the REPOPA project work in work packages No1-No4. This does not mean that they are exhaustive to describe and measure evidence-informed policymaking even in physical activity. It is important to remember that especially work package No4 Delphi work emphasized measurability of the indicators to be included. The SMART (Specific, Measurable, Attainable, Realistic, Time-bound) criteria were used; in addition, objectivity was added as one criterion. Those eight ‘indicators’, which were too complex to be SMART (though relevant) for evidence-informed policymaking, are thus listed separately- and can later be developed towards measurable indicators. Further, with different research aims and questions plus methods, very likely different characteristics of evidence-informed policymaking could be found. In the future e.g. explorative factor analysis method could be used for empirical data sets to study underlying dimensions of evidence-informed policymaking; like this the indicators could potentially be developed into a multidimensional quantitative scale.
Further development and limitations of the current REPOPA framework and indicators
This report presents mainly how REPOPA could operationalize Satterfield et al (2009) framework and partly Knowledge to Action framework (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012) Figure 2 Annex). This initial work functions as an example of the further refinement and framework development, in which also other frameworks will be considered, especially those of organizational change (Weiner 2009), policy and political theories (Walt et al 2008, Kingdon 2010). Thus the present REPOPA framework and indicators need to be implemented and tested in different contexts.
Potential Impact:
Overall REPOPA aim and process: societal impact
The title of the REPOPA project already tells the goal of the project: ‘Research into Policy to enhance Physical Activity’. Thus the whole idea behind the project and its objectives both in terms of scientific work and in terms of collaboration and dissemination, are directly linked to real-life and societal activity such as developing physical activity policies in different settings. The project was built on trans-disciplinary approach and used approaches and methodologies that integrate scientific knowledge and non-scientific and non-formalized knowledge (=other kind of evidence). In this way, REPOPA can contribute to advancing fundamental understanding and solving complex problems while fostering multi-actor engagement in the research and innovation process.
In its task to integrate research knowledge and real-world policymaking, REPOPA has been closely working together in inclusive ways with various societal actors on different levels (researchers, citizens, policy makers, third sector organizations) during the whole project period.
The REPOPA research project has been programmatic, meaning that research objectives, design and methods were not developed in detail a priori but they were developed based on the local needs and together with the stakeholders of the policies. In work package No3 tailored interventions to enhance evidence-informed policymaking was the main objective; the interventions also showed that they helped policymakers to better align both the process and its outcomes with the values, needs and expectations of their communities.
In addition to carrying out research, via its web-based internet platform with national platforms, REPOPA has also built structures for inclusive collaboration to increase sustainability, multi-actor and public engagement in research and innovation. These platforms as well as tools developed such as policy briefs and checklists for evidence-informed policymaking indicators enable easier access to scientific results.
A special feature in REPOPA has been that it has used computer software of Readability Index (www.readability.com) to test and improve its dissemination material for lay audiences.
Further, REPOPA work packages No2, No3 and No4 all cherished public engagement by iterative and participatory multi-actor dialogues to foster mutual understanding and co-create research and innovation outcomes.
Examples of REPOPA work and products with societal impact
Phase 1; work package No1: Having the existing policies as study object and interviewing the persons involved in the preparation of the policymaking, increased the applicability of the results and made it easier to communicate the findings. Scientific papers presented in the scientific conferences informed the research community of the findings of the research and the lay-type articles informed policymakers and wider the community of the results. While there may be some instrumental use of key findings and recommendations in future policy making in countries, a conceptual use of the result findings can be thought to be achieved at least during the interviews conducted with persons involved in the preparation of the policies concerned. Contacts made when interviewing these people set the ground for future, closer collaboration with them. These contacts are especially important for the junior researchers that did not yet have collaboration with the policymaking institution
Phase 2; policy games, work package No2: To achieve more effective and sustainable public health policy, work package No2 cross-sector collaboration and evidence-use in the policy process are essential preconditions. To stimulate both cross-sector collaboration and evidence use, work package No2 developed a real-life based policy game and evaluated it in three EU countries. In each country multiple societal actors such researchers, local policy makers and other relevant actors participated in preparing, designing, adapting and finally playing the policy game. Further, the policy game focused on an actual local policy issue such as developing and implementing physical activity policy in a specific neighborhood and thus involved various local actors relevant to this policy; besides researchers and policy makers also schools, community councils, care organizations, sports organizations for instance.
Among participants the policy game event (a one day meeting with real life societal actors playing in teams) raised awareness on and provided insights into the roles and positions of all organizations involved, the dynamics of the policy development process, the benefits of collaboration and use of evidence and expertise from the network. The policy game provided participants a safe learning environment closely resembling the real-life context of policymaking. It increased mutual understanding among participants, which in turn stimulated collaboration and trust between the organizations involved and had potential to influence the take-up and governance through actions bringing about institutional change among the policy game participants.
A policy game has shown to be an innovative method to support and realize cross-sectoral integrated health policy at the local level to become implemented and sustainable. It offers the opportunity to various societal organizations to learn how to collaborate and benefit from their organizational network and its expertise and available evidence. It thereby contributes to the co-creation of a shared knowledge base for local public health.
Phase2; Stewardship interventions, work package No3
This work package had the overall aim of finding ways to collaborate between policymakers and researchers that would create win-win situations. For policy makers the win could be to develop more evidence-informed policies and for researchers to carry out more policy/practise relevant research. The activities were based on tailor-made interventions for each setting, thus taking the context into account. The interventions were also built on policy formulation phase of real policies, which was the most appropriate phase of the policy cycle to bring in research knowledge. Using real policies meant that interventions were participatory, salient for the context, built upstream and the research carried out was societally relevant. The stewardship-based locally tailored interventions increased public engagement and also created the space for ethical value-laden issues such as including vulnerable groups. Like this the interventions enhanced inclusiveness, transparency, diversity, and creativity.
The close collaboration between the two parties in tailor-made interventions is a new strategy for achieving successful integration of research and policy and practice. The work package No3 emphasized having an equal participatory approach, where researchers and policymakers took part in the process with the same amount of power. And it worked! In general, the policymakers did strengthen the use of research evidence together with other sources of knowledge such as from stakeholders and policy target groups. Examples of the target groups benefitting from the work package No3 interventions are in general vulnerable groups in Denmark and elderly and their physical activity needs and possibilities both in Denmark and The Netherlands, and children and their physical activity in Italy.
The work package developed tools that can be used for making organizational change beyond the project time. These tools can help to carry out context analysis, stakeholder analysis, and logical frameworks for enhancing participatory policymaking and organizational change in knowledge integration. The tools can also help to involve knowledge from the policy target groups through participation of these groups in the policymaking. This can enhance for example taking into account the needs of different socio-economic groups in policymaking. The interventions carried out improved and enhanced longer lasting collaboration relationships between researchers and policymakers. The policymakers have expressed that this kind of collaboration is important to them since after them they feel more comfortable to contact researchers for different information needs in the future.
Phase 3; developing indicators for evidence-informed policymaking; work package No4
The work package No4 drew together the previous REPOPA results concerning knowledge translation from research to policymaking and tested, validated and negotiated these results among stakeholders to finally formulate them into usable tools such as indicators for evidence-informed policymaking.
The methodology adopted to reach this aim was a modified Delphi study that led to the co-production of knowledge by means of collaboration between selected policymakers and researchers in the field of health and cross-sectors. Their contribution allowed integrating, validating and contextualizing a list of indicators for evidence-informed policy making, aimed at assessing if and at what extent a policy is informed by evidence.
The indicators, organized in thematic domains (human resources, documentation, communication and participation, monitoring and evaluation), include issues that foster societal impact such as multiple directions of communication exchange, inclusion of the variety of evidence from research results to stakeholders and civil society and to cross-sector policymaking processes; and involvement of vulnerable groups.
Another issue related to societal impact addressed by the work package No4 is the international and local perspectives joined together by the Delphi study, that included both an international at distance stage (with 76 panellists) and a face-to-face stage conducted at national level (involving 103 experts that were asked to perform a SWOT analysis and mapping the indicators to policy phases). Moreover, this process led to the availability of the indicators in six different languages (Danish, Dutch, English, Finnish, Italian, Romanian), which strengthens their potential impact in different settings.
Further, work package No4 developed evidence briefs and checklists to be taken into account while using the indicators, to be included in guidance resources to promote the use of the indicators at national and European level. With these tools REPOPA work will raise awareness and science literacy. Some of these tools have already been circulated and tested among stakeholders at national level, and positive feedback has been received as policy makers are taking into account the work package No4 results. For example, the Italian Regional council member in charge for Health is presenting own experience in REPOPA work package No4 as a best practice in the international business convention Meet in Italy for Life Sciences 2016, which took place in Rome from the 25th to 28th October 2016.
Dissemination; work package No6
This work package had the overall aim to disseminate project outputs and outcomes targeting stakeholders and wider public, raising interest and awareness of the REPOPA activities and results. Collaborating closely with all work packages, part of work package No 6 has built structures for collaboration to increase the sustainability of evidence-informed policymaking. The work package pursued this in the local and national environment of REPOPA partner countries and in the online environment of the project REPOPA website. In the local and national environment REPOPA partners reached out to existing platforms, e.g. Finland, or created their own platform, e.g. Denmark to further encourage the use of evidence-informed policymaking. In the online environment REPOPA created the opportunity to share up-to-date information, documents to be discussed, and debated with different stakeholders on the country specific pages and open discussions using twitter or a country discussion box, both included in the website, the most popular, sustainable and cost-effective being the use of already created social media environments, in our case, twitter.
Consequently, most of the impact comes from the communication activities created and promoted to make the research activities known to multiple audiences ranging from the science field via peer-reviewed publications and conferences to lay people via project poster and flyers, but most importantly, project evidence briefs and animation videos. The evidence briefs were developed to disseminate and inform policymakers and researchers about the REPOPA interventions and research, having both print and online versions while the animation videos were intended to describe in lay terms the policy interventions tested in the project and can be used for teaching purposes.
The final symposium of REPOPA, which was arranged in Brussels in the last project month, was a showcase for the project output and contribution. Representatives of the six European REPOPA countries and international organizations including EC, WHO, European Implementation Network, sport organizations and so on actively brought their feedback to the project and ideas for further developments at the symposium and in the world café as a part of it. The symposium participants can further function as advocates of the main message of REPOPA. A positive sign of the interest in the REPOPA symposium and message was a marked increase in twitter messages as well as in REPOPA website visits during and after the symposium. REPOPA website will also be kept active to update the publication list and main REPOPA-related activities. A spin-off in the Dutch policy game has received much interest also internationally.
Evaluation; work package No5
The purpose was to improve the performance and results of the REPOPA project through monitoring and evaluation. Project members were consulted to design the evaluation, thereby encouraging the uptake of formative findings by Consortium members. Project progress and collaboration within the Consortium were monitored and the findings were presented with an annual report with action recommendations. This continual improvement approach helped inform project decisions about how to work more effectively as a Consortium. At the end of the project, the project’s synergies and impact were evaluated and it was examined how relevant REPOPA’s evidence-informed outputs are for science and policy.
The evaluation showed that REPOPA achieved societal impact in several ways:
Junior researchers involved in the project improved several skills they can bring to future evidence-informed policymaking research. Members of the REPOPA team broadened and deepened their relationships with physical activity stakeholders within municipalities and at national and international levels. They successfully used these networks and policy platforms to share REPOPA findings and new approaches for evidence-informed policy-making with decision-makers and researchers. The project work packages used robust scientific methods in their research activities that will be of interest to and can be replicated by other researchers working on similar health issues. REPOPA tested two interventions – a Stewardship approach and a policy simulation game. Although these need further examination in other contexts and for other health issues, they appear to be promising approaches to enhance the use of evidence in the inter-sectoral policymaking process. The measureable indicators developed through the Delphi approach will be useful for those evaluating the use of evidence in policy-making organizations.
REPOPA’s examination of evidence-informed policy making for inter-sectoral policies is particularly timely because these types of policies are of concern to many countries. REPOPA’s outputs may continue to spread through the informal networks and channels for sharing evidence that were created during the project.
Conclusion social innovation and societal impact of REPOPA
REPOPA is an innovative and one of the few research projects in the health promotion area to study real-word policymaking processes in a variety of contexts and in ways that deliberately involved policymakers in research activities. To be able to support relevant and salient policymaking, a rather ambitious programmatic research approach was implemented. This meant carefully studying the contexts and needs before designing further steps such as interventions. Though this approach is challenging in terms of time pressure, overlapping tasks and especially in terms of coping with real life changes in the political environment, REPOPA reached its research objectives and was also able build sustainable structures for evidence-informed policymaking.
The societal impact pathways are in a way inbuilt in the REPOPA project, since actual policymakers and other stakeholders were involved in co-creating new knowledge on how best to collaborate and enhance integration of research knowledge and real-world policymaking.
Further, by carrying out research and tools development in six European countries representing different European regions, created good conditions to do knowledge transfer across societies. This increased knowledge can later facilitate potential European efforts in future policymaking by providing insight information of the different local contexts and their cultures in the policymaking area. The web-based umbrella platform and national platforms for evidence-informed policymaking, which REPOPA developed, can function as sustainable structures to continue collaboration between researchers and stakeholders in locally relevant ways.
In terms of scientific impact, REPOPA contributed to methods development, especially in the areas of using policy game as a vehicle for knowledge co-creation, in terms of applying locally tailored, Stewardship-based interventions, and in terms of developing and contextualizing measurable indicators for evidence-informed policymaking. Further, REPOPA contributes to the refinement of the theoretical framework for evidence-informed policymaking, especially for operationalizing the framework for indicator development.
REPOPA increased its societal impact by producing usable tools for multiple audience dissemination of research findings and good practices via its website, policy briefs and animations. Also, via its junior researcher involvement and training, REPOPA contributed to the next generation competence building in the area of evidence-informed policymaking.
European added value of REPOPA
REPOPA is the one of few international research projects with the objective to increase evidence-informed policymaking by integrating scientific research and actual real-world policymaking by including European countries from different regions with different societal structures and resources.
REPOPA brings innovation and added value in several ways: First, it is programmatic meaning that its design and methods were not decided in beforehand as requested in the prevailing, traditional research paradigm in public health. Instead, to enable context-relevant outcome, REPOPA combined the best available research evidence and recent theoretical frameworks with the policy stakeholder, target group and context analysis as the basis for tailored interventions. This also means that even if the interventions had the same overall goal to increase evidence-informed policymaking and also used the same outcome measures, the intervention contents, process and intensity varied according to the context. Fidelity in REPOPA was understood as adhering to the theory instead of the intervention process and contents. The comparison across contexts was assessed as the increase of the evidence-informed policymaking in each context relative to the baseline in that context.
Further, the REPOPA interventions happened in real (not only realistic) policymaking contexts and/or real policymakers collaborating with researchers. This means that political changes and other practice-related issues and priorities influenced the intervention process. This also meant that the know-how gained from the interventions is context-relevant and enhances later implementation.
This approach offered a win-win relationship between researchers and policy and practice groups- an issue seldom seen- not least due to differing world views and work practices. REPOPA took this challenge – and can confidently state the positive impact seen already in the course of the project in changing researcher-policymaker collaboration, networks and practice.
To learn more in detail of the international and multidisciplinary, programmatic research process of five years, process analysis and also impact analysis was carried out in the project.
Finally, REPOPA intervention findings were formulated, validated and contextualized in six European countries into evidence-informed policymaking indicators and guidance tools together with (inter)national stakeholders. So REPOPA finished its project work by providing also practical tools and resources, which can be used and refined to fit other contexts. Even if REPOPA used physical activity policymaking as its research theme, the findings and tools developed can be applied also in other areas, especially in the areas of lifestyle, social and environmental policy development.
Recommendations and lessons learnt in the course of REPOPA
The recommendations presented are for researchers in other programmatic and multi-country research teams, and for future research projects funded by the EC and other funding agencies.
Recommendations for researchers
- Balance scientific and lay dissemination priorities in the project schedule and provide resources for both in the budget.
- Provide adequate resources to develop targeted, tailored materials for policymakers. Producing effective materials for lay audiences takes time and resources, and possibly outside expertise.
- Commit additional project time (person-months, project duration) and financial resources to achieve programmatic, cross-site synergies.
- Deliberate team science strategies are needed to support and maximize synergies and cross-learning in multi-site Consortia projects.
- Consider how and what trainee (junior researcher) capacities can be enhanced through a multi-site project Consortium.
Recommendations to the EC and other research funding agencies
- Funding agencies should consider how they could provide the longer term research support required for research teams to build relational capital with decision-makers, to adapt and implement interventions and to conduct longer-term post-intervention follow up.
- Ensure that the criteria used for the peer review of scientific relevance are explicit in their inclusion of non-quantitative research designs such as case study methods.
- Build structures for potential harmonized ethics clearance practices in social science and policy research for EC funded multi-country studies.
- Encourage project teams to include costing estimates as part of intervention studies to better inform feasibility considerations.
- Encourage the explicit use of gender analysis within studies to build strength in this area.
- Consider use of the model of an embedded evaluation team for other EC-funded projects.
- Identify best practices for this evaluation approach across EC-funded projects.
Recommendations for scientific journals
- Journals could be more open to publishing implementation research reports form natural settings even if they do not always adhere to the prevailing paradigm of strict control of research context.
Future dissemination activities planned after the project period
First, the REPOPA project website with its umbrella platform for evidence-informed policymaking and updates of publications and other dissemination activities (e.g. conference and meeting presentations) will be kept active, including twitter, and updated regularly. The website will be managed together by BBU and SDU.
REPOPA data storage will be centrally and safely organized by SDU to safeguard the data and compliance to good practice and ethical rules in the use of the data gathered. Active publishing is also encouraged and facilitated by the created publication plans and guidelines by using the internal SharePoint site run by SDU.
A meeting report of the Final Invitational Symposium of the REPOPA project will be written and circulated widely. This circulation will include also the lessons learned in the world café session organized involving European country representatives, EC representatives, representatives of international organizations and those of research institutions.
Junior training is continued by ongoing PhD work; other kind of training such as MSc work is also encouraged.
Dissemination and implementation of the research findings, lessons learnt, best practices as well as tools such as indicators for evidence-informed policymaking, will be continued internationally, nationally and locally. In addition, submission of letters (see e.g. Aro 2016), viewpoints, debate articles and so on will be encouraged to keep the discussion going on in enforcing evidence-informed policymaking. Also further grant funding will be sought to carry out implementation studies on the research findings and tools developed in the project in different contexts.
Collaboration with the networks and relationships established in the project time will be continued and scaling-up of the collaborative policy development of the REPOPA interventions will be enhanced.
Special attention will be put on the efforts to use lay language (with software checks on readability) in the dissemination both orally with media and with different societal stakeholders and in the form of newsletters, lay articles, leaflets, workshops and meetings to be arranged to disseminate the REPOPA work.
Future steps and needs in research in evidence-informed policymaking
Areas of further research from the project’s research work packages may include:
- What are the roles of different policy actors and how can they improve the research integration process?
- How can we improve transparency of research use in policymaking?
- How sustainable are the findings of the interventions carried out in work packages No2 (policy game) and No3 (Stewardship-based tailored interventions) and how could the sustainability be improved?
- What are the implementation challenges in different contexts of the tested interventions (both work package No2 and No3) and of the indicators developed for evidence-informed policymaking?
- What are the competences and needs of both researchers and policymakers in evidence-informed policymaking?
- How best to improve policy/evidence briefs, understanding and readability among different stakeholder groups participating in evidence-informed policymaking?
Areas for further research from the project evaluation perspective may include:
- What is the combined impact of the policy game and the stewardship approach, particularly in settings where cross-sectoral collaboration is very weak (i.e. vertical silos of decision-making are strong)?
- Do those involved in the evidence-informed policy interventions such as those tested in the REPOPA project replicate/apply their learning and approaches to other policies and domains of work?
- What organizational structures are required to support these kinds of approaches?
- Does the one-day intervention of a policy game provide the necessary “force” to change collaborative behaviour and decision-making processes? How might it complement other approaches to engage policy makers such as those involved in deliberative dialogues?
References
Aro AR. Towards evidence-informed policymaking in physical activity. Journal of Preventive Medicine 2016;1;3:18
Aro AR, Smith J, Dekker J. Contextual evidence in clinical medicine and health promotion. European Journal of Public Health 2008;18:548-549.
Aro AR, Bertram M, Hämäläinen R-M, Goor LAM van de, Skovgaard T, Valente A, Castellani T, Chereches R, Edwards N on behalf of the REPOPA Consortium. Integrating research evidence and physical activity policy making- REPOPA project. Health Promotion International 2016;31:430-439.
Aro AR, Radl-Karimi C, Loncarevic N, Bertram M, Joshi R, Thøgersen M, Pettersen CLH, Skovgaard T, van de Goor I, Spitters H, Valente A, Castellani T, Cori L, Janse, J, Dorgelo A. and Pos S. on behalf of the REPOPA Consortium (2015). Stewardship-based intervention: WP3 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project. http://www.repopa.eu/sites/default/files/D3.2.Report_Stewardship%20based%20intervention.pdf
Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evidence and Policy 2010; 6:145-159.
Bertram M, Radl-Karimi C, Loncarevic N, Thøgersen M, Skovgaard T, Jansen J, Castellani T, Dorgelo A, Valente A, Cori L, Aro AR. Planning locally tailored interventions on evidence informed policy making- Needs assessment, design and methods. Health Systems and Policy Research 2016;3;2:15
Bertram M, Loncarevic N, Castellani T, Valente A, Gulis G, Aro AR. How could we start to develop indicators for evidence-informed policy making in public health and health promotion? Health Systems and Policy Research 2015;2;2:14
Bolger F, Stranieri A, Wright G, Yearwood J. Does the Delphi Process Lead to Increased Accuracy in Group-Based Judgmental Forecasts or Does It Simply Induce Consensus amongst Judgmental Forecasters? Technological Forecasting and Social Change 2011;78;9:161671–80.
Bolger F, Wright G. Improving the Delphi Process: Lessons from Social Psychological Research. Technological Forecasting and Social Change 2011;78;9:1500–1513.
Bouldekid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE 2011;6:6
Bowen S, Zwi AB. Pathways to "Evidence-Informed" Policy and Practice: A Framework for Action. PLoS Medicine 2005, 2;7:600-605.
Borgatti, S.P. 2002. NetDraw Software for Network Visualization. Analytic Technologies: Lexington, KY. Retrieved from: https://sites.google.com/site/netdrawsoftware/download
Borra´s S. Policy learning and organizational capacities in innovation policies. Science and Public Policy 2011;38:725–734.
Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annual Review of Public Health 2009;30:175-201.
Chereches RM, Rus D, Aro AR, Loncarevic N, Bertram M, Ballav Adhikari T, Valente A, Tudisca V, van de Goor LAM, Spitters HPEM, Hämäläinen RM, Ståh, T, Edwards N, Roelofs S, Anderson CS, Lau CJ, on behalf of the REPOPA Consortium (2016). Report on dissemination, monitoring and research into policy repository: WP6 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project (submitted)
Ciliska, D, Thomas, H, Buffet, C, 2012, An Introduction to Evidence-Informed Public Health and A Compendium of Critical Appraisal Tools for Public Health Practice (Revised). Hamilton, National Collaborating Centre for Methods and Tools.
Castellani T, Valente A, Cori L, Bianchi F. Detecting the use of evidence in a meta-policy. Evidence and Policy 2016;1;.12.
Delborne J, Schneider J, Bal R, Cozzens S, Worthington R. Policy pathways, policy networks and citizen deliberation: Disseminating the results of World Wide Views on Global Warming in the USA. Science and Public Policy 2013;40;3:378–392.
De Leeuw, E, Clavier C, Breton E. Health policy – why research it and how: health political science. Health Research Policy and Systems 2014;12:55.
Duke RD, Geurts, JLA. Policy games for strategic management. Pathways into the unknown. Amsterdam: Dutch University Press, 2004.
Eklund Karlsson L, Winge Jakobsen M, Winblad Heiberg M, Aro AR. Involvement of external stakeholders in local health policymaking process: a case study from Odense Municipality, Denmark. Evidence & Policy: A Journal of Research, Debate and Practice, available online since 17 April 2016. DOI: 10.1332/174426416X14609162710134.
Eklund Karlsson L, Haaber Pettersen C and Aro AR. Governance and Youth Participation in local health policy making - Case study Odense and Esbjerg, Denmark. (under preparation)
Edwards N. Advancing knowledge translation: Opportunities for mutual learning between Canada and United States, Presentation to National Cancer Institute, Bethesda, Maryland. 2012
Edwards N, Di Ruggiero N. Exploring which context matters in the study of health inequities and their mitigation. Scandinavian Journal of Public Health 2011;39 (Suppl):43-49.
Edwards N, ViehbeckS, Hämäläinen R-M, Rus D, Skovgaard T, Goor LAM van de, Valente A, Syed AM, Aro AR. (2013). Challenges of Ethical Clearance in International Health Policy and Social Sciences Research: Experiences and Recommendations from a Multi-Country Research Programme. Public Health Reviews (Online), 34(1).
Edwards, N., Roelofs, S., and Anderson, C. (2016). WP5: REPOPA Evaluation and Impact Report: WP5 final report of University of Ottawa, Ottawa, Canada.
Fischer ARH, Wentholt MTA, Rowe G, Frewer LJ. Expert involvement in policy development: A systematic review of current practice, Science and Public Policy 2014;41:332–343.
Fletcher J, Marchildon GP. Using the Delphi Method for Qualitative, Participatory Action Research in Health Leadership. International Journal of Qualitative Methods 2014;13:1–18.
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrook PA, Vogt TM. Evaluating the overall impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Education Research 2006;21:688-694.
Geurts, JLA, Duke, RD, Vermeulen, PAM. Policy gaming for strategy and change. Longe Range Planning. 2007, 40, 6, 535-58
Goor LAM van de, Quanjel M, Spitters H, Swinkels W, Boumans J, Eklund Karlsson L, Aro AR, Jakobsen MW, Koudenburg OA, Chereches R, Sandu P, Rus D, Roelofs S, Lau CJ, Glümer C, Jørgensen T, Jansen J, Dorgelo A, Pos S on behalf of REPOPA Consortium (2015). In2Action: Development and evaluation of a Policy Game Intervention to enhance evidence-use in HEPA policy making: WP2 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project. http://www.repopa.eu/sites/default/files/D2.2.%20Report_Game%20simulation%20intervention.pdf
Goor LAM van de, Hämäläinen, R-M, Syed AM, Juel CL, Spitters, HPEM, Eriksson Karlsson L, Rus D, Valente SA, Castellani T, Aro AR. What hinders integration of evidence in public health policy: Differences in worlds of policy makers and researchers? –Results from the REPOPA study in six EU member states (re-submitted).
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? The Journal of Continuing Education in the Health Professions 2006;26;1:13-24.
Green LW, Glasgow RE. Evaluating the relevance, generalization and applicability of research. Evaluation & The Health Professionals 2006;29;1:126-153.
Guidance Note for Researchers and Evaluators of Social Sciences and Humanities Research 2010. http://ec.europa.eu/research/participants/data/ref/fp7/89867/social-sciences-humanities_en.pdf
Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technological Forecasting and Social Change 1996;53;2:185–211.
Hawe P, Shiell A, Riley T. Complex interventions: ‘how out of control’ can a randomized controlled trial be? British Medical Journal 2004;328:1561-1563.
Hämäläinen RM, Villa T, Aro AR, Fredsgaard MW, Larsen M, Skovgaard T, van de Goor LAM, Spitters HPEM, Chereches R, Rus D, Sandu P, Bianchi F, Castellani T, Cori L, Valente A, Edwards N, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, Wichbold C, Cavill N, Dorgelo A, and Jansen J. (2013). Evidence-informed policy making to enhance physical activity in six European Countries: WP1 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project http://www3.sdu.dk/multi/repopa/sites/default/files/D1.1_Role%20of%20evidence%20in%20policy%20making_14062013.pdf
Hämäläinen, R-M, Aro AR, Goor LAM van de, Lau CJ, Jakobsen MW, Chereches RM, Syed AM & REPOPA Consortium. Exploring the use of research evidence in health-enhancing physical activity policies. Health Research Policy and Systems 2015;13:43.
Hämäläinen R-M, Aro AR. Ohjelmallinen tutkimusprosessi politiikkatoimien tutkimuksessa- tapaustutkimuksena liikuntapolitiikka ja tutkimustiedon käyttö. (Programmatic research process in policy research- physical activity policy and use of research knowledge. Hallinnon Tutkimus 2015; 34;3:249-263.
Hämäläinen R-M, Aro AR, Juel Lau C, Rus D, Cori L, Syed A, the REsearch into POlicy to enhance Physical Activity (REPOPA) Consortium. Cross-sector cooperation in health-enhancing physical activity policymaking: more potential than achievements? Health Research Policy and Systems 2016;14:33. (a)
Hämäläinen R-M, Sandu P, Syed AM, Jakobsen MW. 2016. An evaluation of equity and equality in physical activity policies in four European countries. International Journal for Equity in Health 2016 (in press). (b)
Jakobsen, MW, Lau, CJ, Skovgaard, T, Hämäläinen, R-M, Aro, AR. Use of research evidence in three cases of local policymaking on physical activity (under revision)
Kingdon, JW, 2010, Agendas, alternatives and public policies (2nd edn), Harlow: Longman Classics.
Knott J, Wildawsky A. If dissemination is a solution, what is the problem? Knowledge, Creation, Diffusion, Utilization 1980;1:537-538.
Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Administration Review 2003;63;2:192-205.
Linstone, HA, Turoff M. The Delphi Method. Techniques and applications 2002:53.
Litman T. Developing Indicators for Comprehensive and Sustainable Transport Planning Victoria Transport Policy Institute 2011. http://www.vtpi.org/sus_tran_ind.pdf
Nuffield Council: Public Health: Ethical Issues. Nuffield Council on Bioethics 2007. http://nuffieldbioethics.org/wp-content/uploads/2014/07/Public-health-ethical-issues.pdf
Okoli C, Pawlowski SD. The Delphi Method as a Research Tool: An Example, Design Considerations and Application. Information & Management 2004;42(1):15–29.
Reis RS, Salvo D, Ogilvie D, Lambert E, Goenka S, Brownson RC. Lancet Physical Activity Series 2 Executive Group. Scaling up physical activity interventions worldwide: stepping up to larger and smarter approaches to get people moving. The Lancet 2016;24;388(10051):1337-48.
Rowe G, Wright G. The Delphi Technique: Past, Present, and Future Prospects — Introduction to the Special Issue Technological Forecasting and Social Change 2011;78 (9): 1487–90.
Satterfield JM, Spring B, Brownson RC, Mullen EJ, Newhouse RP, Walker BB, Whitlock EP. Toward a transdisciplinary model of evidence-based practice. Milbank Quarterly 2009;87:368-390.
Simonsen J, Bærenholdt JO, Buscher M, Scheuer JD. (Eds). Design Research: Synergies from Interdisciplinary Perspective. 2010. Routledge, London.
Skovgaard T, Nielsen MBD, Aro AR. Evidence in Health Promotion and Disease Prevention, Sundhedsstyrelsen (Danish National Board of Health), Dec 2007 http://www.sst.dk/Publ/Publ2007/CFF/Evidens_forebyggelse/Evid_Health_Prom_jan2008.pdf
Spitters HPEM, Lau, CJ, Sandu, P, Quanjel, M, Rus, D, Glumer, C. Oers, JAM van, Goor, LAM van de. Stakeholder network analyses in local public health policy making in three countries (under revision)
Straus SE, Holroyd-Leduc J. Knowledge-to-action cycle. Evidence-Based Medicine 2008;13(4):98-100.
Syed AM, Hjarnoe L, Aro AR: The Delphi Technique In Developing International Health Policies: Experience From The SARS Control Project. The Internet Journal of Health 2009;8;2.
Valente A, Castellani T, Larsen M, Aro AR. Models and visions of science–policy interaction: Remarks from a Delphi study in Italy, Science and Public Policy 2015;42:228–241.
Valente A, Tudisca V, Castellani T, Cori L, Bianchi F, Aro AR, Syed A, Radl-Karimi C, Bertram M, Skovgaard T, Loncarevic N, van de Goor I, Spitters H, Jansen J, Swinkels W, Ståhl T, Chereches RM, Rus D, Bozdog E, Sandu P, Edwards N, Roelofs S, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, on behalf of the REPOPA Consortium (2016). Delphi-based implementation and guidance development: WP4 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project (submitted)
Walt G, Shiffman J, Schneide, Murray SF, Brugha, Gilson L. “Doing” health policy analysis: Methodological and conceptual reflections and challenges. Health Policy and Planning 2008;23;5: 308–317.
Wehrens R, Bekker M, Bal R. Coordination of research, policy and practice: A case study of collaboration in the field of public health. Science and Public Policy 2011;38:755–66.
Weiner BJ. A theory of organizational readiness for change. Impelmentation Science 2009;4:67.
World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013 Nov 27;310(20):2191-4.
List of Websites:
REPOPA information and partners
REPOPA website: www.repopa.eu
REPOPA email: repopa@health.sdu.dk; REPOPA Twitter: ProjectREPOPA
REPOPA Coordinator: Prof. Arja R Aro, email: araro@health.sdu.dk; Phone: +45 6011 1874
REPOPA coordinating institute: University of Southern Denmark; www.sdu.dk;
REPOPA Consortium partners*
- National Institute for Health and Welfare (THL), Finland; leader of WP1;
- Tilburg University/Tranzo, the Netherlands; leader of WP2;
- University of Southern Denmark (SDU), Denmark; leader of WP3 and WP7;
- National Research Council of Italy (CNR), Italy; leader of WP4;
- University of Ottawa, Canada (uOttawa); leader of WP5;
- Universitatea Babes-Bolyai (UBB), Romania; leader of WP6;
- Research Centre for Prevention and Health (RCPH), Denmark; WP 1, 2, 3, 4 and 6 team member
- Dutch Institute for Healthcare Improvement ( CBO), The Netherlands; WP 1, 2, 3, 4 and 6 team member. REPOPA partner until October 12 2015.
- Herefordshire Primary Care Trust (HPCT), United Kingdom; WP 1, 4 and 6 team member. REPOPA Partner until March 31st, 2013.
The EC funded REPOPA project found practical ways to support policymakers in their use of research evidence in developing physical activity policies. Simulation policy game, locally tailored policymaking interventions, indicators to assess policies’ evidence-base as well as national internet platforms are examples of the REPOPA products.
REPOPA stands for REsearch into POlicy to enhance Physical Activity; a five year project (October 2011-September 2016, www.repopa.eu). The project was funded by European Union, 7th Framework Program. Researchers from six EU countries of Denmark, Finland, Italy, the Netherlands, Romania, UK, plus Canada were project partners.
Why do we need integration of research and policymaking? Researchers, European Commission, World Health Organization and others provide recommendations to policymakers to take physical activity seriously and integrate it in everyday lives of people. This is more easily said than done! Policymakers juggle with numerous pressures, points of views and resources. Research evidence presented in scientific publications is not the primary driver of their work; instead, political pressures, local priorities, values and resources influence their decisions. REPOPA researchers took the challenge to work hand-in-hand with real-world policymakers to learn how best to co-create policies so that research evidence is taken into account. In other words, REPOPA used the evidence-informed policymaking approach, meaning that research evidence is NOT the primary driver of decision making; instead local contexts, resources and needs are taken into account as well.
REPOPA found several issues researchers can help policymakers with: available and applicable research when needed and getting rid of academic jargon. More should be done to build procedures to gather research knowledge and to have meetings with researchers. Games are nowadays an attractive way to learn: simulation policy game proved to increase mutual understanding between policymakers and researchers; it also showed the importance of organizational and leadership collaboration across sectors. The practical outcome here is a model for policy games which is already applied outside REPOPA.
REPOPA researchers also joined real world physical activity policy groups and helped them use research evidence. These locally tailored interventions helped policymakers to take into account needs and values of the people whom the policies concern. They also triggered politicians to request for more research evidence. Here the practical outcomes are best practice interventions in three countries to co-create evidence-informed policies locally.
REPOPA was not only an academic project; it also developed the findings into usable tools such as indicators for evidence-informed policymaking. These indicators function as a checklist in policymaking to assess how well the policy in question is actually evidence-informed. The project also built a web-based umbrella platform with country platforms for evidence-informed policymaking in physical activity. These platforms bring together different actors and sectors in the countries to network and collaborate in combining research evidence and real-life policymaking. REPOPA used physical activity as the theme in its work; however, the lessons learned and tools developed can be applied to other fields, especially in the area of healthy living.
Project Context and Objectives:
The aim
The Research into Policy to enhance Physical Activity project (REPOPA; www.repopa.eu) had the main aim to integrate scientific research knowledge (=research evidence), expert know-how and the real world policymaking processes. This was done to increase synergy, build sustainable structures and best practices for future health promotion and disease prevention among Europeans.
There is strong evidence on health benefits of regular physical activity and how different physical environments influence active living (Reis et al 2016). Using research evidence can also help to build effective policies. However, there is still a need to get more knowledge on how to integrate research evidence in health enhancing physical activity (HEPA) policies. HEPA policies can vary from direct physical activity policies to those creating environments, which support physical activity.
Evidence-informed instead of evidence-based
When the REPOPA proposal was originally written in 2010, health promotion literature was debating if the clinical medicine term ‘evidence-based’, mostly dealing with individual patients, would fit the health promotion work, since the latter works on equal footing with local stakeholders with their priorities and values (e.g. Aro et al 2008). Evidence-informed policymaking has been suggested for the health promotion (and public health field), since in that work ‘research evidence only informs’ the decisions. This means that research evidence is NOT the primary driver of decision making in health promotion; instead adaptations to local contexts, resources and needs are needed and used (Brownson et al 2009, Ciliska et al,, Graham et al. 2006, Bowen, Zwi 2005). REPOPA took the term ‘evidence-informed’ to describe the approach in its integration of scientific research, expert know-how and real world policymaking (Aro et al 2016).
In line with the above, the theoretical frameworks guiding the REPOPA work have been those up-to-date in 2010 for this kind of work. In the course of the project, further theoretical development has happened, especially in more political science field (Borra’s 2011, Delborne et al 2013, Fischer et al 2014, Wehrens et al 2011) and especially in network analysis (de Leeuw et al 2014). These newer frameworks have informed e.g. article writing of the REPOPA findings. REPOPA had an objective to build a new framework to guide the development and use of indicators to assess if and how evidence-informed decisions or policies are. This new REPOPA indicator framework builds both on previous frameworks and on the work completed throughout REPOPA but especially in the work package No4 dealing with indicator development via Delphi process.
One of the central frameworks in REPOPA has been Evidence-informed Public Health (adjusted from Satterfield et al 2009, Figure 1, Annex) describing how best evidence-informed decisions happen when the best available research evidence, characteristics of the local population and context, including local priorities, resources, know-how and values, all are taken into account, and further, when all this happens respecting the cultural and environmental context. The other frameworks include Knowledge-to Action Cycle (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012), Figure 2, Annex), Stages of Knowledge Use (Knott, Wildawsky 1980, Landry et al 2003) with its seven steps of knowledge use have informed REPOPA, Systems Approach (Best, Holmes 2010) showing that policymaking is seldom a linear process; instead several environmental and social issues make the process more ‘messy’, even circular. Also Organizational Change theories (e.g. Weiner 2009) helped especially building the interventions in work package No2 on policy gaming to increase cross-sectoral collaboration. Though not an actual framework, Nuffield Council (2007) contribution on public health ethics related to Stewardship approach, has been very central for the whole REPOPA, but especially so for work package No3, which carried out locally tailored interventions on real world policies.
Why programmatic/phased research and what it meant
REPOPA applied so called programmatic or phased research design. This means that the whole research process with its protocols for interventions in successive work packages was NOT planned in detail beforehand (Aro et al 2016). On the opposite, each research phase and specific interventions were tailored to the local needs and knowledge gathered; first, knowledge gathered at the baseline, meaning beginning of the project, and then the knowledge of each phase informed the successive phases. This differs from the traditional, controlled, pre-agreed research design in the medical field, where designs, methods and interventions are registered (and often published) before starting the research project. In the evidence-informed work, interventions and tools are tailored to the context; thus context is not controlled for; it is an essential part of the interventions.
Programmatic research design was chosen since it enables the research process to be sensitive to each phase and context; thus all actions were tailored to the situation to gain maximal fit and maximize feasibility. The programmatic design also increased opportunities for REPOPA to create research synergies (Simonsen et al 2010) that added value to the project. On the other hand, this meant that there was limited time between different project phases to plan the next steps; for the researchers involved this meant that they were in parallel involved in wrapping up one phase and building already the next one (Hämäläinen, Aro 2015).
The REPOPA Consortium
The REPOPA Consortium brought together scientific researchers and experts from different disciplines, policymakers and other stakeholders from different societal sectors and from a variety of countries. Some of the REPOPA countries had longer history of working across sectors in health promotion, such as Finland, UK and the Netherlands; some countries had recently started this work (Denmark, Italy); in some, rather top-down traditional centralized system prevailed (Romania). Canada was very likely the most advanced in its explicit knowledge-transfer culture especially in the health care area; Canada was given the evaluator role in the REPOPA project. The structure of the Consortium per se enabled knowledge transfer between the partners; it also greatly added to understanding the importance and influence of context in the REPOPA research and tool development.
Objectives of REPOPA
The objectives of REPOPA were built so that in the course of five years in the varying contexts of six European countries we could a) analyze first the use of evidence in physical activity policies, b) then test different interventions to enhance evidence-informed policymaking, and c) based on these, to develop both tools for later use by policymakers and researchers such as indicators to assess if and how policies are evidence-informed, and policy briefs. So the idea was to deliver a full package of knowledge and practical tools to apply the knowledge (Aro et al 2016).
1st phase
In the first project year, work package No1 had the objectives to explore and analyze the ways evidence was being used in formulating the local, regional and national policies to enhance physical activity (Figure 3, Annex). Further, the objective was to identify the needs of the policymakers for the use of the evidence (facilitators and barriers) and whether these needs were being met by the research community. The third objective was to evaluate ways of delivering and communicating the evidence to the policymakers; and the fourth, to develop both a framework and indicators for the evidence informed policy development.
2nd phase
The 2nd phase of the project developed two sets of interventions to be tested each in three real world policy cases in three countries. One of the sets (work package No2) had the objectives to develop and use a policy game to study a) organizational and process aspects in cross-sector physical activity policymaking, more specifically, in leadership between organizations; b) organizational networks; c) communication and organizational change processes; and d) facilitators and barriers in evidence informed policy making.
The other set of interventions (work package No3) had the objective to study if the Stewardship approach increased the level of evidence-informed policymaking in selected, real-world policy cases. Stewardship approach was based on the needs, context and stakeholder analysis, and the aim was to increase integration of research evidence with other kinds of evidence in close academia-policy maker collaboration. Further, these interventions also aimed at finding ways to identify and effectively engage specific stakeholders in physical activity policymaking.
3rd phase
Building on work packages No 1-3, the 3rd phase of REPOPA (work package No4) had the objective to draw together the REPOPA results, developed methods, frameworks, indicators and best practices in the integration of research evidence and policymaking in physical activity. Further, the aim was to test, validate and negotiate the results among the stakeholders in REPOPA countries and beyond. Finally there was the aim to develop the output into guidance resources and evidence briefs.
Non-research objectives throughout the project period
Parallel with the four research work packages, REPOPA also had a clearly stated objective (work package No5) to monitor and evaluate the project working process, content, scientific products and added value of each work component as well as the synergies among work package component. Additionally, one aim was to apply a framework to analyze the impact of the project in terms of Reach, Effectiveness, Adoption, Implementation and Maintenance of the project activities (Green, Glasgow 2006). Also parallel with the other work packages a dissemination strategy was built with (work package No6) a detailed external communication plan aiming at ensuring the best visibility of the REPOPA activities and outcomes at EU level, targeting stakeholders and wider public, raising interest and awareness of REPOPA activities and results. Further, it had objectives to establish an umbrella platform to support country-specific evidence-informed policymaking platforms created in the course of the project, and also to develop a European advocacy plan and support producing policy briefs to enhance evidence-informed physical activity policymaking. Finally, the administrative objective (work package No7) was the Consortium management and overall coordination.
Project Results:
REPOPA results phase 1
The first year results in the REPOPA project confirmed that research evidence was not systematically or explicitly used in physical activity policymaking; it was other kinds of information and priorities, which were more central. In many cases it was not possible to identify the original research sources as they were mixed with other kinds of evidence, such as experiences, values, political statements, and ambitions. Interestingly, in this study, a sample of local-level policies seemed to use research evidence more explicitly than national-level policies. Examples of other kinds of information used in the HEPA policies were good practices, information from media (newspapers, internet), information about society (economics, traditions), political statements and community activity (values, inspirations and citizens´ opinions). (Hämäläinen et al 2015).
How was the data gathered: The data was gathered by carrying qualitative document and policy analysis using thematic contents analysis of 21 HEPA policies in six EU countries of REPOPA (Table 1, Annex) and by interviewing policymakers and experts (n=86) who had been involved in the preparation of the HEPA policy documents analyzed (Hämäläinen et al, 2013).
Barriers: The main barriers for the use of research evidence in HEPA policymaking were: traditions, certain bureaucratic and organizational factors as well as the nature of policy making processes and political interests, such as pursuit of voters and a value base, research not being available or applicable when needed, academic language being difficult, and no procedures to gather research knowledge or meet researchers. Furthermore, there was lack of time, of financial and human resources, lack of competences and of joint criteria for the use of research evidence in the policy processes (formulation, development, implementation, monitoring and evaluation). As a conclusion, the insufficient collaboration between researchers and policymakers in general was seen as a barrier for the use of research evidence in policymaking. (Goor van de et al, re-submitted).
Facilitators: Main facilitators were: support from administrative departments to use research evidence and an organizational culture that supported the use of research evidence. Other facilitators were achievable and relevant research which is also easy to apply in practice. Furthermore, an important facilitator was direct or institutional relationships between policymakers and researchers. The research bodies may function as knowledge brokers through open forums (such as seminars) or through direct relationships. However, it is crucial to find a common language between policymakers and researchers. (Goor van de et al, re-submitted).
Cross sector cooperation: Working across sectors happened by stakeholder involvement, governance structures and coordination structures and processes. Results provide insights into key governance structures and coordination mechanism as well as processes in HEPA policies in six European countries. Findings show that there are many different structures and ways of working across sectors and also mechanisms for involving stakeholders in HEPA policies. (Hämäläinen et al 2016a).
Italian team analyzed a meta-policy on health promotion with the aim of understanding the use of evidence and the knowledge conversion process in a multi-actor policy trajectory, studying the different sources of evidence and their interaction (Castellani et al 2016).
Case studies were performed to get deeper understanding of external stakeholders´ involvement in the (Danish) local municipal policymaking contexts (e.g. Eklund Karlsson et al 2016) and in youth involvement in Danish physical activity policymaking (Eklund Karlsson et al, in preparation). The first study concluded that politicians had substantial influence on the involvement of external stakeholders, allowing only few to contribute in a closed policymaking process. Main involvement was through personal contacts or regular hearings. The second study concluded that mechanisms to facilitate youth participation in policymaking in the case study communities were lacking. Thus, the Danish goal of improved participatory policy making at the local level was not met by the study communities, especially regarding youth participation.
Manuscripts are under revision e.g. on comparing Danish municipalities in the use of research evidence (Jacobsen et al, under revision).
Initial indicators for evidence-informed policymaking
The policy analysis and stakeholder interviews carried out by REPOPA helped to produce initial indicators, guided by frameworks, for evidence-informed policymaking. The indicator development was completed later in the project by work package No4. Please see the chapter for the suggested indicator framework of the REPOPA project.
Ethics clearance procedure of the project
The EC required ethics clearance according to each partner country rules and regulations and to have it done before data collection started. As REPOPA with its policy research aim, belongs to mostly social science field, the country-specific practices varied a lot. Finland and UK required clearance for the whole international project and for all its sub-projects. However, as the home country Denmark of the coordinating institute, SDU, did not have any procedures for this, ad hoc system needed to be created. An ethics road map for the Consortium was developed by the coordinating team; this road map provided guidance built upon international ethics guidance such as Helsinki Declaration (World Medical Association 2013) and EC draft guidance for social science research http://ec.europa.eu/research/participants/data/ref/fp7/89867/social-sciences-humanities_en.pdf
None of the EU countries carrying out the document analysis required ethics clearance, but for the stakeholder interviews it was necessary both in Finland and UK. Clearance was also needed in Canada for evaluation activities and in UK for the Delphi study for indicator development. Other countries needed to deliver an official certificate that no ethical clearance was needed; in some cases this proved to be hard to get since there was no existing agency for this kind of certificate.
The ethics clearance process took over half a year from the active data gathering of the first project phase. The Consortium published a scientific peer-reviewed paper on this experience (Edwards et al 2013).
Summary
The first REPOPA phase added to the knowledge base of evidence-informed policymaking by analyzing the evidence actually used in a variety of HEPA policies across six EU countries and facilitators and barriers to the use of evidence. This information can benefit future policy development. In the REPOPA project the know-how of the first project phase fed into the successive phases such as interventions and indicator development.
REPOPA results phase 2
Based on the first year findings, the project went further to develop two kinds of interventions to learn how policymakers could be helped in finding and using research evidence.
Policy games to enhance use of evidence in policy making
Work package No2 developed and tested an innovative intervention for HEPA policymaking. It developed a policy simulation game to stimulate use of evidence in local policymaking, especially to enhance collaboration between organizations. A simulation game that resembles the real world policy-making context offers a safe environment to experience and learn about rather complex cross-sector collaboration and use of evidence.
What is a policy (simulation) game?
A simulation game is an experimental approach stemming from organizational sciences where it has been used in organizational change contexts (Duke, Geurts 2004, Geurts et al 2007). A simulation game is built on a profound systems analysis (Spitters et al, under review). This analysis visualizes all relevant organizations involved, their roles and positions and most important, their main drive to be involved in this policymaking context. The main stakeholders can be selected from the systems analysis; these are the “models” for the main roles in the game; they are invited to play the game. So the game resembles the real world context but is also placed in a fictive municipality. It is a one day experience of practicing cross-sector policymaking.
How was the data gathered?
The policy game was carried out in The Netherlands, Denmark and Romania. The teams in each country designed the game, chose real policy cases, did systems analysis and prepared the performance. Policy games were adapted to national and local contexts to fit the organizational actors and structures.
The game was a one day activity in each of the three countries. In total 18-20 participants per country, resembling real world colleagues, participated. They played eight to nine different roles in the teams of two to three persons in the game, based on the main organizations involved in HEPA policy making in the specific municipality. The teams were asked to construct a municipal HEPA policy implementation plan filled with as many as possible HEPA interventions, organized with and supported by other stakeholders and based on (some type of) evidence (see Figure 4, Annex).
Measurements of the experiences from the participants were done by observations of the game and through online questionnaires on three points in time: before the game, directly after and at six months follow up.
Results of the policy games
Results of four aspects of collaboration in organizational networks are presented: of cross-sector collaboration, organizational change, leadership aspects and use of evidence. (Goor van de et al 2015).
Learning experiences of the game for participants showed that cross-sector collaboration requires purposeful communication, coordination and direction. Further, the results showed that game boosts collaboration in real world context, which this experience is useful to daily work, and that game brought positive attitude towards use of knowledge and expertise.
In relation to organizational network, there were increased insights in organizational network and local policy context and increased commitment to and understanding of policy process, increased insights in positions and roles of organizations and sectors, and increased insight in the potential of the organization’s own contribution in evidence-informed policymaking.
In relation to leadership aspects, it was learned that clear directions and both formal and informal leadership promoted cross-sector collaboration, increased awareness of the importance of finding a common language, showing vision, inspiring and involving others. Potential for change in intentions in the games both in Denmark and The Netherlands appeared to be more limited than in Romania in terms of promoting trust, finding common language, communicating visioning and creating an environment open to all opinions.
In relation to use of evidence, game participants recognized barriers in the use of evidence in local policymaking: stakeholders were not very acquainted with the use of evidence in policymaking, it was not clear how to apply theoretical or research-based knowledge in daily practice, how to weigh different sources of evidence, or what is most important, there was lack of time and lack of resources.
Summary:
Simulation policy game proved to increase mutual understanding between policymakers and different organizations involved. It also helped to see how important organizational and leadership support is for using research evidence especially in cross-sector collaboration. It also increased insights into HEPA policy processes, including organizations’ networks and stakeholder roles. There was a change of attitudes and behavior (intentions) towards collaboration, towards leadership aspects and use of knowledge in a positive way. The policy game also helped to show the participants the benefits of the use of expertise and various sources of evidence, for example from organizations in the local network; this stimulates the process of integration of evidence into the local policymaking context.
What was also learned was that a similar policy (simulation) game can be built and executed within three different EU countries. However, the potential of a simulation game appeared to differ across countries, mainly depending on differences in administrative structure and policy context.
Stewardship-based, locally tailored interventions
Why Stewardship interventions?
Stewardship approach (Nuffield Council 2007) in health interventions is considered an ethical public health approach. Stewardship approach gives the government a leading role; it is based on liberal thinking, which emphasizes guidance, enables structures, services, equity, and avoids coercion. This is in line with the Nordic welfare model and it fits well physical activity policymaking in Western Europe.
In practice, Stewardship approach is based on the needs, context and stakeholder analysis. In REPOPA the approach was used to run interventions with the aim to increase integration of research evidence with other kinds of evidence in close academia-policy maker collaboration. Further, these interventions also aimed to find ways to identify and effectively engage specific stakeholder groups in physical activity policymaking. In addition, the Reach; Effectiveness; Adoption; Implementation; Maintenance (RE-AIM) evaluation framework by Glasgow et al (2006) was adapted and used for evaluating the interventions.
How was the data gathered?
The six selected policy case studied dealt with HEPA policies. Each participating country team of Denmark, Italy and the Netherlands selected two policy cases based on a set of common criteria. In Denmark the interventions aimed at increasing cross-sector collaboration and strategic policymaking. In the Netherlands, the interventions aimed at strengthening neighborhood sports and physical activity planning and developing physical activity for those 65+. In Italy the interventions aimed at knowledge and research translation as well as information exchange related to Olympics of students and to healthy roads Pedibus activities.
Intervention contents were tailored to the context meaning that they had a common goal of increasing evidence-informed policymaking but different means of reaching it. Standardization thus happened for the common framework, function and process, but the intervention contents varied according to the local needs (Hawe et al 2004). Intervention contents varied from workshops to training sessions, networking, knowledge exchange and consultation (Table 2, Annex). The intervention length varied from six to 18 months, depending on the policy process in question. The instruments to measure the intervention effect included a set of common questions developed for the study based on the needs assessment done in the beginning of the interventions (Bertram et al 2016), on the use of research knowledge, stakeholder knowledge, needs and values of the policy target groups. The initial REPOPA indicators were included in the core instrument questions used in all intervention settings. In addition, process evaluation was carried out in all settings.
Results of the Stewardship interventions: The interventions increased the level of evidence-informed policymaking in the selected, real-world policy case studies in Denmark, Italy and the Netherlands. Especially, there was an increase in the access, requests and use of research knowledge, in using stakeholder knowledge as well as needs and values of the policy target groups. There was also a decrease in barriers for using research knowledge in policymaking (Aro et al 2015b).
Even if the type and size of the intervention studies did not allow quantitative analysis, there was a clear indication that different stakeholder groups became more aware (Valente et al 2015), more active in searching for knowledge and in networking, and more open to values of others. However, as the 12 months follow-up results showed, it was a challenge to provide sustainable results with not so intensive and often sporadic intervention activities placed in the busy normal working life of the participants). All in all, the initial results post-intervention and also the increased networking and appreciation of the local needs and values, provide a good starting point for further interventions, boosters, research and implementation programs. These kinds of needs-based interventions as well as collaborative policymaking of policymakers and researchers seem to be an effective way to enhance evidence-informed policymaking.
Summary: Stewardship-based interventions, which were tailored to local policy needs, proved to be useful for policymakers to learn to take the needs and values of the policy target groups into account. Also politicians learned to request for more research evidence. However, sustainability of the results needs to be supported by continued collaboration or booster sessions between researchers and policymakers. The instruments (questionnaires) developed to measure the outcome of the interventions very likely would also serve as general tools outside intervention evaluations to measure the developments and trends in evidence-informed policymaking.
REPOPA results 3rd phase
Developing indicators for evidence-informed policymaking
The aim of this phase was to draw together the results, methods, frameworks, indicators and best practices in the knowledge translation from research to policymaking in physical activity developed in the previous project phases. In this phase the idea was to test, validate and negotiate these results and products among the stakeholders from the participating countries and among representatives from other EU countries and to formulate into guidance resources and evidence briefs.
How was the data gathered?
The Delphi technique (Gupta, Clarke 1996, Linstone, Turoff 2002, Okoli, Pawlowski 2004, Fletcher, Marchildon was used; it is a qualitative analysis methodology that aims to extract the collective knowledge of a group of experts in the study of a complex system. The Delphi process was implemented in three stages: two 2014, Boulkedid et al. 2011, Syed et al 2009, Bolger, Wright 2011, Rowe, Wright 2011) internet-based Delphi rounds at international level and a third face-to-face Delphi stage at national level. The process included an internal consultation among the REPOPA researchers and literature analysis, summarizing findings from the previous phases, and identifying draft indicators related to evidence-informed policymaking as well as organizing preliminary indicators in thematic sets. The aim was to develop indicators which would be SMART (Specific, Measurable, Attainable, Relevant, Time-bound) allowing them to be used to (quantitatively) measure the level of policies being evidence-informed.
The two internet-based Delphi rounds involved 76 experts from researchers, policymakers and politicians in the field of health, physical activity and other sectors, coming from the six European REPOPA countries and international organizations, to evaluate and improve the thematic sets of indicators proposed by the REPOPA researchers (Figure 5, Annex). In the first round they were asked to answer an online questionnaire to rate the relevance and feasibility of the draft indicators, to comment their ratings and propose further indicators to be included. In the second round they were asked to answer again an online questionnaire to rate relevance and feasibility of those draft indicators, which had been evaluated in the first round but which had not reached consensus on high relevance and feasibility. Summary of the results gathered in the first round in terms of ratings and comments was provided in order to make the rating easy. In this round the participants were also asked to evaluate new draft indicators developed on the basis of their own suggestions.
The third Delphi stage, which was a face-to-face round, consisted of six national conferences, which took place in the six European REPOPA countries between January and March 2016. The aim was to contextualize at national and local level the indicators that had been evaluated as highly relevant and feasible in the internet-based Delphi rounds. National conferences were organized in a synchronized way following a common methodology, involving researchers and policymakers in the field of health, physical activity and across sectors, partly selected from the panel of the previous Delphi rounds. Before the national conferences, the participants in each country assigned the indicators into policy phases using an online questionnaire; this was done to see when in policymaking cycle each indicator was useful and relevant. At the conferences, the participants performed a SWOT analysis on the indicators (finding their Strengths, Weaknesses, Opportunities and Threats) and identified their barriers and facilitators; they also suggested recommendations to overcome the possible obstacles in using the indicators and foster evidence-informed policymaking. The used referral frameworks were models describing the evolving science and science-policy relationships and theories of knowledge production and exchange, especially the Knowledge-to-Action cycle (Straus, Holroyd-Leduc et al 2008, Graham et al 2006).
Results of the Delphi study to develop indicators for evidence-informed policymaking
The Delphi study produced a list of 25 measurable indicators and 8 further indicator themes/complex indicators, which are not (yet) measurable. The 25 indicators belong to the following four sets: Human resources – Competences and Networking, Documentation - Retrieval/Production, Communication and Participation, and Monitoring and Evaluation (Table 3, Annex). The eight indicator themes/complex indicators point to relevant and important issues in evidence-informed policymaking, but they need further development as indicators.
Fact sheets and policy briefs for policymakers
This REPOPA phase also produced fact sheets of the national conference output for each country. In addition, evidence briefs were developed for each country on the basis of the national conference results and tested for feasibility and usability among participants of the conferences. For the overall policy brief of the Delphi study, please see www.repopa.eu
Guidance resources
The results of the national conferences were also used to formulate a checklist to guide policymakers in using the indicators for evidence-informed policymaking. The checklist is a part of the guidance resources for policymakers and stakeholders on REPOPA indicators and their use. Guidance resources were structured as follows: 1. Short introduction (on evidence-informed policymaking and why the indicators were developed); 2. Development and contextualization of the measurable indicators as well as the final list of REPOPA indicators; 3. Short instructions on how to use the indicators and the checklist to guide the use the indicators; 4. Glossary, final notes.
Summary: Using Delphi process, the REPOPA project developed the findings into usable tools such as indicators for evidence-informed policymaking. These 25 measurable indicators can be used as a checklist in policymaking to assess if and how well the policy in question is actually evidence-informed. Further, evidence briefs were developed to summarize the evidence gathered for each of the six country settings of indicator development.
Dissemination in the REPOPA project
The dissemination goal of REPOPA was both to spread news about what had been done and developed in the project and to build sustainable structures to enhance evidence-informed policymaking so that it would continue after the project lifetime as well. The work started by developing a dissemination strategy with a detailed external communication plan that aimed at ensuring the best visibility of the REPOPA activities and outcomes at EU level, targeting stakeholders and wider public, raising interest and awareness of REPOPA activities and results. To be able to do this comprehensively, a stakeholder mapping was done in the beginning of the project. The stakeholder mapping was done in all partner countries, at a small scale, to identify in the first six months of the project, stakeholders, areas for future improvement, how can we transpose our findings to other health-related fields and indicators for dissemination. One of the findings reveals the importance of in-country platform as a way to translate findings to other health-related fields: “Create legitimate centers that can ensure a clear translation of research results in evidence for policymaking”.
A special task for dissemination in REPOPA was establishing a web-based umbrella platform to support country-level evidence-informed policymaking platforms created in the course of the project (See photo 1 in the Annex, http://repopa.eu/content/eipm-umbrella-platform). Each partner could either search for in-country work group or launch a forum that had an interest in evidence-informed policymaking on local, regional or national level. An example is Finland, where a common platform for HEPA policy discussion was established by the working group in conjunction with others that allowed access to information on physical activity within the Ministry of Education and Culture. Other examples come from e.g. Denmark, where the platform took the form of an interest group ”Research, Practice and Policy – better collaboration and knowledge integration for Public Health” as part of the National Public Health Association. The other countries had a more local approach. For example, in Romania, a Work Group Public Health and Work Group Sports and Community was established at the end of 2013 as part of the Cluj Management and Planning Group, a task force created for the development of the new Cluj-Napoca Development Strategy for 2014-2020. Moreover, the Romanian partner has continued its work as part of the platform by joining national and international networks in 2016. The Romanian partner has applied and has received HEPA Europe temporary membership confirmation. The European network for the promotion of health-enhancing physical activity, until confirmation by the network at its next annual meeting (September 2016), being the first Romanian institution member of the HEPA Europe Network. In Italy, CNR has started to build the Italian National Platform in 2013, involving stakeholders, researchers and policy makers in the field of public health and cross-sector areas at a local and national level, plus a few experts from international organizations. The platform includes research managers of universities and research centers, administrators of health organizations (for instance, local hospitals and the National Health Institute) and political institutions (for instance, the Lazio Regional Council and the Ministry of Health); sector associations (some examples: Associazione italiana di epidemiologia, Associazione Italiana di Economia Sanitaria, Associazione Nazionale dei Medici delle Direzioni Ospedaliere, Cittadini Reattivi, the Italian National Olympic Committee, the European Network for Sport Science Education and Employement, the European Association of Hospital Managers); sector magazines, like Epidemiologia e Prevenzione, L'ospedale, European Journal of Sport Science.
The REPOPA website features of the platform include also rather detailed description of each country context and the status of the evidence-informed policymaking in each context. The platforms are supposed to increase sustainability e.g. in case of staff members of institutions involved change.
Dissemination in the Consortium was planned to happen systematically; for this reason the project developed working protocols and templates including internal and external communication guide, internal deliverables and activities guidelines, publication guidelines and continuously updated publication plans, and internal dissemination reporting template. These templates and guidelines can function as tools also beyond the REPOPA project. (Chereches et al 2016 submitted).
Channels for dissemination
Together with the platform launch, REPOPA has launched its twitter account, which is integrated in the platform as well. The role of the twitter account is to open discussions with other researchers, practitioners, policymakers and actors interested in the REPOPA work and outcomes. Below are some highlights of the twitter account in the last month of the project (September, 2016). Our September 2016 tweets had 1,475 tweet impressions and gathered 131 profile visits, adding eight new followers to our page. Most of our tweets focused on evidence- informed policymaking, sharing our own knowledge, for example: “Policymakers can benefit from support by researchers - #SDU #nhprc2016 http://bit.ly/2dabE7z “ or acknowledging existing knowledge and information disseminated by others, for example: “How can we use evidence to strengthen policy? Introducing the Evidence-Based Policymaking Collaborative: http:/urbn.is/2cDBOD9 #whatworks”. The account will remain active after the project ends, supported by UBB and SDU staff.
Newsletters: twice a year the REPOPA newsletter informed internal and external audiences about the project progress and outcomes. Leaflets and posters: yearly the project leaflets and posters where used as important dissemination tools at meetings, conferences, and workshops. Over 600 leaflets were disseminated.
Conference presentations: a total of 92 abstracts were presented at national, European and international conferences, out of which 35 were published in peer-reviewed international journals. Workshops: each year, REPOPA has organized at least one workshop at the European Public Health Association Conference (EPH).
Publications of peer-reviewed articles and chapters: a total of 16 peer-reviewed articles and chapters and 29 peer-reviewed abstracts were published based on the results of the REPOPA research and dissemination work; a number of publications are under revision. The REPOPA Consortium plans to continue collaborating in publishing valuable data. Popular journals: a total of 30 articles were published in national popular journals, institutional newsletters and international ones as well.
Networks: the REPOPA Consortium has joined networks like icSPORTS, European Implementation Collaborative Network (http://www.implementation.eu/) and Research to Action Network (http://www.researchtoaction.org/) HEPA Europe Network (http://www.euro.who.int/en/health-topics/disease-prevention/physical-activity/activities/hepa-europe).
Evidence briefs: Print and online version of evidence briefs were developed to disseminate research work of the REPOPA project – a total of four evidence briefs were developed for different work packages of REPOPA.
Evidence Brief no. 1: Evidence Informed Policy through Simulation Games – A practice in collaboration; based on the work package No22 simulation game work. REPOPA investigated if a policy game intervention - based on needs, context and systems analysis – could be used to improve communication and collaboration between stakeholders in a cross- and multi-sector health policy making process. In which way does a policy game enhance the level of evidence-informed HEPA policy making and lead to a more efficient and successful policy? Some results are presented in the evidence brief along with the steps we used in simulation game.
Evidence Brief no. 2: Recommendations for Locally Tailored Interventions – Knowledge integration in physical activity policymaking; based on the work package No3 Stewardship-approach work. Policymakers often have a hard time to find and apply research evidence in their real life contexts, which are loaded with different stakeholders, priorities and values. So far there has been rather little research on interventions to support policymaking in balancing these different kinds of evidence and information. REPOPA took this challenge and developed and implemented locally tailored interventions. The evidence brief goes through the steps of the tailored-stewardship intervention.
Evidence Brief no. 3: Recommendations for Using Evidence in Real World Physical Activity Policy Making; based on the work package No4 Delphi study work. Evidence-informed policies take into account both research evidence and contextual resources, priorities and values. There is a need for tools to assess how well policies are evidence-informed. Based on our Delphi study we developed and validated indicators to do this and to offer guidance for the integration of evidence in policymaking. Recommendations are detailed in the evidence brief.
Evidence Brief no. 4: Recommendations for Identifying Optimal Evaluation Processes for Funded Research Projects; based on the work package No5 internal and external evaluation research work. Research funding agencies need to demonstrate the health and policy impact of their research funds using optimally-designed evaluation processes. Some controversy and debate exist about the value of external project evaluations, and expectations and requirements for project teams for process evaluations. Our experience evaluating a multi-country research project offers one effective approach and is presented in the evidence brief.
Animations: So far three animations were developed to better describe the interventions implemented in the project REPOPA as lay summary versions of the work. The animations will be used for dissemination and teaching. For the animations, please visit www.repopa.eu
In parallel to specific work package No6 dissemination activities, a major role was played by the research work packages in disseminating and exploiting the outputs of the research work, which has a stronger impact in research knowledge and project sustainability.
Final REPOPA Symposium in Brussels
The culmination point of the REPOPA project was the final, invitational Symposium organized in Brussels the 8th of September 2016. This symposium discussed the output and experiences of the project and continued to brainstorm on the issues found most problematic in REPOPA and thus needing further work. http://www.repopa.eu/news/researchers-can-support-policymakers-their-work.
This brainstorming was done using the world café method in which groups of experts going around discuss the themes dedicated for each table. All in all we had 20+ external experts participating in the Symposium; they came from the six REPOPA countries representing ministries, sport organizations, universities and from the EU and international organizations. The themes of the world café tables included the following: facilitators and barriers as well as competences needed in evidence-informed policymaking, contextual factors and implementation challenges in developing evidence-informed policies. See photo 3 from the world café in the Annex.
E-survey in the end of the project
The e-survey was developed together with the partners in the last year of the project. It aimed to test the developed evidence-informed policymaking indicators as well as synthesize information provided from the whole project period, including the invitational final symposium September 8th 2016 in Brussels, Belgium and the 7th conference of HEPA Europe, 28-30 September 2016, Belfast, Northern Ireland, United Kingdom. The survey aimed to gather knowledge to improve the validity and reliability of the indicators, and also to test if items in the survey were clear and understandable, relevant and pertinent to their organization. The development of the e-survey was covered by the ethics exemption approval nr. 31/62 from 16.10.2012 of the Ethics Committee of Babes-Bolyai University from Cluj-Napoca, Romania – the coordinator of work package No6 – Dissemination. The application of the e-survey includes an informed consent, as well. The questions are meant to be answered from the perspective of organization/institute of the respondents and aiming for testing the usability of this instrument as regards the readiness for evidence-informed policymaking practices. The e-survey is a 21 items online survey that takes about five to ten minutes. The UBB project team has invited stakeholders to take part in this online survey.
Summary: REPOPA dissemination both spread the output and news from the project and built an umbrella platform with country-level platforms to create sustainability for evidence-informed policymaking in the REPOPA countries and beyond (Chereches et al 2016 submitted).
Project process and impact evaluation
REPOPA Consortium wanted to work transparently and to learn from its working process; for this purpose, a special work package (No5) was established and given to a Canadian partner to lead. The Canadian partner was the only non-European partner in the project. The partner had had long experience in evidence-informed policy work and real-world collaboration in projects.
Evaluation tasks were to monitor and evaluate the working process, content, scientific product and added value and synergies of project parts, dissemination activities plus the potential impact of the project as whole. In addition, the aim was to apply the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation and Maintenance, (Glasgow et al 2006) to analyze impact of project activities in the participating countries/context and in relation to development in policymaking more widely.
How was the evaluation done?
A mixed methods approach was used for both the formative process evaluation and the final summative outcome evaluation. The formative monitoring and evaluation strategy was participatory, consensus-orientated and utilization-focused. The evaluation rounds were done in annual cycles (2013-2016); action recommendations were given to the Consortium to stimulate Consortium discussion and action planning to build on project strengths and opportunities, and address implementation challenges. For the project working processes, annual interviews were conducted with work package and country teams; topics included Consortium diversity, stakeholder engagement, REPOPA knowledge products and dissemination products, and priorities for upcoming work. Also document reviews of periodic reports, final reports, internal six months reports and minutes of different Consortium meetings, were conducted annually to examine project and work package progress, management, and implementation challenges. The Consortium collaboration survey was conducted annually. The survey assessed five domains: communication, collaboration, knowledge translation, project management, and evaluation. The junior researcher competency self-assessment survey was delivered annually to all Consortium members who were trainees. They were asked to rate their level of improvement in the previous year, on 27 competencies and skills related to research methods, knowledge translation, evaluation, research writing, project management and communications, and mentorship. The social network survey examined internal scientific contacts among project team members and external contacts with stakeholders. Network maps were made using NetDraw (Borgatti, 2002) to help visualize the REPOPA networks.
The final project evaluation was conducted using an external summative approach. It drew on the four years of data collected during annual monitoring to assess project working process, synergies, dissemination, and impact. For synergies among work package components and value added to the project, annual interviews with work package and country teams were used; in addition to publications, research, innovation, and challenges to achieving synergies. For dissemination, national platforms and the web-based umbrella platform developed by the project were examined, and project dissemination approaches were reviewed. For scientific and policy relevance, evaluation was done on final work package reports, frameworks, interview schedules and publications; and gender and context were used as additional criteria.
Results on scientific relevance
The project work packages provide considerable value added, with a number of key strengths: rigorous methods (including case studies), good heterogeneity in types of stakeholders and sectors involved and rich descriptions of context that informed several processes (e.g. context mapping) developed uniquely for each work package.
Work package No1 yielded insights into cross-sectoral policy making that may inform both policy makers and researchers. Findings highlighted some critical considerations in facilitating cross-sectoral policy development approaches and yielded methodological advances in the meta-analysis of cross-sectoral policies. Both work package No2 and No3used a context-informed adaptation of their intervention approaches. They provide strong illustrations of how to address the tension of standardizing form versus function in interventions. The use of actual (rather than hypothetical cases) strengthens the applicability of their findings. Work package No4 developed a list of measureable indicators for evidence-informed policymaking, refined with input from stakeholders across six European countries by means of a Delphi process. Processes used in both conceptualizing and refining indicators were robust.
Limitations: Although not a requirement of the Description of Work, the lack of any costing data or specific documentation of human resource requirements makes it difficult to assess the feasibility of adapting and using the interventions in other settings. The gender considerations and analysis as described in the project documents were also met, but a more explicit gender analysis framework could have brought more in-depth information. It is likely that team members were working with “early adopters” and some of the findings may only be transferable to other municipalities that are willing to engage (or already have relationships) with external researchers. Work package No1 only looked at the phases of agenda setting and policy development. Work package No4 provides some promising indicators for evidence-informed policymaking; those which were least developed were in a sub-category of “complex indicators”. These warrant further development given the nature of inter-sectoral decision-making (Edwards et al 2016, submitted).
Results on project working processes
The REPOPA project accomplished the ambitious scope of work outlined in the Description of Work despite several implementation challenges. Collaboration networks that developed within and across work package teams helped Consortium members accomplish their ambitious scope of work (Figure 6, Annex). Between 2013 and 2016, there were improvements in team collaboration in the areas of communication, collaboration, knowledge translation and evaluation. Involving junior researchers in REPOPA proved mutually beneficial for both the project and these trainees. A total of 12 junior researchers were linked to the project at different points in time. Across all project years, junior researchers most often reported self-improvements in a variety of research methods and competencies; they least often identified improvements in either knowledge translation or evaluation skills. Improvements in writing skills were most notable in the final year of REPOPA. Country and work package teams developed and extended their networks with policy stakeholders as the project progressed. There were shifts noted from single to multiple sector engagement and from the involvement of municipal to national and international decision-makers. External connections became deeper and more deliberate although not necessarily more numerous. In the first year, researchers reported that the primary benefit of working with external stakeholders was understanding the policy context. In the last two years, they described their external stakeholders as primarily assisting with the dissemination of research findings and providing access to decision-makers.
Results on Synergies among Work Package Components and Added Value for the Project
During interviews, Consortium members stated their commitment to maximize the added value of a multi-country initiative and its multi-disciplinary team. The structure of work packages, planning meetings and the commitment of work package leads helped to facilitate the cross-fertilization of ideas, approaches and findings. Policy makers’ engagement as intervention participants in all phases of the project was seen by the team as an important success. While the involvement by Consortium members in multiple work packages contributed positively, it also carried a cost, since it meant that a number of team members faced multiple and significant competing priorities. When members were pulled in too many directions, manuscript writing efforts were most often deferred.
Results: Dissemination
By the end of the project, all countries had established national platforms though to varying extents. Teams were starting to use national platforms to disseminate REPOPA findings. Plans to continue the web-based umbrella platform beyond the end of the project may provide a useful resource to support national platform efforts. Because country teams involved in work packages No 2, 3 and 4 had to prioritize research activities for much of the project duration, dissemination tools developed by dissemination work package were underused.
REPOPA used a variety of dissemination strategies to reach researchers, policy stakeholders, and the general public. As a multi-country project, teams had to find the right balance between producing dissemination materials in English versus tailoring language and content to the needs of national policymakers. Stakeholder engagement is essential to intervention success and creates expectations from external participants (and related time commitments for the project team) that go beyond the effort required for more traditional dissemination efforts (e.g. conference presentations and publications).
Summary
Participatory, utilization-oriented process evaluation seemed to strengthen project implementation and science. It very likely increased utility, uptake, and ownership of evaluation findings and related recommendations. Projects that deliberately bring together multiple countries, contexts, and interventions may deliver outputs with stronger scientific and policy relevance. They also face particular challenges in tailoring interventions and developing targeted dissemination strategies for a variety of policy stakeholders. An optimal evaluation design may need to involve a combination of participatory process evaluation that supports iterative planning and decision-making, along with an outcome evaluation to assess impact and the added value of synergies. Imbedding an evaluation team into a Consortium is an effective tool for internal and external evaluation. (Edwards et al 2016, submitted)
Towards an integrated REPOPA framework and indicators for EIPM
Several suggestions have been presented as facilitators and barriers of evidence-informed policymaking but there have been very few attempts to develop a generic and integrated framework to coherently describe the areas and issues to be included in such a framework for evidence-informed policymaking Building on the published literature, findings and lessons learned in different work packages REPOPA project drafted a conceptual framework to guide the development of indicators to assess if and how evidence-informed policymaking is The work started by mapping the theories and frameworks published in the field and operationalizing them for evidence-informed policymaking process using a practical scenario of developing physical activity policy in a school setting (Bertram et al 2015).
Definitions
The framework is a conceptual map or structure linking indicators to theories, goals and planning of policymaking. It is a theme-oriented approach reflecting the key challenges in evidence-informed policymaking. (Eurostat report, http://ec.europa.eu/eurostat/web/sdi/indicator-framework accessed 211116)
The indicator is a variable selected and defined to measure progress towards an objective (Litman 2011).
Policymaking process is evidence-informed when different stakeholders, including researchers, as active participants create their salient and feasible policies by taking into account contextual needs, priorities, values, resources and skills, pay attention to equity and ethics as well as evaluation and accountability of the policy- all this in the wider cultural and social context (Satterfield et al 2009, Edwards 2012).
The process of developing REPOPA framework and indicators
Contribution of work package No1 to the REPOPA framework and indicator development
The work package No1 policy analysis and interview results confirmed the relevance of the preliminary REPOPA framework and indicators, which we drafted in the Description of Work. The framework for the intersection of public health decision making (Satterfield et al 2009; Figure 1, Annex) which describes the overlapping circles of best available research evidence, resources, including practitioner/policy maker expertize and contextual characteristics such as population, its needs, values and preferences - all happening in the environmental and organizational context, seemed to be well-fitting for the REPOPA work. In line with this, a health promotion evidence paradigm (Aro et al 2008, Skovgaard et al 2008, Edwards and Di Ruggiero 2011) that helps to assess salient contextual factors and challenges for policy implementation, was supported by the work package No 1 results.
The Knowledge-to-Action Cycle (Figure 2 Annex), (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012), described in the REPOPA Description of Work, also turned out to provide a viable model for the analysis of and later interventions on evidence-informed policymaking. Building on the research knowledge synthesis, providing locally relevant knowledge and assessing facilitators and barriers of knowledge integration and working towards tailored interventions were all enforced by the results. Further, in the course of REPOPA, Edwards (2012) developed a revised version for CIHR’s (Canadian Institutes of Health Research) Institute of Population and Public Health (IPPH) of Graham et al.’s (2006) original Knowledge-to-Action Framework. The revised version includes scalability of knowledge, inter-sectoral collaboration, implementation systems, economic analyses, and equity and ethics considerations. These elements (except implementation systems due to the scope of work package No1) were also covered in the work package No1 results; especially cross-sector collaboration (Hämäläinen et al 2015; and equity (Hämäläinen et al 2016b, in press), and were relevant for further research such as REPOPA interventions in work packages No2 and No3. In addition, as systems approach (Best, Holmes 2010) shortly described in the REPOPA Description of Work suggests, European physical activity policymaking as studied in REPOPA, seems to benefit from the non-linear, complex systems perspective, which allows unpredictable changes but also helps to better understand policy processes.
Further, Stages of Knowledge Use (Knott, Wildawsky 1980, Landry et al 2006) framework was usable in the REPOPA work; this framework describes six stages of knowledge use seen as a process, which includes reception, cognition, discussion, reference, effort and influence. Elements of these e.g. influence, was one of the central questions to study in work package No3. Weiner’ framework (2009) of organizational readiness for change functioned well especially in work package No2 policy games. Phases of policymaking (Walt et al 2008) guided centrally the work done in work package No4 when mapping the evidence-informed policymaking indicators into policy phases in different countries.
The aim of developing preliminary REPOPA indicators based on the work package No1 results using policy analysis and stakeholder interviews, was to suggest a usable method for assessing the process of developing and implementing policies rather than establishing the status of policies. Hence, the proposed initial indicators from work package No1 identified possible areas for improvement in integrating research evidence into real-life policymaking. They also provided input for the other work packages of REPOPA; the main further work was done in work package No4.
The list below gives the set of initial indicators formulated in the end of the 1st research phase of REPOPA, work package No1; in the subsequent work packages they were further re-formulated, tested, validated and contextualized.
1) Is the best available research evidence taken into account?
• If so, is it based on an evidence hierarchy (systematic review, evidence synthesis, article synopses, randomized controlled trials, cohort studies, case-control studies, etc.): yes/no; please specify
2) Is research evidence integrated with contextual needs?
• Yes/no. If yes, what kind of contextual needs e.g. feasibility, political priorities, resources, population characteristics etc. are taken into account?
3) Are relevant stakeholders involved?
• Yes/no. If yes, which stakeholders and how are stakeholders involved? And to what degree is the research community represented?
4) Have all relevant sectors been identified and involved in the policy development and implementation?
• Yes/no. If yes, which sectors?
5) Are equity issues taken into account in the policy making process?
• Yes/no. If yes, how is equity taken into account? (E.g. inclusion of those whom the policy concerns, consideration of possible special needs for vulnerable or ethnic minority groups in terms of access to physical activity facilities)
6) Is policy evaluation and/or policy impact assessment in place?
• Evaluation: outcome, process, context evaluation. Yes/no: please specify
• Impact: health impact, environmental impact, social impact. Yes/no: please specify.
7) Is a systems approach taken into account, meaning that policy development can be non-linear, with inter-dependences between levels, triggered by e.g. media events, negotiated interests and communication? Yes/no, please specify.
8) Are supporting structures (evidence platforms, working groups, and networks) and knowledge dissemination of evidence-informed policy making in place/built/encouraged? Yes/no. Please specify.
Contribution of work packages No2 and No3 to the REPOPA framework and indicator development
In the work packages No2 and No3 the programmatic research approach chosen for REPOPA was most clearly implemented. In these the systems analysis of the intervention contexts and needs assessment of the stakeholders were the basis for salient and relevant interventions – in line with Satterfield et al (2009) evidence-informed policymaking and Knowledge to Action Cycle (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012) Figure 2 Annex) frameworks. Further, especially in the work package No3 using Stewardship approach (Nuffield Council 2007) to guide the intervention development, the inclusive, equity emphasis was explicit. Using these frameworks these work packages contributed to eliciting potential indicators for evidence-informed policymaking.
Work package No2 policy game interventions in Denmark, The Netherlands and Romania aimed to find out the ways in which policy games can increase the use of research evidence and other evidence in policymaking (Goor van de et al 2015). The intervention results provided further issues to build draft indicators for evidence-informed policy making such as leadership, communication and collaboration within organizational network, including stakeholders, importance of cross-sector collaboration for HEPA policy, where to find useful knowledge, increasing use of knowledge and expertise of stakeholders.
The work package No3 case studies analyzed in Denmark, The Netherlands and Italy aimed to investigate if the Stewardship approach increases the level of evidence-informed policymaking (Aro et al 2015, Bertram et al 2016). The results supported issues emerging from work package No1 and No2 and provided new inputs for the formulation of indicators in especially two areas: Human resources – Skills and competences to promote evidence-informed policymaking and networking, highlighting the relevance of involving stakeholders, vulnerable groups and target groups as active partners in the policy design and implementation; and Communication and participation, proposing indicators related to if and how to gather knowledge from stakeholders and target groups and highlighting, in this way, the relevance of regularly informing researchers and stakeholders about the policy development and implementation process.
Contribution of work package No4 to the REPOPA framework and indicator development
Within work package No4, further analysis of literature on indicators, on their typology and development criteria was done (Bertram et al., 2015). Building on the results of the previous phases of the REPOPA project, as well as on the internal consultation round, a three-round Delphi study was performed to formulate and validate a list of indicators for evidence-informed policymaking (Valente et al, 2016 submitted). First, two rounds of international surveys were run; these produced 25 measurable indicators organized in four thematic areas of human resources, communication and participation, documentation, retrieval and production, and monitoring and evaluation. In addition eight so called ‘complex indicators’ were identified, which could not (yet) be made measurable. After that each of the six REPOPA EU countries arranged national conferences to further contextualize and validate the indicators developed in the internet rounds.
REPOPA framework: operationalization of the Satterfield framework
REPOPA empirically developed, validated and contextualized indicators to measure the central characteristics of the Satterfield et al (2009) (Figure 1, Annex) framework of the intersection of decision/policymaking. REPOPA was able to operationalize otherwise abstract characteristics or knowledge sources in the Satterfield framework.
The Satterfield circle of population needs, values and preferences was empirically confirmed in its importance in work package No3 intervention results (Aro et al 2015, Bertram et al, in preparation). It was found that e.g. in addition to having certain stakeholder groups involved in decision making, also organizational procedures are important for sustain evidence-informed policymaking. Population needs, values and preferences were further operationalized in REPOPA indicator set ‘Communication and Participation’.
The Satterfield circle of best available research evidence is covered by REPOPA indicator set ‘Documentation’. This information was initially produced in work package No1, and it was further confirmed in work packages No2, No3 and No4.
The Satterfield circle Resources, including practitioner expertize is covered by the REPOPA indicator set ‘Human resources’. This knowledge was empirically produced from work packages No2 and No3.
The Satterfield background circle (Figure 1, Annex) Environmental and organizational context was a central descriptor of larger, overall context influencing evidence-informed policymaking e.g. in work package No2 and No3. However, in the Delphi study, indicator candidates related to Environmental and Organizational context were not deemed measurable; they were called ‘complex indicators’ and included issues such as cross-sector collaboration, organizational culture and leadership in evidence-informed policymaking, accountability for policies, involving researchers early in the policy process, commitment of policymakers to enhance citizen participation, requests from policymakers for research evidence and evaluation of policy results.
Using the EU Sustainable Development classification of indicators to headline, operational, explanatory indicators; the latter can be further divided into contextual indicators, indicators under development and indicators to be developed; the current REPOPA indicators are at best explanatory; and the complex indicators could be contextual indicators or indicators under development http://ec.europa.eu/eurostat/web/sdi/indicator-framework.
REPOPA indicator set Monitoring and Evaluation is not directly operationalizing any of the Satterfield et al (2009) circles. On one hand, though deemed measurable, these validated indicators are also related to the category of ‘complex indicators’ (not measurable at this stage), among which there is ‘evaluation of policy results’. On the other hand, Monitoring and Evaluation could be better used to operationalize a phase of another central framework guiding REPOPA work- that of Knowledge to Action Cycle (Straus, Holroyd-Leduc, 2008. Graham et al 2006), where there is a phase of evaluation.
The Satterfield intersection of best available research evidence, population characteristics etc., and resources etc., which is labeled as decision making, depicts evidence-informed policymaking in REPOPA (Figure 1, Annex). This means that when all three small circles overlap and they overlap in the context of policymaking, policymaking can be said to be evidence-informed. Research evidence is taken into account, but so are contextual issues such as priorities, values, resources and so on. This Satterfield et al (2009) descriptive way of presenting the indicator framework does not yet tell about how well/to what extent the overall policymaking is evidence-informed; unless one could consider the size of the overlapping area telling this- this again depends also on the availability of each kind of knowledge and input (best available research evidence).
Added value of the REPOPA framework and indicators
The added value of the REPOPA framework and indicator work is the first step towards operationalization of the concepts and measurability of the evidence-informed policymaking indicators. The REPOPA framework and indicators cover those aspects of evidence-informed policymaking in physical activity area, which were seen important based on the previous literature and the REPOPA project work in work packages No1-No4. This does not mean that they are exhaustive to describe and measure evidence-informed policymaking even in physical activity. It is important to remember that especially work package No4 Delphi work emphasized measurability of the indicators to be included. The SMART (Specific, Measurable, Attainable, Realistic, Time-bound) criteria were used; in addition, objectivity was added as one criterion. Those eight ‘indicators’, which were too complex to be SMART (though relevant) for evidence-informed policymaking, are thus listed separately- and can later be developed towards measurable indicators. Further, with different research aims and questions plus methods, very likely different characteristics of evidence-informed policymaking could be found. In the future e.g. explorative factor analysis method could be used for empirical data sets to study underlying dimensions of evidence-informed policymaking; like this the indicators could potentially be developed into a multidimensional quantitative scale.
Further development and limitations of the current REPOPA framework and indicators
This report presents mainly how REPOPA could operationalize Satterfield et al (2009) framework and partly Knowledge to Action framework (Graham et al 2006, Straus, Holroyd-Leduc 2008, Edwards 2012) Figure 2 Annex). This initial work functions as an example of the further refinement and framework development, in which also other frameworks will be considered, especially those of organizational change (Weiner 2009), policy and political theories (Walt et al 2008, Kingdon 2010). Thus the present REPOPA framework and indicators need to be implemented and tested in different contexts.
Potential Impact:
Overall REPOPA aim and process: societal impact
The title of the REPOPA project already tells the goal of the project: ‘Research into Policy to enhance Physical Activity’. Thus the whole idea behind the project and its objectives both in terms of scientific work and in terms of collaboration and dissemination, are directly linked to real-life and societal activity such as developing physical activity policies in different settings. The project was built on trans-disciplinary approach and used approaches and methodologies that integrate scientific knowledge and non-scientific and non-formalized knowledge (=other kind of evidence). In this way, REPOPA can contribute to advancing fundamental understanding and solving complex problems while fostering multi-actor engagement in the research and innovation process.
In its task to integrate research knowledge and real-world policymaking, REPOPA has been closely working together in inclusive ways with various societal actors on different levels (researchers, citizens, policy makers, third sector organizations) during the whole project period.
The REPOPA research project has been programmatic, meaning that research objectives, design and methods were not developed in detail a priori but they were developed based on the local needs and together with the stakeholders of the policies. In work package No3 tailored interventions to enhance evidence-informed policymaking was the main objective; the interventions also showed that they helped policymakers to better align both the process and its outcomes with the values, needs and expectations of their communities.
In addition to carrying out research, via its web-based internet platform with national platforms, REPOPA has also built structures for inclusive collaboration to increase sustainability, multi-actor and public engagement in research and innovation. These platforms as well as tools developed such as policy briefs and checklists for evidence-informed policymaking indicators enable easier access to scientific results.
A special feature in REPOPA has been that it has used computer software of Readability Index (www.readability.com) to test and improve its dissemination material for lay audiences.
Further, REPOPA work packages No2, No3 and No4 all cherished public engagement by iterative and participatory multi-actor dialogues to foster mutual understanding and co-create research and innovation outcomes.
Examples of REPOPA work and products with societal impact
Phase 1; work package No1: Having the existing policies as study object and interviewing the persons involved in the preparation of the policymaking, increased the applicability of the results and made it easier to communicate the findings. Scientific papers presented in the scientific conferences informed the research community of the findings of the research and the lay-type articles informed policymakers and wider the community of the results. While there may be some instrumental use of key findings and recommendations in future policy making in countries, a conceptual use of the result findings can be thought to be achieved at least during the interviews conducted with persons involved in the preparation of the policies concerned. Contacts made when interviewing these people set the ground for future, closer collaboration with them. These contacts are especially important for the junior researchers that did not yet have collaboration with the policymaking institution
Phase 2; policy games, work package No2: To achieve more effective and sustainable public health policy, work package No2 cross-sector collaboration and evidence-use in the policy process are essential preconditions. To stimulate both cross-sector collaboration and evidence use, work package No2 developed a real-life based policy game and evaluated it in three EU countries. In each country multiple societal actors such researchers, local policy makers and other relevant actors participated in preparing, designing, adapting and finally playing the policy game. Further, the policy game focused on an actual local policy issue such as developing and implementing physical activity policy in a specific neighborhood and thus involved various local actors relevant to this policy; besides researchers and policy makers also schools, community councils, care organizations, sports organizations for instance.
Among participants the policy game event (a one day meeting with real life societal actors playing in teams) raised awareness on and provided insights into the roles and positions of all organizations involved, the dynamics of the policy development process, the benefits of collaboration and use of evidence and expertise from the network. The policy game provided participants a safe learning environment closely resembling the real-life context of policymaking. It increased mutual understanding among participants, which in turn stimulated collaboration and trust between the organizations involved and had potential to influence the take-up and governance through actions bringing about institutional change among the policy game participants.
A policy game has shown to be an innovative method to support and realize cross-sectoral integrated health policy at the local level to become implemented and sustainable. It offers the opportunity to various societal organizations to learn how to collaborate and benefit from their organizational network and its expertise and available evidence. It thereby contributes to the co-creation of a shared knowledge base for local public health.
Phase2; Stewardship interventions, work package No3
This work package had the overall aim of finding ways to collaborate between policymakers and researchers that would create win-win situations. For policy makers the win could be to develop more evidence-informed policies and for researchers to carry out more policy/practise relevant research. The activities were based on tailor-made interventions for each setting, thus taking the context into account. The interventions were also built on policy formulation phase of real policies, which was the most appropriate phase of the policy cycle to bring in research knowledge. Using real policies meant that interventions were participatory, salient for the context, built upstream and the research carried out was societally relevant. The stewardship-based locally tailored interventions increased public engagement and also created the space for ethical value-laden issues such as including vulnerable groups. Like this the interventions enhanced inclusiveness, transparency, diversity, and creativity.
The close collaboration between the two parties in tailor-made interventions is a new strategy for achieving successful integration of research and policy and practice. The work package No3 emphasized having an equal participatory approach, where researchers and policymakers took part in the process with the same amount of power. And it worked! In general, the policymakers did strengthen the use of research evidence together with other sources of knowledge such as from stakeholders and policy target groups. Examples of the target groups benefitting from the work package No3 interventions are in general vulnerable groups in Denmark and elderly and their physical activity needs and possibilities both in Denmark and The Netherlands, and children and their physical activity in Italy.
The work package developed tools that can be used for making organizational change beyond the project time. These tools can help to carry out context analysis, stakeholder analysis, and logical frameworks for enhancing participatory policymaking and organizational change in knowledge integration. The tools can also help to involve knowledge from the policy target groups through participation of these groups in the policymaking. This can enhance for example taking into account the needs of different socio-economic groups in policymaking. The interventions carried out improved and enhanced longer lasting collaboration relationships between researchers and policymakers. The policymakers have expressed that this kind of collaboration is important to them since after them they feel more comfortable to contact researchers for different information needs in the future.
Phase 3; developing indicators for evidence-informed policymaking; work package No4
The work package No4 drew together the previous REPOPA results concerning knowledge translation from research to policymaking and tested, validated and negotiated these results among stakeholders to finally formulate them into usable tools such as indicators for evidence-informed policymaking.
The methodology adopted to reach this aim was a modified Delphi study that led to the co-production of knowledge by means of collaboration between selected policymakers and researchers in the field of health and cross-sectors. Their contribution allowed integrating, validating and contextualizing a list of indicators for evidence-informed policy making, aimed at assessing if and at what extent a policy is informed by evidence.
The indicators, organized in thematic domains (human resources, documentation, communication and participation, monitoring and evaluation), include issues that foster societal impact such as multiple directions of communication exchange, inclusion of the variety of evidence from research results to stakeholders and civil society and to cross-sector policymaking processes; and involvement of vulnerable groups.
Another issue related to societal impact addressed by the work package No4 is the international and local perspectives joined together by the Delphi study, that included both an international at distance stage (with 76 panellists) and a face-to-face stage conducted at national level (involving 103 experts that were asked to perform a SWOT analysis and mapping the indicators to policy phases). Moreover, this process led to the availability of the indicators in six different languages (Danish, Dutch, English, Finnish, Italian, Romanian), which strengthens their potential impact in different settings.
Further, work package No4 developed evidence briefs and checklists to be taken into account while using the indicators, to be included in guidance resources to promote the use of the indicators at national and European level. With these tools REPOPA work will raise awareness and science literacy. Some of these tools have already been circulated and tested among stakeholders at national level, and positive feedback has been received as policy makers are taking into account the work package No4 results. For example, the Italian Regional council member in charge for Health is presenting own experience in REPOPA work package No4 as a best practice in the international business convention Meet in Italy for Life Sciences 2016, which took place in Rome from the 25th to 28th October 2016.
Dissemination; work package No6
This work package had the overall aim to disseminate project outputs and outcomes targeting stakeholders and wider public, raising interest and awareness of the REPOPA activities and results. Collaborating closely with all work packages, part of work package No 6 has built structures for collaboration to increase the sustainability of evidence-informed policymaking. The work package pursued this in the local and national environment of REPOPA partner countries and in the online environment of the project REPOPA website. In the local and national environment REPOPA partners reached out to existing platforms, e.g. Finland, or created their own platform, e.g. Denmark to further encourage the use of evidence-informed policymaking. In the online environment REPOPA created the opportunity to share up-to-date information, documents to be discussed, and debated with different stakeholders on the country specific pages and open discussions using twitter or a country discussion box, both included in the website, the most popular, sustainable and cost-effective being the use of already created social media environments, in our case, twitter.
Consequently, most of the impact comes from the communication activities created and promoted to make the research activities known to multiple audiences ranging from the science field via peer-reviewed publications and conferences to lay people via project poster and flyers, but most importantly, project evidence briefs and animation videos. The evidence briefs were developed to disseminate and inform policymakers and researchers about the REPOPA interventions and research, having both print and online versions while the animation videos were intended to describe in lay terms the policy interventions tested in the project and can be used for teaching purposes.
The final symposium of REPOPA, which was arranged in Brussels in the last project month, was a showcase for the project output and contribution. Representatives of the six European REPOPA countries and international organizations including EC, WHO, European Implementation Network, sport organizations and so on actively brought their feedback to the project and ideas for further developments at the symposium and in the world café as a part of it. The symposium participants can further function as advocates of the main message of REPOPA. A positive sign of the interest in the REPOPA symposium and message was a marked increase in twitter messages as well as in REPOPA website visits during and after the symposium. REPOPA website will also be kept active to update the publication list and main REPOPA-related activities. A spin-off in the Dutch policy game has received much interest also internationally.
Evaluation; work package No5
The purpose was to improve the performance and results of the REPOPA project through monitoring and evaluation. Project members were consulted to design the evaluation, thereby encouraging the uptake of formative findings by Consortium members. Project progress and collaboration within the Consortium were monitored and the findings were presented with an annual report with action recommendations. This continual improvement approach helped inform project decisions about how to work more effectively as a Consortium. At the end of the project, the project’s synergies and impact were evaluated and it was examined how relevant REPOPA’s evidence-informed outputs are for science and policy.
The evaluation showed that REPOPA achieved societal impact in several ways:
Junior researchers involved in the project improved several skills they can bring to future evidence-informed policymaking research. Members of the REPOPA team broadened and deepened their relationships with physical activity stakeholders within municipalities and at national and international levels. They successfully used these networks and policy platforms to share REPOPA findings and new approaches for evidence-informed policy-making with decision-makers and researchers. The project work packages used robust scientific methods in their research activities that will be of interest to and can be replicated by other researchers working on similar health issues. REPOPA tested two interventions – a Stewardship approach and a policy simulation game. Although these need further examination in other contexts and for other health issues, they appear to be promising approaches to enhance the use of evidence in the inter-sectoral policymaking process. The measureable indicators developed through the Delphi approach will be useful for those evaluating the use of evidence in policy-making organizations.
REPOPA’s examination of evidence-informed policy making for inter-sectoral policies is particularly timely because these types of policies are of concern to many countries. REPOPA’s outputs may continue to spread through the informal networks and channels for sharing evidence that were created during the project.
Conclusion social innovation and societal impact of REPOPA
REPOPA is an innovative and one of the few research projects in the health promotion area to study real-word policymaking processes in a variety of contexts and in ways that deliberately involved policymakers in research activities. To be able to support relevant and salient policymaking, a rather ambitious programmatic research approach was implemented. This meant carefully studying the contexts and needs before designing further steps such as interventions. Though this approach is challenging in terms of time pressure, overlapping tasks and especially in terms of coping with real life changes in the political environment, REPOPA reached its research objectives and was also able build sustainable structures for evidence-informed policymaking.
The societal impact pathways are in a way inbuilt in the REPOPA project, since actual policymakers and other stakeholders were involved in co-creating new knowledge on how best to collaborate and enhance integration of research knowledge and real-world policymaking.
Further, by carrying out research and tools development in six European countries representing different European regions, created good conditions to do knowledge transfer across societies. This increased knowledge can later facilitate potential European efforts in future policymaking by providing insight information of the different local contexts and their cultures in the policymaking area. The web-based umbrella platform and national platforms for evidence-informed policymaking, which REPOPA developed, can function as sustainable structures to continue collaboration between researchers and stakeholders in locally relevant ways.
In terms of scientific impact, REPOPA contributed to methods development, especially in the areas of using policy game as a vehicle for knowledge co-creation, in terms of applying locally tailored, Stewardship-based interventions, and in terms of developing and contextualizing measurable indicators for evidence-informed policymaking. Further, REPOPA contributes to the refinement of the theoretical framework for evidence-informed policymaking, especially for operationalizing the framework for indicator development.
REPOPA increased its societal impact by producing usable tools for multiple audience dissemination of research findings and good practices via its website, policy briefs and animations. Also, via its junior researcher involvement and training, REPOPA contributed to the next generation competence building in the area of evidence-informed policymaking.
European added value of REPOPA
REPOPA is the one of few international research projects with the objective to increase evidence-informed policymaking by integrating scientific research and actual real-world policymaking by including European countries from different regions with different societal structures and resources.
REPOPA brings innovation and added value in several ways: First, it is programmatic meaning that its design and methods were not decided in beforehand as requested in the prevailing, traditional research paradigm in public health. Instead, to enable context-relevant outcome, REPOPA combined the best available research evidence and recent theoretical frameworks with the policy stakeholder, target group and context analysis as the basis for tailored interventions. This also means that even if the interventions had the same overall goal to increase evidence-informed policymaking and also used the same outcome measures, the intervention contents, process and intensity varied according to the context. Fidelity in REPOPA was understood as adhering to the theory instead of the intervention process and contents. The comparison across contexts was assessed as the increase of the evidence-informed policymaking in each context relative to the baseline in that context.
Further, the REPOPA interventions happened in real (not only realistic) policymaking contexts and/or real policymakers collaborating with researchers. This means that political changes and other practice-related issues and priorities influenced the intervention process. This also meant that the know-how gained from the interventions is context-relevant and enhances later implementation.
This approach offered a win-win relationship between researchers and policy and practice groups- an issue seldom seen- not least due to differing world views and work practices. REPOPA took this challenge – and can confidently state the positive impact seen already in the course of the project in changing researcher-policymaker collaboration, networks and practice.
To learn more in detail of the international and multidisciplinary, programmatic research process of five years, process analysis and also impact analysis was carried out in the project.
Finally, REPOPA intervention findings were formulated, validated and contextualized in six European countries into evidence-informed policymaking indicators and guidance tools together with (inter)national stakeholders. So REPOPA finished its project work by providing also practical tools and resources, which can be used and refined to fit other contexts. Even if REPOPA used physical activity policymaking as its research theme, the findings and tools developed can be applied also in other areas, especially in the areas of lifestyle, social and environmental policy development.
Recommendations and lessons learnt in the course of REPOPA
The recommendations presented are for researchers in other programmatic and multi-country research teams, and for future research projects funded by the EC and other funding agencies.
Recommendations for researchers
- Balance scientific and lay dissemination priorities in the project schedule and provide resources for both in the budget.
- Provide adequate resources to develop targeted, tailored materials for policymakers. Producing effective materials for lay audiences takes time and resources, and possibly outside expertise.
- Commit additional project time (person-months, project duration) and financial resources to achieve programmatic, cross-site synergies.
- Deliberate team science strategies are needed to support and maximize synergies and cross-learning in multi-site Consortia projects.
- Consider how and what trainee (junior researcher) capacities can be enhanced through a multi-site project Consortium.
Recommendations to the EC and other research funding agencies
- Funding agencies should consider how they could provide the longer term research support required for research teams to build relational capital with decision-makers, to adapt and implement interventions and to conduct longer-term post-intervention follow up.
- Ensure that the criteria used for the peer review of scientific relevance are explicit in their inclusion of non-quantitative research designs such as case study methods.
- Build structures for potential harmonized ethics clearance practices in social science and policy research for EC funded multi-country studies.
- Encourage project teams to include costing estimates as part of intervention studies to better inform feasibility considerations.
- Encourage the explicit use of gender analysis within studies to build strength in this area.
- Consider use of the model of an embedded evaluation team for other EC-funded projects.
- Identify best practices for this evaluation approach across EC-funded projects.
Recommendations for scientific journals
- Journals could be more open to publishing implementation research reports form natural settings even if they do not always adhere to the prevailing paradigm of strict control of research context.
Future dissemination activities planned after the project period
First, the REPOPA project website with its umbrella platform for evidence-informed policymaking and updates of publications and other dissemination activities (e.g. conference and meeting presentations) will be kept active, including twitter, and updated regularly. The website will be managed together by BBU and SDU.
REPOPA data storage will be centrally and safely organized by SDU to safeguard the data and compliance to good practice and ethical rules in the use of the data gathered. Active publishing is also encouraged and facilitated by the created publication plans and guidelines by using the internal SharePoint site run by SDU.
A meeting report of the Final Invitational Symposium of the REPOPA project will be written and circulated widely. This circulation will include also the lessons learned in the world café session organized involving European country representatives, EC representatives, representatives of international organizations and those of research institutions.
Junior training is continued by ongoing PhD work; other kind of training such as MSc work is also encouraged.
Dissemination and implementation of the research findings, lessons learnt, best practices as well as tools such as indicators for evidence-informed policymaking, will be continued internationally, nationally and locally. In addition, submission of letters (see e.g. Aro 2016), viewpoints, debate articles and so on will be encouraged to keep the discussion going on in enforcing evidence-informed policymaking. Also further grant funding will be sought to carry out implementation studies on the research findings and tools developed in the project in different contexts.
Collaboration with the networks and relationships established in the project time will be continued and scaling-up of the collaborative policy development of the REPOPA interventions will be enhanced.
Special attention will be put on the efforts to use lay language (with software checks on readability) in the dissemination both orally with media and with different societal stakeholders and in the form of newsletters, lay articles, leaflets, workshops and meetings to be arranged to disseminate the REPOPA work.
Future steps and needs in research in evidence-informed policymaking
Areas of further research from the project’s research work packages may include:
- What are the roles of different policy actors and how can they improve the research integration process?
- How can we improve transparency of research use in policymaking?
- How sustainable are the findings of the interventions carried out in work packages No2 (policy game) and No3 (Stewardship-based tailored interventions) and how could the sustainability be improved?
- What are the implementation challenges in different contexts of the tested interventions (both work package No2 and No3) and of the indicators developed for evidence-informed policymaking?
- What are the competences and needs of both researchers and policymakers in evidence-informed policymaking?
- How best to improve policy/evidence briefs, understanding and readability among different stakeholder groups participating in evidence-informed policymaking?
Areas for further research from the project evaluation perspective may include:
- What is the combined impact of the policy game and the stewardship approach, particularly in settings where cross-sectoral collaboration is very weak (i.e. vertical silos of decision-making are strong)?
- Do those involved in the evidence-informed policy interventions such as those tested in the REPOPA project replicate/apply their learning and approaches to other policies and domains of work?
- What organizational structures are required to support these kinds of approaches?
- Does the one-day intervention of a policy game provide the necessary “force” to change collaborative behaviour and decision-making processes? How might it complement other approaches to engage policy makers such as those involved in deliberative dialogues?
References
Aro AR. Towards evidence-informed policymaking in physical activity. Journal of Preventive Medicine 2016;1;3:18
Aro AR, Smith J, Dekker J. Contextual evidence in clinical medicine and health promotion. European Journal of Public Health 2008;18:548-549.
Aro AR, Bertram M, Hämäläinen R-M, Goor LAM van de, Skovgaard T, Valente A, Castellani T, Chereches R, Edwards N on behalf of the REPOPA Consortium. Integrating research evidence and physical activity policy making- REPOPA project. Health Promotion International 2016;31:430-439.
Aro AR, Radl-Karimi C, Loncarevic N, Bertram M, Joshi R, Thøgersen M, Pettersen CLH, Skovgaard T, van de Goor I, Spitters H, Valente A, Castellani T, Cori L, Janse, J, Dorgelo A. and Pos S. on behalf of the REPOPA Consortium (2015). Stewardship-based intervention: WP3 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project. http://www.repopa.eu/sites/default/files/D3.2.Report_Stewardship%20based%20intervention.pdf
Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evidence and Policy 2010; 6:145-159.
Bertram M, Radl-Karimi C, Loncarevic N, Thøgersen M, Skovgaard T, Jansen J, Castellani T, Dorgelo A, Valente A, Cori L, Aro AR. Planning locally tailored interventions on evidence informed policy making- Needs assessment, design and methods. Health Systems and Policy Research 2016;3;2:15
Bertram M, Loncarevic N, Castellani T, Valente A, Gulis G, Aro AR. How could we start to develop indicators for evidence-informed policy making in public health and health promotion? Health Systems and Policy Research 2015;2;2:14
Bolger F, Stranieri A, Wright G, Yearwood J. Does the Delphi Process Lead to Increased Accuracy in Group-Based Judgmental Forecasts or Does It Simply Induce Consensus amongst Judgmental Forecasters? Technological Forecasting and Social Change 2011;78;9:161671–80.
Bolger F, Wright G. Improving the Delphi Process: Lessons from Social Psychological Research. Technological Forecasting and Social Change 2011;78;9:1500–1513.
Bouldekid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE 2011;6:6
Bowen S, Zwi AB. Pathways to "Evidence-Informed" Policy and Practice: A Framework for Action. PLoS Medicine 2005, 2;7:600-605.
Borgatti, S.P. 2002. NetDraw Software for Network Visualization. Analytic Technologies: Lexington, KY. Retrieved from: https://sites.google.com/site/netdrawsoftware/download
Borra´s S. Policy learning and organizational capacities in innovation policies. Science and Public Policy 2011;38:725–734.
Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annual Review of Public Health 2009;30:175-201.
Chereches RM, Rus D, Aro AR, Loncarevic N, Bertram M, Ballav Adhikari T, Valente A, Tudisca V, van de Goor LAM, Spitters HPEM, Hämäläinen RM, Ståh, T, Edwards N, Roelofs S, Anderson CS, Lau CJ, on behalf of the REPOPA Consortium (2016). Report on dissemination, monitoring and research into policy repository: WP6 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project (submitted)
Ciliska, D, Thomas, H, Buffet, C, 2012, An Introduction to Evidence-Informed Public Health and A Compendium of Critical Appraisal Tools for Public Health Practice (Revised). Hamilton, National Collaborating Centre for Methods and Tools.
Castellani T, Valente A, Cori L, Bianchi F. Detecting the use of evidence in a meta-policy. Evidence and Policy 2016;1;.12.
Delborne J, Schneider J, Bal R, Cozzens S, Worthington R. Policy pathways, policy networks and citizen deliberation: Disseminating the results of World Wide Views on Global Warming in the USA. Science and Public Policy 2013;40;3:378–392.
De Leeuw, E, Clavier C, Breton E. Health policy – why research it and how: health political science. Health Research Policy and Systems 2014;12:55.
Duke RD, Geurts, JLA. Policy games for strategic management. Pathways into the unknown. Amsterdam: Dutch University Press, 2004.
Eklund Karlsson L, Winge Jakobsen M, Winblad Heiberg M, Aro AR. Involvement of external stakeholders in local health policymaking process: a case study from Odense Municipality, Denmark. Evidence & Policy: A Journal of Research, Debate and Practice, available online since 17 April 2016. DOI: 10.1332/174426416X14609162710134.
Eklund Karlsson L, Haaber Pettersen C and Aro AR. Governance and Youth Participation in local health policy making - Case study Odense and Esbjerg, Denmark. (under preparation)
Edwards N. Advancing knowledge translation: Opportunities for mutual learning between Canada and United States, Presentation to National Cancer Institute, Bethesda, Maryland. 2012
Edwards N, Di Ruggiero N. Exploring which context matters in the study of health inequities and their mitigation. Scandinavian Journal of Public Health 2011;39 (Suppl):43-49.
Edwards N, ViehbeckS, Hämäläinen R-M, Rus D, Skovgaard T, Goor LAM van de, Valente A, Syed AM, Aro AR. (2013). Challenges of Ethical Clearance in International Health Policy and Social Sciences Research: Experiences and Recommendations from a Multi-Country Research Programme. Public Health Reviews (Online), 34(1).
Edwards, N., Roelofs, S., and Anderson, C. (2016). WP5: REPOPA Evaluation and Impact Report: WP5 final report of University of Ottawa, Ottawa, Canada.
Fischer ARH, Wentholt MTA, Rowe G, Frewer LJ. Expert involvement in policy development: A systematic review of current practice, Science and Public Policy 2014;41:332–343.
Fletcher J, Marchildon GP. Using the Delphi Method for Qualitative, Participatory Action Research in Health Leadership. International Journal of Qualitative Methods 2014;13:1–18.
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrook PA, Vogt TM. Evaluating the overall impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Education Research 2006;21:688-694.
Geurts, JLA, Duke, RD, Vermeulen, PAM. Policy gaming for strategy and change. Longe Range Planning. 2007, 40, 6, 535-58
Goor LAM van de, Quanjel M, Spitters H, Swinkels W, Boumans J, Eklund Karlsson L, Aro AR, Jakobsen MW, Koudenburg OA, Chereches R, Sandu P, Rus D, Roelofs S, Lau CJ, Glümer C, Jørgensen T, Jansen J, Dorgelo A, Pos S on behalf of REPOPA Consortium (2015). In2Action: Development and evaluation of a Policy Game Intervention to enhance evidence-use in HEPA policy making: WP2 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project. http://www.repopa.eu/sites/default/files/D2.2.%20Report_Game%20simulation%20intervention.pdf
Goor LAM van de, Hämäläinen, R-M, Syed AM, Juel CL, Spitters, HPEM, Eriksson Karlsson L, Rus D, Valente SA, Castellani T, Aro AR. What hinders integration of evidence in public health policy: Differences in worlds of policy makers and researchers? –Results from the REPOPA study in six EU member states (re-submitted).
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? The Journal of Continuing Education in the Health Professions 2006;26;1:13-24.
Green LW, Glasgow RE. Evaluating the relevance, generalization and applicability of research. Evaluation & The Health Professionals 2006;29;1:126-153.
Guidance Note for Researchers and Evaluators of Social Sciences and Humanities Research 2010. http://ec.europa.eu/research/participants/data/ref/fp7/89867/social-sciences-humanities_en.pdf
Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technological Forecasting and Social Change 1996;53;2:185–211.
Hawe P, Shiell A, Riley T. Complex interventions: ‘how out of control’ can a randomized controlled trial be? British Medical Journal 2004;328:1561-1563.
Hämäläinen RM, Villa T, Aro AR, Fredsgaard MW, Larsen M, Skovgaard T, van de Goor LAM, Spitters HPEM, Chereches R, Rus D, Sandu P, Bianchi F, Castellani T, Cori L, Valente A, Edwards N, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, Wichbold C, Cavill N, Dorgelo A, and Jansen J. (2013). Evidence-informed policy making to enhance physical activity in six European Countries: WP1 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project http://www3.sdu.dk/multi/repopa/sites/default/files/D1.1_Role%20of%20evidence%20in%20policy%20making_14062013.pdf
Hämäläinen, R-M, Aro AR, Goor LAM van de, Lau CJ, Jakobsen MW, Chereches RM, Syed AM & REPOPA Consortium. Exploring the use of research evidence in health-enhancing physical activity policies. Health Research Policy and Systems 2015;13:43.
Hämäläinen R-M, Aro AR. Ohjelmallinen tutkimusprosessi politiikkatoimien tutkimuksessa- tapaustutkimuksena liikuntapolitiikka ja tutkimustiedon käyttö. (Programmatic research process in policy research- physical activity policy and use of research knowledge. Hallinnon Tutkimus 2015; 34;3:249-263.
Hämäläinen R-M, Aro AR, Juel Lau C, Rus D, Cori L, Syed A, the REsearch into POlicy to enhance Physical Activity (REPOPA) Consortium. Cross-sector cooperation in health-enhancing physical activity policymaking: more potential than achievements? Health Research Policy and Systems 2016;14:33. (a)
Hämäläinen R-M, Sandu P, Syed AM, Jakobsen MW. 2016. An evaluation of equity and equality in physical activity policies in four European countries. International Journal for Equity in Health 2016 (in press). (b)
Jakobsen, MW, Lau, CJ, Skovgaard, T, Hämäläinen, R-M, Aro, AR. Use of research evidence in three cases of local policymaking on physical activity (under revision)
Kingdon, JW, 2010, Agendas, alternatives and public policies (2nd edn), Harlow: Longman Classics.
Knott J, Wildawsky A. If dissemination is a solution, what is the problem? Knowledge, Creation, Diffusion, Utilization 1980;1:537-538.
Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Administration Review 2003;63;2:192-205.
Linstone, HA, Turoff M. The Delphi Method. Techniques and applications 2002:53.
Litman T. Developing Indicators for Comprehensive and Sustainable Transport Planning Victoria Transport Policy Institute 2011. http://www.vtpi.org/sus_tran_ind.pdf
Nuffield Council: Public Health: Ethical Issues. Nuffield Council on Bioethics 2007. http://nuffieldbioethics.org/wp-content/uploads/2014/07/Public-health-ethical-issues.pdf
Okoli C, Pawlowski SD. The Delphi Method as a Research Tool: An Example, Design Considerations and Application. Information & Management 2004;42(1):15–29.
Reis RS, Salvo D, Ogilvie D, Lambert E, Goenka S, Brownson RC. Lancet Physical Activity Series 2 Executive Group. Scaling up physical activity interventions worldwide: stepping up to larger and smarter approaches to get people moving. The Lancet 2016;24;388(10051):1337-48.
Rowe G, Wright G. The Delphi Technique: Past, Present, and Future Prospects — Introduction to the Special Issue Technological Forecasting and Social Change 2011;78 (9): 1487–90.
Satterfield JM, Spring B, Brownson RC, Mullen EJ, Newhouse RP, Walker BB, Whitlock EP. Toward a transdisciplinary model of evidence-based practice. Milbank Quarterly 2009;87:368-390.
Simonsen J, Bærenholdt JO, Buscher M, Scheuer JD. (Eds). Design Research: Synergies from Interdisciplinary Perspective. 2010. Routledge, London.
Skovgaard T, Nielsen MBD, Aro AR. Evidence in Health Promotion and Disease Prevention, Sundhedsstyrelsen (Danish National Board of Health), Dec 2007 http://www.sst.dk/Publ/Publ2007/CFF/Evidens_forebyggelse/Evid_Health_Prom_jan2008.pdf
Spitters HPEM, Lau, CJ, Sandu, P, Quanjel, M, Rus, D, Glumer, C. Oers, JAM van, Goor, LAM van de. Stakeholder network analyses in local public health policy making in three countries (under revision)
Straus SE, Holroyd-Leduc J. Knowledge-to-action cycle. Evidence-Based Medicine 2008;13(4):98-100.
Syed AM, Hjarnoe L, Aro AR: The Delphi Technique In Developing International Health Policies: Experience From The SARS Control Project. The Internet Journal of Health 2009;8;2.
Valente A, Castellani T, Larsen M, Aro AR. Models and visions of science–policy interaction: Remarks from a Delphi study in Italy, Science and Public Policy 2015;42:228–241.
Valente A, Tudisca V, Castellani T, Cori L, Bianchi F, Aro AR, Syed A, Radl-Karimi C, Bertram M, Skovgaard T, Loncarevic N, van de Goor I, Spitters H, Jansen J, Swinkels W, Ståhl T, Chereches RM, Rus D, Bozdog E, Sandu P, Edwards N, Roelofs S, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, on behalf of the REPOPA Consortium (2016). Delphi-based implementation and guidance development: WP4 final report of the REsearch into POlicy to enhance Physical Activity (REPOPA) project (submitted)
Walt G, Shiffman J, Schneide, Murray SF, Brugha, Gilson L. “Doing” health policy analysis: Methodological and conceptual reflections and challenges. Health Policy and Planning 2008;23;5: 308–317.
Wehrens R, Bekker M, Bal R. Coordination of research, policy and practice: A case study of collaboration in the field of public health. Science and Public Policy 2011;38:755–66.
Weiner BJ. A theory of organizational readiness for change. Impelmentation Science 2009;4:67.
World Medical Association. Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013 Nov 27;310(20):2191-4.
List of Websites:
REPOPA information and partners
REPOPA website: www.repopa.eu
REPOPA email: repopa@health.sdu.dk; REPOPA Twitter: ProjectREPOPA
REPOPA Coordinator: Prof. Arja R Aro, email: araro@health.sdu.dk; Phone: +45 6011 1874
REPOPA coordinating institute: University of Southern Denmark; www.sdu.dk;
REPOPA Consortium partners*
- National Institute for Health and Welfare (THL), Finland; leader of WP1;
- Tilburg University/Tranzo, the Netherlands; leader of WP2;
- University of Southern Denmark (SDU), Denmark; leader of WP3 and WP7;
- National Research Council of Italy (CNR), Italy; leader of WP4;
- University of Ottawa, Canada (uOttawa); leader of WP5;
- Universitatea Babes-Bolyai (UBB), Romania; leader of WP6;
- Research Centre for Prevention and Health (RCPH), Denmark; WP 1, 2, 3, 4 and 6 team member
- Dutch Institute for Healthcare Improvement ( CBO), The Netherlands; WP 1, 2, 3, 4 and 6 team member. REPOPA partner until October 12 2015.
- Herefordshire Primary Care Trust (HPCT), United Kingdom; WP 1, 4 and 6 team member. REPOPA Partner until March 31st, 2013.
