Final Report Summary - JPIS TO CO-WORK (JPI’s: a process of mutual learning: TOwards a COmmon adoption of frameWORK) Executive Summary:The main objective of JPIs To Co-Work was to provide Joint Programming Initiatives (JPI), as transnational research programmes, with elements to better implement specific functions known as “Framework Conditions” (FC). FC as defined in the document “Voluntary Guidelines on Framework Conditions for Joint Programming in Research 2010” include peer review, forward looking activities, evaluation of joint programmes, funding, dissemination and use of findings, and intellectual property rights. In a broader sense, we intended to promote the best contribution of JPIs to the priorities of the European Research Area within the strategy Europe 2020.JPIs To Co-Work has organized two types of activities: • One has been to review existing documents and experiences. • The second has been a process of exchange and mutual learning between all running JPIs. The project has provided a forum for discussion, exchange of experiences and best practices regarding FC that resulted in a process of mutual learning amongst JPIs. We tried to make our own contribution to a positive development of FC in all JPIs, and to support the research community (funding programme owners, research policy makers, researchers, stakeholders) with a common language and operational instruments regarding Joint Programming Initiatives. We included governance, ex-ante evaluation and innovation as additional topics in our work, as candidates to be added to the list of framework conditions for a complete joint programme development. For each Framework Condition we conducted desk work reviewing all available documents and experiences. A briefing paper was produced based on this review and circulated to attendees of workshops. During workshops we presented the review, and real examples and experiences. All workshops included a substantial part of group discussions. After the workshops a final report was written. The documents, workshop presentations and workshop reports can all be found at the webpage: https://web.archive.org/web/20170914155837/http://jpis2cowork.eu/.We organised with the JPIs the exercise to joining efforts to address the evaluation. We meet in Brussels twice to develop a common strategy for evaluation, based on the document “Exploring collaboration for evaluating joint programming initiatives”.The opportunity to exchange ideas and experiences was very much appreciated by members of JPIs, underlying the idea that JPIs face many similar situations related to FC such as governance or funding. At the same time, the pace of implementation and relative importance of FCs can be rather different depending on the societal and scientific challenges they are addressing. JPIs to Co-Work has also shown that joint programming entails not only technical issues but also policy decisions, that make it necessary to connect national policy making with its transnational counterpart.Project Context and Objectives:The idea of a European Research Area (ERA) stressed the importance of overcoming the fragmentation of research activities, programmes and policies across Europe. It recognized the need of good coordination of research programmes and priorities, including a significant volume of jointly programmed public research investment at European level involving common priorities, coordinated implementation and joint evaluation. In July 2008, the European Commission launched its proposal with the Communication ‘Towards joint programming in research: Working together to tackle common challenges more effectively’ This proposal was welcomed by the Council, which agreed to launch the process, focused on those challenges which no Member State is capable of solving on its own .The Commission and Member States explored the potential of Joint Programming in research, and launched the first activities in this field by 2010. As Joint Programming Initiatives and other ERA initiatives (art. 185, Joint Technology Initiatives, ERA-nets) involve different institutions from different countries, with different procedures, the need of providing guidelines on the so called Framework Conditions (FC) to facilitate a harmonised planning and implementation of joint research programmes was stated at the beginning of the JPI process. Therefore the High Level Group for Joint programming (GPC), consisting of high-level representatives of Member States and Associated Countries developed the ‘Voluntary guidelines for Framework Conditions’ , published in November 2010.Ten JPIs are currently running at different stages of development. Therefore, there is a need and the opportunity to learn from each other exchanging knowledge and concerns. JPIs To Co Work could contribute to provide advice and to choose the best approaches to FCs helping to build a common language to further develop JPIs. It must be noted that the participants in this project and the Advisory Board members are involved in nine running JPIs.List of Beneficiaries: 1.- INSTITUTO DE SALUD CARLOS III ISCIII. Spain 2. - JOANNEUM RESEARCH FORSCHUNGSGESELLSCHAFT MBH POLICIES. Austria 3. - VETENSKAPSRADET - SWEDISH RESEARCH COUNCIL SRC. Sweden 5. - AIT AUSTRIAN INSTITUTE OF TECHNOLOGY GMBH. AIT Austria 7. - INSTYTUT PODSTAWOWYCH PROBLEMOW TECHNIKI POLSKIEJ AKADEMII NAUK IPPT. Poland 8. - DEUTSCHES ZENTRUM FUER LUFT - UND RAUMFAHRT EV-DLR Germany1.1 Objective:The objective of this project was to provide a forum for discussion, exchange of experiences and best practices, and implement a process of mutual learning amongst on-going and future Joint Programming Initiatives (JPIs) around the so-called Framework Conditions and the best contribution of JPIs to the European Research Area within the strategy Europe 2020.Framework Conditions were defined by the European Research Area Committee – Groupe de Programmation Conjointe - as essential functions to be addressed by JPI. They are:1 Peer review procedures2 Foresight activities3 Evaluation of Joint Programming4 Funding of cross-border research by national or regional authorities5 Optimum dissemination and use of research findings6 Protection, management and sharing of intellectual property rightAdditionally we included governance, ex-ante evaluation, and innovation as topics to be potentially added to the list of FC. The tasks of the project were basically • Analyse existing literature and experiences relative to FC• Prepare and hold the workshops on FC, providing there enough space for discussion and exchange of ideas, and trying to understand them from a strategic policy making perspective• Disseminate our findings amongst JPIs and GPC members1.2 Work progress and achievements during the project:During the Project the main objectives of the project has been achieved. Through the four workshops we have made, we contributed to provide to the research community (funding programmes, researchers, stakeholders, research policy makers) a common language and operational instruments regarding Joint Programming initiatives.WP2The specific objective in this WP was to contribute to the improvement of Foresight and Ex-ante evaluation activities of JPIs, and more specifically:• To provide materials, documents and case studies to further refine and improve Foresight activities of JPIs.• To provide materials, documents and case studies to select criteria and procedures for Ex-ante evaluation of JPIs.• To provide a forum for discussion, exchange of experiences and best practices, and mutual learning between JPIs regarding Foresight activities and Ex-ante evaluation.We have reviewed the most important principles to organize and implement a foresight exercise, and we introduced the topics of impact assessment and ex-ante evaluation as related to and enriching foresight. These last two methodologies provide the link between foresight in JPIs and foresight in European Commission policies.WP3We introduced governance as a candidate topic to be included amongst framework conditions, and related it to evaluation. The objective was to contribute to the best Governance of JPIs and sound Evaluation principles and procedures, and more specifically:• To analyse and provide best practices on JPIs Governance.• To provide materials, documents and case studies to analyse evaluation including monitoring and ex-post evaluation, learning from experiences of JPIs so far and other ERA initiatives.• To provide a forum for discussion, exchange of experiences and best practices, and mutual learning between JPIs in relation to governance and evaluation.The introduction of governance along with the rest of framework conditions has been welcomed by the participants of the workshop and the final conference. The GPC announced in the final conference the intention to include governance in the discussions on joint programming. The proposal for a common evaluation framework for all JPIs has had also support from all JPIs. It was used to develop the evaluation plans in JPI Antimicrobial Resistance, and has been also used for FACCE, Healthy Diet for a Healthy Life, and More Years Better Lives. It was presented to the GPC which showed interest in learning about it. WP4This WP tried to contribute to the improvement of Review, Funding and Management activities of JPIs linked not only to calls but to a whole range of different joint actions, and more specifically:• To provide materials, documents and case studies to further refine and improve review activities of JPIs and funding of joint actions.• To provide materials, documents and case studies to analyse funding mechanisms of cross border research, by national and regional programmes and governing structures and procedures of JPIs.• To provide a forum for discussion, exchange of experiences and best practices, and mutual learning between JPIs regarding Peer Review, Funding of cross border research and Governance issues if the calls.We defined a list of different potential joint actions of JPIs in addition to the more traditional competitive calls for transnational collaborative projects according to the model ERAnets. We also underlined the fact JPIs address societal challenges and therefore the traditional peer review model has to be widened to allow for stakeholders involvement. WP5The objective of WP 5 was to contribute to the improvement of Dissemination, Innovation, Open Access policy and Protection of Intellectual Property Rights (IPR), and more specifically:• To provide materials, documents and case studies to further refine and improve Dissemination, Innovation and take up of research results and products.• To provide a forum for discussion, exchange of experiences and best practices, and mutual learning between JPIs regarding Dissemination and Innovation initiatives.• To provide materials and a forum for discussion to improve the use of intellectual property in JPIs.This work package showed the need to address dissemination and protection of IPR as depending on each other and both related to two more general goals, one being further research capacity building (dissemination and open access) and the other being innovation (IPR). We tried to analyse them from a strategic point of view, as essential tasks, and not only as lateral products.Project Results:The main S&T results of the project have been reflected in a Technical Annex to the Voluntary Guidelines. The technical annex summarizes the information and practices collected for the documents of each work package, and the discussions of the workshops. The Annex does not develop the Voluntary Guidelines in a strict sense. In fact, the Voluntary Guidelines go deeper in some topics than the Annex does. The Annex takes a more policy oriented perspective, and reflects the issues that draw more attention from the point of view of JPIs members. The Annex tries to point out those aspects of framework conditions that are more strategic or difficult to implement because of contextual factors. This Annex has been reviewed by the authors and by Kirsten Hollaender from DLR and Wolfgang Pol from Joanneum. Technical Annex to the Framework Conditions.Introduction.Carlos Segovia. This document is one of the outputs of the project JPIs To Co-Work funded by FP7. During thirty months the project has been working on the Framework Conditions (FC) defined in the document “Voluntary Guidelines on Framework Conditions for Joint Programming in Research” .The main objective of JPIs To Co-Work has been to provide JPIs with elements to better decide on the options they have to implement the Framework Conditions. JPIs To Co-Work has organized a process of mutual learning between all running JPIs. The project has provided a forum for discussion, exchange of experiences and best practices regarding FC that resulted in a process of mutual learning amongst JPIs. We tried to make our own contribution to a positive development of FC in all JPIs and ERA initiatives, and to support the research community (funding programmes, research policy makers, researchers, stakeholders) with a common language and operational instruments regarding Joint Programming initiatives. In a broader sense, we intended to promote the best contribution of JPIs to the ERA within the strategy Europe 2020.The methodology we have used was straightforward. For each Framework Condition we conducted desk work reviewing all available documents and experiences. A brief paper was produced based on this review and circulated to attendees of workshops. During workshops we presented the review, and real examples and experiences. All workshops included a substantial part of group discussions. After the workshops a final report was written. The documents, workshop presentations and workshop report can all be found at the webpage: https://web.archive.org/web/20170914155837/http://jpis2cowork.eu/.We decided to include some topics in addition of the FC discussed in the “Voluntary Guidelines”. These topics were impact assessment and ex-ante evaluation, governance and innovation. They are included in this Technical Annex. The technical Annex.The objective of this technical annex is to guide the reader to select topics to reflect on when implementing a JPI, topics that we have concluded should not be overlooked. In some cases we even risk to propose specific perspectives to look at the topics at hand, perspectives we have learned thanks to the privileged position we have had collecting opinions of so many and so committed persons that are actively participating in JPIs. The Annex follows the structure followed by the project to review the FC. It begins addressing foresight, impact assessment and ex-ante evaluation. It continues with governance and (ex-post) evaluation of JPIs. Then it develops ideas on funding, including funding of JPIs administrative structures and funding of joint activities. It continues with the review of proposals and call management. The Annex addresses dissemination, protection of IPR and innovation as the final chapter.Integrating Framework Conditions.Framework Conditions – plus governance, impact assessment and ex-ante evaluation and innovation - are all functions that have to be addressed when implementing a Joint Programming Initiative (JPI). They relate to different managerial, administrative or even regulatory actions. The JPI is the programme that results from the combination of all these actions or ways in which a JPI addresses the FC. However, the FC are by no means a simple “recipe” to achieve a successful JPI. The FC play a subordinate role to Member States insights, approaches and expertise and can serve as additional instruments to facilitate JPI processes.To a certain extent, FC are dependent on each other. We can look at them as in a logic chain beginning in foresight, then governance, ex-ante impact assessment, call management, peer review, funding, dissemination and protection of intellectual property rights and finally evaluation. These steps cannot be taken in a strictly linear perspective, though. For instance, dissemination has to be included in the plans from the beginning, and of course when launching specific calls. Even more strategic is to design monitoring and evaluation just at the time we define the objectives of the JPI. Otherwise we can be tempted to propose goals that are too ambitious, or whose achievement cannot be verified, or that are too unspecific. Or if you postpone the design of the evaluation you may end up evaluating exactly what you did well. In the same vein, foresight not always precedes the governance structure. The selection of actors involved in boards determines the result of the foresight exercise, the scientific agenda and the funding. Foresight processes involving stakeholders rarely precedes the development of Strategic Research Agendas. The type of actors involved in stakeholder fora and the quality of the evidence base determines the result of the foresight exercise, and hence on the formulation of the strategic research agenda and of research topics. The transparency of the process can have a strong impact on the legitimacy of the outcome for policy makers, the researchers and other stakeholders across involved countries.A second way of looking at FC is considering them elements of a system. This means that while they are all working for the same goal, they are all linked and when you touch one of them you will move also something of the others.Although following the logic chain is useful to organize the work in consecutive steps, to analyse one FC at a time, the systems perspective counsels not to forget the influence of this FC in all the others. Framework Conditions as entry doors.The experience in ERA nets, JPIs and other European multinational collaborative actions has shown that a great degree of variability is possible. Variable geometry is the name we use for that.Member States (MS) can be only observers in a management board. They may participate in a call but not in the following. They can make contributions in different forms and quantities. They may support joint projects or align institutional agendas. They may open their market for researchers in variable degrees. Just as one MS can share 10% of its funding resources in a joint action of a JPI, it can also share only 15% of its information system. This is the current situation in 2014. A MS can join a JPI for the foresight exercise and use it to for its national programmes, and not participate in any other JPI activity for a time. Another MS can participate just in a couple of calls. The third way of looking at FC is consequently as dimensions in which the growth of JPIs can be observed. MS can approach a JPI preferably through some of these dimensions and gradually engage themselves in others. Recommendations for Foresight and Ex-ante Assessment.Klaus Kubeczko.Forward Looking Activities and Foresight Processes.Many member states have their own instruments and expertise on to conduct foresight and to develop Strategic Research Agendas which serves the needs of the national level of RTI policy making. Hence, processes and methodologies might be quite different hindering the trans-national joint programming activities and making them inefficient. Therefore this technical annex intends to provide a framework that can be applied in the context of trans-national programming fostering coordination of the European Research Area.The following question will be addressed: What are the outcomes of Foresight Exercises? How to implement Foresight in JPIs? Who can help, if JPIs’ ask for support for their activities?What are the outcomes of Foresight Exercises?A Foresight exercise produces formal and informal outcomes. While formal outcomes are incorporated into products and deliverables, informal outcomes derive from the foresight process itself. Foresight often has more than one outcome.• Foresight activities can align national strategies and the formation of loose networks of yet unconnected areas and disciplines. This might lead to better solutions for tackling the societal challenges addressed by JPIs.• Megatrends studies or roadmaps can feed into strategic research agendas with formal and tangible outcomes. • Foresight foster networking in the coordination of the trans-national initiative, as they can support and legitimise JPI’s activities and agendas and hence also help to institutionalise this new network of ministries and agencies involved. • Another intangible outcome of networking can be the linking of previously unconnected stakeholders, both at the level of trans-national coordination as well as on the level of experts.The following table gives an overview of various outcomes which can be expected:Outcomes of Foresight ExercisesActivities Tangible outcomes Intangible outcomesFormalisation Report, book Dissemination Workshops, newsletters, press articles, web sites Results and evaluation circulating within networksNetworking Institutionalisation of networks Development of new networks or new links within existing onesStrategic process Formal incorporation of results within strategic processes Informal incorporation of results within strategic processesSource: European Foresight PlatformFor further reading please see: http://www.foresight-platform.eu/community/foresightguide/practicing-foresight-taking-stock-and-advancing-knowledge/why-do-foresight/outcomes/http://innovation-policy.org.uk/compendium/section/Default.aspx?topicid=31§ionid=155How to implement Foresight in JPIs?For a successful Foresight Process to be implemented, the pre-foresight preparatory phase very much decides about success or failure, both with respect to a) formal outcomes, which in most cases is to contribute to the formulation of the Strategic Research Agenda, as well as for b) building genuine stakeholder commitment to action.Preparing the foresight.The following Pre-Foresight Checklists outlines the key steps that have to be taken into account when setting up a Foresight Process.• Assess feasibility:Before starting, it is important to ask if foresight is the right approach and to analyse the context and positioning. In this phase the organisers evaluate whether a foresight exercise is appropriate given the context and whether it will be able to yield valuable impacts. • Scope the exercise: Once the formal decision to start a foresight has been taken, this step is about designing the process, where the main structural decisions are discussed and taken. Scoping decisions have to be taken with respect to: a. Focus - define the topicb. Scope - define the limits of the topicc. Objectives - define what for are you performing foresight (thinking the future, debating the future, shaping the future)d. Outcomes - define what do you expect from it e. Users - describe who will use the results and how this relates to societal challengesf. Time horizon - define how far do you want to extend your exercise• Involve experts and societal stakeholders: A further step is deciding on the level of participation of experts and stakeholders. Relevant stakeholder groups have to be selected and an appropriate set of stakeholders identified.Involving experts and stakeholders needs to take into account: a. the kind of experts (researchers, practitioners) to be included, b. how many should be included, c. from which disciplines and proficiencies should they come, and what level of specialisation, and the capacity to work in interdisciplinary context (broad overview, innovative, strong specialisation). Doing foresight.• Choose methodologies and techniques: The next step is defining the methodological framework (e.g. scenario development, technology assessments, sustainability assessments, forecasting...) and choosing the actual techniques depending on the aim and requirement of stakeholder participation. Devising the methodology is effectively an element of the broader scoping phase. As methodological choices are crucial and complex they deserve deeper focus and a dedicated step. For choosing the methods it is important to understand the differences between qualitative vs. quantitative methods, predictive vs. non-predictive methods, and exploratory (exploring the future under a number of assumptions) vs. normative methods (how to achieve a specific future). A broad range of well-established foresight techniques and communication tools can be used in foresight processes. As there hardly exist tailor made tools and processes for the purposes of JPIs, tailoring of foresights is of paramount importance and can draw on tested tools and standardised techniques. A wide variety of techniques has been developed over several decades, such as Back casting, Brainstorming, Causal layered analysis, Delphi, Dialogue, Driver analysis, Folksonomies, Gaming, Integrated sustainability assessment, multi criteria assessments, Issues trees, Modelling and simulation, Narrative, Plausibility matrix, Reverse engineering, Review of abstracts, Roadmaps, Scenarios, Seven questions, State of science reviews, STEEP, Systems maps, The Fifth scenario, Trend analysis, Visioning, Windtunneling ...• Filter the appropriate techniques considering the following criteria:a. relationship of techniques to the phases of the process (exploratory, normative).b. relationship of techniques to the stages of the process c. relative amount of expertise that each technique calls ford. relative time horizon that you are consideringe. degree of stakeholder and expert involvementFor further reading please see: • http://hsctoolkit.bis.gov.uk/The-tools.html • http://www.foresight-platform.eu/community/foresightguide/practicing-foresight-taking-stock-and-advancing-knowledge/how/methodology/main-methods/• http://www.foresight-platform.eu/community/foresightguide/practicing-foresight-taking-stock-and-advancing-knowledge/why-do-foresight/limitations/• http://www.foresight-platform.eu/community/foresightguide/practicing-foresight-taking-stock-and-advancing-knowledge/how/• http://www.foresight-platform.eu/community/foresightguide/practicing-foresight-taking-stock-and-advancing-knowledge/how/methodology/main-methods/Who can help?JPIs are advised to take advantage of expertise and resources for conducting forward looking activities and foresight processes. Information is available from several websites and expert organisations. As the same solutions for all are not possible, training and support in the pre-foresight phase for those responsible for foresight and the development of a Strategic Research Agenda could be ways of transferring knowledge (e.g. member organisations of the European Foresight Platform provide training courses).There is a well-established broad range of methods, process designs and databases available in some countries and at EU level such as: • European Foresight Platform – EFP The European Foresight Platform (EFP) is a network building program supported by the European Commission’s (FP 7). It aims at building a global network of networks bringing together different communities and individual professionals to share their knowledge about foresight, forecasting and other future studies methods. On the EFP website (http://www.foresight-platform.eu/) you find o information about current and past foresight projects (http://www.mappingforesight.eu/ o Conferences, workshops and future studies information for policy makers, practitioners and beginners (http://www.foresight-platform.eu/community/foresightguide/foresight-training/) cf. the successor of the well-known ForLearn foresight guide. o Trainings are available both online and through seminars from research organisations; universities and private firms (see http://www.foresight-platform.eu/community/education-sites/).• The FOR-LEARN Online Foresight GuideIt not only describes individual methods but also gives advice on how to set-up a foresight exercise and process. (http://forlearn.jrc.ec.europa.eu/index.htm)• UK Foresight (http://www.bis.gov.uk/foresight)• Foresight's Horizon Scanning Centre Tools for strategic thinking developed by the Foresight's Horizon Scanning Centre: http://www.bis.gov.uk/foresight/our-work/horizon-scanning-centre/good-practice/toolkit• Horizon Scanning Toolkit (http://www.bis.gov.uk/foresight/) • Sigma Scan (http://www.sigmascan.org/Live/) The Sigma Scan is a searchable set of 256 brief papers exploring potential future issues and trends over the next 50 years which may have an impact on UK public policy. The papers cover a wide range of subjects, from climate science to social science, space exploration, economics and human rightsEx-Ante Assessment of a Joint Programming InitiativeEx-ante assessment can have two different functions for Joint Programming Initiatives. The first function is to inform policy makers about the expected societal value of an intended policy initiative by the JPI in order to provide ex-ante legitimacy to the decision making process. Most often the term used is Impact Assessment. The second function is to allow for policy learning and to set the grounds for a monitoring and evaluation framework for the JPI. This is mostly referred to as Ex-ante Evaluation. As Impact Assessments have already been provided in the early phase of decision about which JPIs to select, Ex-ante Evaluation can be linked to it. As societal challenges are involved, it is increasingly important to involve stakeholders in ex-ante assessments. Foresight can be particularly supportive to integrate stakeholders into the process of defining an intervention-logic of the research or innovation program, to provide an evidence base for a joint understanding of the program goals, and to clarify basic assumptions underlying the selection of criteria and indicators. The following table summarises the main characteristics of both impact assessment and ex-ante evaluation:Comparing impact assessment and ex-ante evaluation Impact assessment Ex-ante evaluationSupporting ? Strategic decision making? Policy or programme selection ? Policy learningPurpose ? Legitimation? Shared understanding of agents involved ? Set the basis for monitoring and ex-post evaluationAlready defined ? A societal challenge addressed by the JPI ? A programme or policy of the JPIProcess ? Analyse societal value of programmes ? Analyse causal pathsTo be defined ? Programme or policy options ? Describe the assumptions underlying the intervention logic of the proposed interventions in the programme: causal paths linking interventions and activities with expected outcomes; basic assumptions in terms of expectation about developments of relevant framework conditions; criteria and indicators for evaluationOutputs ? Expected societal impact of different policies to address a societal challenge ? Monitoring and evaluation framework for interim and ex-post evaluationRecommendations for Governance.Jacek Gierlinski, Anna Pytko, Ewa Szkiladz, Carlos Segovia.Governance is the function of selecting and involving the actors contributing to Joint Programming Initiatives (JPIs), and setting up the rules and procedures and using them to make this involvement constructive for the purposes of JPIs. Because JPIs are intended to develop research capacity necessary to address the Grand Societal Challenges through coordination of national research programmes, governance is primarily and most importantly about rules and procedures that facilitate this coordination.Best practices of governance of JPIs should address the following issues:• The mandate of JPI governing bodies• The balance between objectives, capabilities and responsibilities in JPI’s governing structure.• Lessons emerging from the best practices of conventional project or programme management.• Relations between different JPI’s governance bodies, such as Management Board, Steering Committee, Scientific Advisory Board, Stakeholders Advisory Board.• Fair involvement opportunities within JPIs for all Member States so as to use all available capacities.How to organize governance in JPIs.• Keep the strategic role of Governance always in mind.Governance is on the base of policy making, putting the right incentives for research performance and promoting innovation. Governance structure and procedures guide the approach to framework conditions. The members, structure and procedures of governing bodies determines:a. The scope and quality of foresight.b. The quality of review processes (peer review and suitable review methods for societal relevance) by providing access to greater pools of reviewers and best practices across Europe, plus the development of new approaches to address societal relevance.c. May facilitate avoiding unnecessary duplications and covering gaps as members share information at European level.d. Promotion of synergies, specialization according to the information at hand.e. Widest dissemination of research outputs.• Define some general values for the JPI.Values such as always working for win-win solutions, co-ownership, and fair play principles may be of help guiding the JPI. Build mutual trust.• Select a structure.Most international programmes in European context have a Management Board (MB) and a Scientific Advisory Board (SAB). Having to address a challenge that is defined in societal terms, all JPIs have also a Stakeholders Advisory Board (SHAB). Additionally, consider advantages and disadvantages of forming a Steering Committee that follows activities closer and more frequently than the MB. Define the most suitable profile for country representatives for the MB (programme owners), as well as best profiles for SAB and SHAB members. Finally design a secretariat to provide administrative and managerial support.• Try to limit bias.It is important that all those concerned by the societal challenge are present in the JPI and ideally contributing with their own resources and capabilities. The type of countries present in the MB of a JPI may frame the scientific agendas and the scientific questions to be resolved. The same is valid for members of the SAB and SHAB.• Form multi- and trans-disciplinary advisory boards.Given the multidimensional characteristics of societal challenges, build multi- and trans-disciplinary SAB and SHAB.• Use variable geometry and promote smart specialization.Allow for different degrees and modalities of involvement of countries in a JPI. Building on mutual trust, promote smart specialization. If countries feel confident that they can benefit from the research performed by others, while they can contribute with their own specialized research in return, they may be more determined towards specialization within the ERA. And higher specialization and concentration of critical mass will increase scientific productivity and returns to investments.• Share information.In order for JPIs to take informed and effective decisions, such as avoiding unnecessary duplications, covering gaps, or facilitating smart specialization - and including monitoring and evaluation of the JPI itself - JPIs need to manage European wide information. The JPI should consider:o Periodically mapping research done acrooss Europe or at least in participating countries. o Regularly sharing the databases on research funded by national agencies, which may additionally requireo Adapting national data bases to a common language and common research classification system. • Organize the coordination with other initiatives where relevant and feasible.This includes for instance ERANETs in the same or related field, other JPIs, Joint Technology Initiatives (JTI), Article 185 programmes, H2020, and European Research Infrastructure Consortia (ERIC). • Analyse opportunities for collaboration.Different JPIs may benefit from collaboration or centralization of certain common procedures and activities. Common structures could be in charge of organizing for instance review procedures for all JPIs, or common secretariat services could be provided. • Encourage national coherence towards JPIs.Several countries are participating in many JPIs, and it would make sense for them to have a national general policy for all JPIs, whilst considering the different challenges the JPI respond to and their different points of departure. This would help researchers understanding these policies and provide them with a degree of predictability of future activities.• Decide on going international.Analyse advantages and disadvantages of involving countries outside EuropeExpected output of good governance.• Mutual trust.Sound governing has to build mutual trust, which in turn is necessary to open national programmes and allow them to be mutually interdependent.• Best quality, smooth and timely operations.Good governance will facilitate the best available expertise to address framework conditions and other relevant decisions. Good governance should provide the necessary forum for all agents involved to share information and reach common understanding of issues at stake. It should facilitate decisions for joint activities and smooth implementation. Recommendations for Evaluation.Jacek Gierlinski, Anna Pytko, Ewa Szkiladz, Carlos Segovia.Evaluation can take different forms. It may assess whether a JPI has met its goals and also unintended consequences, and what are the reasons that explain these results (summative approach). On the other hand it can take a formative approach. At this stage for most JPIs a summative evaluation approach would be premature. Criteria and indicators should essentially help in a formative approach to improve current practice and pave the way for a collective learning process. JPI share similarities in formats and instruments for governance, strategic planning and implementation. But they respond to fundamentally different challenges related to the specificities of the scientific and political environment they operate in. Hence, even though the set-up of the political process of the JPIs seem to bear many resemblances prima facie, they do actually differ to a large degree in terms of what exactly they aim to achieve. Self-assessment would need to be sensitive to these different contexts and identify JPIs specific objectives and points of departure. It would require a careful approach to find the right balance between comparability of outcomes and doing justice to the individual features of each JPI.There are three levels of analysis for the evaluation of JPIs:a. Research projects and activities;b. The complete JPI as a programme implemented to address a target challenge;c. The Joint Programming concept, as an effective way for cross-border collaboration.This technical annex covers the first two levels, being the first included as an element of the second. This means that the following is referred to the self-evaluation any JPI can do. In the other hand, these two levels of evaluation are a basic input for the evaluation of the Joint Programming concept.Some assumptions.The table below is proposed to organize the framework for the evaluation of any JPI. It is based on the assumption that JPIs have to perform three basic functions:A. Governing research policy making, B. Governing (guiding, incentivize) research and implementation performance C. Involving stakeholders to ensure responsiveness to societal needs and innovation. Stakeholder involvement should occur on all levels and particularly be reflected on the level of research programme and project design. There should also be a formal approach to stakeholder involvement, e. g. integrated in evaluation procedures.The assumption includes the idea that it aims to be exhaustive, any activity or element of a JPI may be included in one of the three functions. The information about these three functions is classified either as structure, process or outcome information. The table provides the intervention logic of a JPI. The content of each cell is proposed following the priorities of the ERA and the framework conditions. Framework for the evaluation of a JPI Governing policy making Governing research performance Stakeholders involvement Structure ? Societal challenge ? JPI’s structures and procedures ? Existing funding quantity & type ? Scientific Advisory Board (SAB)? Strategic Research Agenda (SRA) ? Stakeholders Advisory Board (SHAB)Process ? Decision making? Leadership? External relations ? Peer review? Coordination of funding and agendas at EU level ? Mobility of researchers? Plans for SRA? Improving research capacities ? Use of Open Access ? Input of SHAB? IPR procedures for exploitationOutcome ? Satisfaction of Management Board (MB), SAB, SHAB and continuity of their commitment ? Scientific productivity? Products, tools, devices, policy options ? Innovation in products, tools, procedures and policiesHow to do self-evaluation of a JPI.Achieving societal impact is the key delivery of the joint programming process and the JPIs. JPIs focus on societal challenges and combine scientific quality with societal relevance and societal impact. This requires new approaches to evaluation and careful consideration of criteria for success, which need further development. Evaluations need to acknowledge JPIs as on-going processes and collective learning experience aimed at both, scientifically reliable and socially valuable knowledge.• Complete the above table with the additional elements needed in a specific JPI:o Identify additional elements not included in the table. These additional elements may be determined by the evolution and maturity of the JPI.o Classify them in the dimension of governing policy making, guiding research performance or stakeholder’s involvement.o Classify additional elements also as pertaining to structure, process or outcome.• Specify desirable characteristics of the elements to be evaluated as quality criteria.Define when they should be considered satisfactory.• Decide how to measure whether the criteria are met, defining indicators. • Identify the sources of information to obtain the values of the indicators.Usually interviews, surveys, review of documents or data bases. • Set up the mechanisms to have the needed relevant and reliable information in time.The JPI has to develop its own information system for monitoring and evaluation purposes that allow for decision making. Information should be reliable and be available in time. Information about national programmes in the same field is needed to be able to assess the added value of the JPI (See above in “Governance”). Specify a distinction between monitoring and evaluation as appropriate. While evaluation includes all relevant criteria and indicators, monitoring would include only those providing added value at a given moment in time. Expected output of self-evaluation.• Policy learning.The parties interested in a JPI evaluation include the Commission and the Council at the EU level, national ministries and funding agencies at the participating countries level and the GPC and Management Board at the activity level. On the basis of the evaluation these stakeholders will draw conclusions towards policy learning.• Legitimation of the JPI.o Demonstrate added value The output at the research level should provide information to what extent the JPI has boosted the scientific productivity and societal relevance of research related to the grand societal challenge of the JPI as compared to not having a JPI. o Show effectiveness. Information on how the progress in the specific research domain results in associated progress against the grand societal challenge.o Demonstration of a clear link from research to innovation, and also wider complementary actions such as sectorial regulations and policy measures. From the stakeholders’ perspective it is important that the JPI discovers new tools, products or procedures that offer greater value for the same cost or same value for less cost. It is also important to secure stakeholder’s involvement (notably the industry) or even active participation in research activities. This would help the stakeholders to benefit from the knowledge transfer as well as facilitate the future use of research in order to respond to the societal challenge.Recommendations for Funding.Michael Dinges, Susanne Meyer.JPIs need to create a critical mass of joint actions (i.e. pooling and leveraging financial, institutional and human resources in participating countries). Therefore, next to the alignment of national policies the development of corresponding funding models for the strategic management and programming activities (i.e. their management structures) and the joint research activities of JPIs are important. JPIs need to mobilise funding and exchange to ensure actions of JPIs to be impacting, and all funding mechanisms and instruments need to go along with a certain degree of stability. This includes programmes at Member State level and at European level. Given the limits on H2020 and other European budget lines, it is increasingly clear that the approach towards joint research programmes is aligning national programmes and funding. JPIs expand beyond joint call management and joint calls are considered only one amongst many other ways of mutually beneficial collaboration.In the following, we highlight critical issues and options for framing sustainable funding systems of JPIs for funding of a) strategic and management actions and b) joint actions.Funding of strategic and management actions. How to do it.• Choose the desired combination of funding from a Coordination and Support Action (CSA) and contribution of MB member organizations.The table below synthesises these basic options for funding strategic management activities. Most JPIs use a combination of CSA funds and national contributions, mostly in-kind, to maintain strategic and management activities. • Choose the adequate balance between cash and in-kind national contributions.The contribution of MB members is always in-kind. Added to it, countries may contribute either in cash by fees or providing in-kind services. Fees may represent an administrative difficulty as compared to in-kind contributions in some cases. In the other hand, in-kind contributions may draw the attention of MB members towards operational and administrative duties and to some extent distract them from strategic issues. • Plan for sustainability.The chosen balance between CSA and national funding has to take into account how to address sustainability, depending on the chances to get continuous H2020 support. Implicit in this plan is the right dimension of the JPIs secretariat.Funding of strategic and management actionsOption Benefits ChallengesCoordination and Support Actions funded by H2020 ? Project structure with liability and control mechanisms ensured by EC/H2020 regulations. ? Tasks and responsibilities of partners regulated by project contract.? Involvement of partners regulated by proposal preparation ? Potentially limited actions by Member States to take over responsibilities for JPI development.? High transaction costs through involvement of many partners ? Options for setting up physical management agencies and delegation to national agencies might be limited? Sustainability after CSA.Re-circling: Use national funding to finance strategic management activities of JPIs ? Commitment of Member States visible through financial contributions and hence easy identification of core actors.? Flexibility in management of strategic activities ? Mechanisms for partner involvement need to be negotiated. Duties & rights of Member States have to be defined.? Regulations considering in-kind contributions have to be taken into account.• Choose an organization modelOrganization models are related to funding modalities. National contributions by fees are compatible with the delegation to a national agency, with the creation of a new legal entity or with outsourcing to an existing multinational agency. The following table shows three options with associated benefits and challenges.Organization of strategic and management actionsOption Benefits ChallengesDelegation to a national agency – single point of contact ? Use of existing agencies may save costs.? Routines & knowledge of national agencies can be utilized. ? Neutrality of national agency might be questioned due to prevailing national interest.? Limited involvement of many partners. ? Performance contracts between agency and JPIs need to be developed.Creation of a new legal entity or delegation to a multinational agency ? Increased visibility through new corporate identity.? Relative independence vis a vis national stakeholders. ? Yet another programme management authority. ? Increased complexity of funding systems. ? Performance contracts between agency and JPIs need to be developedVirtual management agencies - organisation by group of agencies / organisations ? Member States can flexible contribute according to their capabilities in terms of financial and in-kind contributions.? Light structures and limited financial cross-border flows necessary. ? High organisational transaction costs.? Responsibilities & duties have to be negotiated across many Member States.? Lower visibility.• Explore collaboration in the case of outsourcing or creation of new entities.Several JPIs may benefit from jointly delegating some administrative and management tasks, such as organizing meetings, organizing calls or disseminating results, to the same entity. This would provide economies of scale, avoid low activity periods, and more homogeneous procedures. Expected output of funding strategic and management actions.• Availability of necessary resources to conduct strategic and managerial tasks.• Commitment of MB members and corresponding governments, which favours sustainability.• Sustainability plans. Funding of Joint Actions.How to do it.• Revisit the Strategic Research Agenda and the annual action plan.• Define the actions to be implemented by the JPIs such as: a. Joint research actions, b. Cross border mobility and training of human resources,c. Knowledge sharing activities, d. Network activities, e. Use of Joint Research Infrastructures, f. Alignment of policies and programmes, and g. International outreach activities.• Set up multi-annual funding plans for these actions, defining how challenges are going to be approached by the JPI. The table below specifies some of the issues to be addressed.Establishing funding plans for a JPIIssue SpecificationBudget needs ? Consider how much funding has to be provided for a multi-annual funding plan in order to effectively implement the Joint Action. ? Detail minimum and maximum budgets and state the desired impact to be achieved. This provides realistic expectations on the achievable impacts of Joint Actions. Ensure a budget size enough for activities to encounter the addressed grand challenge by JPIs.Sources of funding & funding sustainability ? Detail which sources are going to be used and elaborate plans on how sustainability of funding sources can be reached. Most likely, numerous funding sources from various actors will have to be used for implementing the Joint Actions. ? Exemplify the use of instruments of Horizon2020, national programmes, funding from industry members, institutional funding of ministries, etc. ? Elaborate plans and procedures which ensure commitment of countries in terms of sustainability. This includes, for example, specifications for embedding the SRA in the national research landscape in order to better start off with joint actions.Spending priorities ? Define efforts and costs accrue for setting up the Joint Action and the amount of budget to be spent on performing the Joint Action. ? Specify the purpose of the funding at the level of Joint Actions. Funding conditions ? Reach an agreement amongst JPI participants on how far alignment of funding conditions should be achieved (development and implementation of joint conditions, lead agency procedure, development of new agency). • Consider the following questions to determine the best funding model:Best funding modelsIssue SpecificationActors to be involved ? Specify type and numbers of actors to be involved in joint actions. Type of actors may include funding owners, policy makers, programme managers, research performers and other stakeholders. ? Funding rules and incentives must be developed in accordance with the actors´ needs.Type of investment needed ? Funding may be invested primarily in form of institutional funding or project funding (tenders, calls, or free proposals) or a combination of both.Competitive procedures ? Funding for actions may be provided in competitive or non-competitive manner.? In particular actions pooling use of large infrastructures may not rely predominantly on competitive funding.Cross Border Mobility of Funding ? Cross-border mobility of funding might be more or less relevant depending on the specific actions. ? Appropriate approaches need to be designed (e.g. money follows people, money follows research activity).Adaptation of funding conditions ? Funding conditions may have to be adapted for different target groups (universities, RTOs, enterprises, cities, NGOs etc.). The table above indicatively highlights issues that need to be considered for designing Joint Actions and may have a considerable impact on funding modalities and long-term planning of the Joint Actions.• Strengthen the link between funding and added value.Try to get a common minimum set of mutual compatibility conditions of national programmes in all activities possible: evaluation of proposals, selection and reporting procedures, agreement on principles, mutual recognition of procedures, common processes and agendas. o Make a clear distinction between these activities and funding. o Then analyze whether the current link between these activities (evaluation of proposals, selection, and mutual recognition of procedures or common agendas) and funding decisions is the best. o Associate funding decisions to the best available modality of implementation of these activities. • Promote agreements on common funding and other framework conditions across JPIs to ensure coherence.JPIs need to define joint framework conditions for funding of all of their activities and approaches need to be harmonised across JPIs in order to ensure easy accessibility for new entrants.• Promote national programmes coherence on funding modalities towards all JPIs.Expected output of funding joint actions• Achieving critical mass by supporting collaboration of all existing capacities.• Achieving highest excellence by mobilizing the best scientific capabilities across Europe.• Highest effectiveness against societal challenge by prioritizing those actions that focus on the challenge regardless of internal borders between countries, agencies or procedures.• Improving cost-effectiveness by providing best results at any given investment levelo Avoiding unnecessary duplications.o Covering gaps.Recommendations for Call Management and Peer Review.Michael Dinges, Susanne Meyer.How to do it in a JPI.• Clarify the difference between peer review and funding as linked but clearly different processes.Peer review is seen to be “at the heart of any excellence-based research policy and practice” as it informs the decisions on which researchers will be funded and needs to be accompanied by suitable review of societal relevance. However, peer review and funding are clearly different activities. Therefore funding has to be linked to the results of peer review, but the latter can be provided by a variety of agencies the funders may choose from. Hence, JPIs need to develop measures to utilise existing capacities and make best use of established practices and competencies of existing expert organisations. • Tailor review to the JPI.JPIs should not underestimate their peculiarities. JPIs encounter societal challenges by construction. Inter- and trans-disciplinary research is therefore one of the key challenges of JPIs. Inter- and trans-disciplinary character of JPIs should be reflected in Joint Actions and considered in the proposals selection processes and criteria. Major peculiarities of JPIs are:1. Involvement of many countries and partly large projects.2. Inter- and trans-disciplinary nature.3. Encounter of societal challenges (problem orientation): excellence vs. relevance approach.4. Uptake and innovation: User involvement, knowledge transfer, uptake of research results, innovations.• Analyse the following topics of specific interest for JPIs, to be considered in the call management / review process of Joint Actions:Selection criteria for proposals? Harmonize selection criteria for all JPI proposers. Avoid specific national criteria for project partners as this may distort the ranking of proposals.? Consider the core evaluation criteria mentioned in the ESF guideline: a. Relevance and expected impacts (driven by programme policy), b. Scientific quality and societal relevance of the proposed research, c. Research team / qualification of the applicants, d. Research environment.? Tailor selection criteria beyond core criteria towards the needs and objectives of the calls, e.g.: a. The potential impacts and value to user communities, b. The involvement of stakeholder communities, c. Interdisciplinary and cross sectorial collaboration.? Specify the relevance of certain criteria beyond scientific excellence, e.g. with weights: a. How important is the existence of sustainability plans? b. How important are models for ensuring science-society interactions? c. How important are knowledge diffusion measures?d. How important is the expected social or economic impact?Diffusion and implementation? Integrate mechanisms into the call management procedures which ensure that plans for diffusion and implementation as well as integration of relevant societal stakeholders are considered in joint activities funded by JPIs? Selection processes need to carefully consider inter- and trans-disciplinary approaches as well as plans for uptake and diffusion: o Provision of clear advice for proposers and reviewers on the balance between scientific excellence and relevance of proposals.o Provision of clear advice on relevance of diffusion and implementation of measures.General quality criteria? Integrate mechanisms into the call management procedures which ensure that the following issues are addressed:a. Respect for ethical principles on research performanceb. Gender equality or gender relevance in the subject at studyc. Open access to data basesSelection of reviewers? Organize review :o Use of external or individual remote reviewers who assess proposals on their own and separately from other reviewers who may look at the same proposals. o Use of mixed review panels who will collectively discuss and evaluate groups of proposals. The main function of the panel is to evaluate and consolidate external assessments by experts on a group of competing proposals and to rank or prioritise them based on clear and stated criteria and parameters. The review panel’s contributions are normally needed within the last phase of the peer review when final decisions are made.? Identify types of experts needed/mix of expert panels : o Adjust review panels and peer review procedures to the specificities of joint research actions to be carried out by JPIso Use a pool of experts, clustered according to a multi-level scientific classification system. Alternatively, consider matching keywords between proposals and reviewers profiles. ? Decide on the degree of expertise you need in a specialty field (i.e. including specialist and generalist or interdisciplinary panellists). o Interdisciplinary panellists are capable of understanding, judging and comparing a wide variety of proposals without having fully developed expertise on the subject matters.o High overlap in disciplinary panels may induce a relatively tight disciplinary control between participants. o A multidisciplinary design in turn, seems to create a shared sense among panellists that they are accountable for their judgements to a number of different disciplinary communitieso The extent to which panellists hold themselves accountable to each other for their evaluative behaviours important for the peer review process.Connections and responsibilities of agencies? Establish appropriate means/mechanisms to connect funding owners and managers from different nations and backgrounds. Detail tasks and responsibilities.? Detail criteria to be applied for selection of reviewers? Provide homogenous briefings for remote reviewers and panellists, in order to ensure that specificities of JPI actions are taken into account.? Establish means that calls reach the appropriate communities and consider a timing which allows also stakeholders whose work is mainly research to respond to calls.? Detail conflict resolution mechanisms.Consider usability for monitoring and evaluation purposes? Establish Joint project databases which provide a good baseline for respective monitoring and evaluation activities. ? These databases should at least include detailed information on a) Partner details active in a project (type of partner, role of partner, etc.),b) Project facts (duration, funding per partner, funding per type of activity etc.), and c) A limited number of key performance indicators mirroring performance of projects in reviews and potentials for up-take and impacts of projects).Expected output of call management and peer review and review of societal relevance in JPIs.• Focused research with the required inter- and trans-disciplinary approach.• Best peer review available:o Greatest pool of reviewers.o Best expertise.• Higher societal relevance of research.• Greater consistency with ERA objectives.• Monitoring capacity at European level.Who can help?The project JPIs2CoWork has shown that a lot of experience exists regarding the implementation of peer review procedures among JPI Member States and at European level, which JPIs could more extensively use. Peer review expert organisations and national agencies and funding organisations have already gained experiences in managing joint calls with funding stemming from various national and international sources such as the ERA-Net activities. Also, tool-boxes and guidelines such as the European Peer Review Guide of the European Science Foundation (ESF) or the ERALEARN toolbox highlight major principles for adapting appropriate peer review mechanisms. What still needs more attention is the development of appropriate methods to review societal relevance.Recommendations for Dissemination and Intellectual Property Rights looking for Innovation.Barbara Ritz, Elisabeth Sjöstedt, Birgitta Myrman.Societal challenges are the main drivers in the JPIs. A JPI is not a research project or programme for networking and building new researcher communities. Dissemination and take-up of research results are therefore critical issues to be addressed, so as to ensure transparency, promote good science, engage society and raise public awareness. JPIs are refreshing opportunities to do things differently, and to show that there are other ways to handle knowledge transfer and spreading of research results. JPIs need to carefully consider how to handle issues like open access to data, dissemination of results to society, engaging industry, etc., in order to contribute to solve grand societal challenges. Setting up a knowledge management system.How to do it.Dissemination and the protection of intellectual property are better considered as parts of a knowledge management policy and system, that in turn is a piece of any policy aiming at promoting innovation. • Define a knowledge management system and policy.Describe the strategic relevance of managing knowledge and the results of research, by identifying:a. The societal needs.Who needs the results of research to address a societal problem or part of it.How can they use the results, what is the scope of action of all interested actors.What is the best way to transfer the knowledge that the JPI can provide so as to stimulate its use.b. How to Improve capacities for further research:Consider opening your knowledge to the research community.c. Explore whether this knowledge management system can be used for national programmes also. National programmes may use it to manage the results of their national research.• Identify those who need your knowledge, the stakeholders.Describe them as reliably as possible. Each has different needs and preferred ways of communication. Usually part of the following communities:a. Citizensb. Private companies of industry or servicesc. Public administration and policy makers, including those related to researchd. Scientists and research institutions.• Use the Stakeholders Advisory Board. Ask the SHAB to provide information about the specific needs of all stakeholders (except scientists), and about the best channels and modalities of communication. Ensure the involvement of the SHAB in the whole process of knowledge management, from the very beginning. • Design and implement a knowledge management plan. Tailor the plan to the needs and communication channels of groups of stakeholders. The plan should include: o requiring specific activities to be included in all projects o specifying activities at central level of the whole JPIo specifying conditions for national programmes to join the plan. • Set up an Open Access policy. o Define procedures to use Open Access in all publications of research funded by the JPI, to be included in the funding agreements of any activity.o Include recommendations regarding the license of the data published. Licences can contain clauses that also require citation of the underlying research. Data issues that may violate personal integrity (registry data, etc.) must also be considered. An OA policy on research data must be formulated in such a way that there is room for exceptions for such data.o Set up a JPI repository of documents and publications.o Use the OpenAire infrastructure or validate the JPI repository .o Consider Open Access to data bases and use embargo periods as appropriate.• Set up a policy for intellectual property (see below) as part of the integrated knowledge management plan.Expected output of managing knowledge.• Increasing the likelihood that research results are used by stakeholderso Promoting social innovationo Promoting evidence based policy makingo Facilitating innovation of companies.• Increasing the research capabilities by: o facilitating state-of-the-art knowledgeo better and more intensive use of data baseso facilitating the avoidance of unnecessary researcho increasing the leadership of the JPI in the field.Protecting Intellectual Property rights.An active and proactive approach to IP issues should be adopted from the beginning, to optimize the uptake of research results by the economy. In the last decades, the concept of intellectual property (IP) has been broadened to include business and investment tool besides its traditional legal purpose. Indeed, industry and academia are facing a new reality in terms of how to best integrate and optimize the application of IP to suit the demands of the knowledge economy. How to do it.• Define a JPI policy on intellectual property rights (IPR).A clear policy or strategy to connect research performance and the potential users of research results is to be defined in all JPIs to promote innovation. A common IPR policy should be debated early on. Since national laws as well as European laws apply (national patent and EU patent), the funders can decide to “go a specific way” when doing the research and enter this into the agreements with the researchers. • Make the IPR policy compatible and coherent with the knowledge management policy.Open access has to be compatible with the protection of IPR. There must be a screening mechanism in place to select results that are to be protected. Silent periods may be needed before publication. Screening may be an activity delegated to researchers themselves or may be centrally organized. IPR policy should be part of the more general knowledge management policy.• Define a JPI policy to relate to private companies.A joint platform to expose a repository of patents and other protected intellectual properties should be set up. The platform should be the easiest way of finding an interesting output of research for any European company. Explore whether the platform could be useful to expose outputs from national programmes as well. Expected output.• Increased likelihood that research outputs are used by private companies (that as a consequence bring about new products or production procedures, finally providing new solutions for societal needs). • Greater interest of private investors in research.• Greater interest of publicly funded researchers on industrial needs. Potential Impact:The basic objectives of JPIs To Co-Work were:• provide a forum for discussion, exchange of experiences and best practices • implement a process of mutual learning amongst Joint Programming Initiatives (JPIs) around the so-called Framework Conditions • promote the contribution of JPIs to the European Research AreaForum for exchange.JPIs To Co-Work has organized six meetings in the 30 months of duration of the project. The first workshop included presentations of seven JPIs, the discussion of the results of a survey made intended to identify needs felt by JPIs related to the framework conditions, and some questions to be addressed by the project. Four thematic workshops followed, each focused on a couple of framework conditions. The mean of attendees in each has been around 45, and 25 of them have attended most workshops. They were members of the Management Boards of different JPIs, members of the GPC, members of the secretariats, partners of the project, advisors (including persons from IPTS, PLATFORM, and some JPIs), representatives of COST, scientists and the project officer. All workshops included group discussion for more than half of the schedule. The evaluations of the workshops showed that group discussion were most appreciated by attendees, and even more time for discussions was frequently demanded. The final conference was again an opportunity for discussions and exchange of experiences. It was attended by 70 people including members of all JPIs. The website https://web.archive.org/web/20170914155837/http://jpis2cowork.eu/ has opened a page on best practices, including the framework for monitoring and evaluation of JPI AMR, recommendations of PLATFORM for call management, and procedures for the design and management of joint actions of JPI Oceans. Mutual learning.The workshops have provided as described above a unique opportunity for JPIs to learn from each other. JPND has presented its evaluation framework in a couple of opportunities during the project. The final conference included several sessions around the activities of JPIs. All of them made an update on their most recent developments. There was a specific session on best practices, with the presentations of FACCE about mapping, of JPND about the COEN initiative, and of HDHL about the knowledge hub DEDIPAC. We also had the opportunity to share the evaluation framework and discuss it thoroughly after the corresponding workshop. We met twice in Brussels with representatives of the ten JPIs with this objective. The framework was well taken and at least four JPIs asked for advice on their specific evaluation plans. FACCE invited the coordinator to discuss and advice on their own monitoring and evaluation plans in Copenhagen in August 2013. Contribution of JPIs to the ERA.The general perspective adopted by JPIs TO Co-Work has intended to be strategic in the sense of having in mind the survival of joint programming in the context of the ERA objectives. Optimal transnational cooperation and competition, more effective national research systems and optimal circulation, access to and transfer of scientific knowledge have been always in the background of the project.An initial document entitled "A functional approach to joint programming initiatives" setting some basic ideas was made open at the beginning of the project. It has been hit 1660 times in our website. The document presented the idea of defining research programme functions as governing policy making, governing scientific performance, and involving stakeholders, which was the base to later propose an evaluation framework. The document compared JPI with ERAnets and Art. 185 initiatives, and stated that "programmes are not projects".These background has been increasingly identified by workshop participants and project partners. There have been two slightly different approaches depending on the importance conceded to different elements of joint programming. In the first approach the accent was put on the aim of contributing to solve societal challenges. In this approach transnational cooperation and collaboration was seen as motivated by the need to address these challenges, and the attention was put on the peculiarities of the corresponding research programme features, such as stakeholder participation in reviewing project proposals. In the second, the most important element was transnational cooperation and collaboration as a general need for research in Europe, that can be better understood when related to grand societal challenges, but no only. From this second perspective, transnational cooperation and collaboration is needed also to improve national research systems, and the accent is put on the difficulties of opening national programmes and joining forces.The final conference was designed to uncover some of the assumptions and determinants that facilitate or not the development of FC and JPIs in general. Amongst them we discussed:• the relative small number of institutions involved in JPIs, which should facilitate a coherent approach for some basic elements in all of them.• the determining influence of national programmes, which are finally supposed to be coordinated.• the need to connect national and transnational programming through some common research agendas or priorities.• the need for flexibility in the way national programmes choose to connect to JPIs.• the need for JPIs to think more strategically and not losing the overall picture by being busy in operational issues. Our perception is that JPIs TO Co-Work has contributed to raise the understanding of JPIs issues to a more strategic level.List of Websites:https://web.archive.org/web/20170914155837/http://jpis2cowork.eu/The Website was created and launched during the frist six mount of the project. , The web site contains basic information on the aims, objectives and structure of the project, as well as relevant links to JPIs and a members’ area for project partners.There are some private accesses to members of the Advisory Board. The website is updated as required. Updates are expected approximately every three months. After submission of the public project deliverables to the European Commission, the final deliverables will also be posted at the web portal. The web portal will be maintained for six more months after the completion of the project. This website will also include different restricted areas serving as a management tool for the project. The web will provide links to all running JPIs, NETWATCH and ERA-LEARN news.
