Final Report Summary - EVAL-HEALTH (DEVELOPING AND TESTING OF NEW METHODOLOGIES TO MONITOR AND EVALUATE HEALTH RELATED EU-FUNDED INTERVENTIONS IN COOPERATION PARTNER COUNTRIES)
Executive Summary:
EVAL-HEALTH ("Developing and Testing of New Methodologies to Monitor and Evaluate Health Related EU-Funded Interventions in Cooperation Partner Countries". www.eval-health.eu) is an international collaborative research project funded by the European Union 7th Framework Programme (FP7). The main goal of the project is to contribute to strengthen the monitoring and evaluation (M&E) of European Union funded interventions in developing countries in the specific area of international public health. The project had two research components focusing on different aspects of the M&E process:
• to identify and analyze the role of different types of evidence in health policy in low and middle-income countries, including research results.
• Development and testing of a new methodology to monitor and assess the impact of research projects that are funded by FP7 in the area of International Public Health.
EVAL-HEALTH has run for 46 month, carried out by a consortium of 9 organizations from Europe, Asia, Africa and Latin America. The work has been organized in 7 Work Packages:
WP1: Comparative analysis of M&E tools and methods across major international donors in health.
WP2: Analysis of EC Programmes and instruments to deliver Health aid to developing countries and EC policy framework on Health and development.
WP3: Methodology for analysis of the role of evidence in health policy and strategy development in developing countries.
WP4: Development of methodologies and tools for M&E of R&D projects in developing countries.
WP6: Management and project coordination
WP7: Dissemination of research results
The project has produced, as major results, the following :
• A new methodology to monitor and assess the impacts of research project, named Impact Oriented monitoring (IOM), to help EC officers in the identification, recording and assessment of projects. Results and impacts are classified into four dimensions or categories: Knowledge production, capacity building and research targeting, policy and decision making and health and health sector benefits. The methodology has been tested and validated with finished public health projects funded by the FP5 and FP6 and proposes the use of different complementary implementing tools: a project results framework, a coordinator’s survey and an end users' survey and an assessment tool. By using this methodology. The complete description of the methodology was presented as a major project deliverable (D4.8). A scientific paper describing the IOM methodology has also been accepted for publishing in the journal Research Evaluation.
• A better understanding the role of evidence in health policy and strategy development. The research has focused on identifying and analysing the role of evidence in health policy and strategy development in one Asian (India) and one African (Nigeria) country . As results of the research carried out, we have prepared two country reports, one for India and one for Nigeria (Deliverables 5.6 and 5.5 respectively) and The cross-country analysis (Deliverable 5.7). Research carried out has provided evidence to give new insight on:
1. The different types and the key characteristics of evidence that informs health policy development in India and Nigeria.
2. The use of different types of evidence in health policy development, including the facilitators and constraining factors, in India and Nigeria.
3. An analysis of the strengths and weaknesses of existing mechanisms and practices of evidence use in health policy development;
Finally , the project presents and recommends ways of enhancing the role of evidence in health policy development within the study countries and beyond.
Project Context and Objectives:
EVAL-HEALTH project had as major goal to contribute to strengthen monitoring and evaluation of EU funded interventions in developing countries in the specific area of public health. The project has focused on two specific areas: International Public health projects funded by DG Research and Innovation and the role of evidence in health policy and strategy development in developing countries. Each research component has had its own objectives and deliverables, and for the purpose of this document, we will describe each component separately. For this, we can say that the EVAL-HEALTH project has worked towards achieving the following objectives:
A. Developing and testing a methodology to monitor and assess the impact of research projects in the area of international public health.
B. Contributing to a better understanding of the role of different types of evidence, including research, in health policy and strategy development within, and across, developing countries.
A. DEVELOPING AND TESTING A METHODOLOGY TO MONITOR AND ASSESS THE IMPACT OF RESEARCH PROJECTS IN THE AREA OF INTERNATIONAL PUBLIC HEALTH.
How to better assess the impact of research projects in the area of international public health and health systems is an important issue for the European Commission, mainly as a means to know the impact that individual projects may be achieving but also as way to understand how far EC public health research programmes are contributing to address global health challenges. EVAL-HEALTH had the goal of developing and testing a methodology that could support EC officers in improving the way research projects are monitored and assessed, providing tools that can help answering questions such as:
• To which extent are EC funded health research project producing impacts?.
• Are funded projects contributing towards the expected impacts listed in the Work Programme?.
There is a growing interest and need within research funding councils and agencies, international organizations and the European Union to identify and evaluate the outputs, outcomes and impacts of funded research as a means to both understand the value of investment and to increase accountability. More and more, organizations that fund research are under increasing pressure to justify and demonstrate that research provides value to the community. A great deal has been already published regarding good practice in the evaluation of research, including a growing body of material concerned with the elaboration of frameworks and methods suitable to evaluate the impact of research, both in general as for specific areas of knowledge, such as health, agriculture and environmental research.
Research outputs such as journal articles and other scholarly publications are a tangible result of research projects, easy to identify and to measure. Bibliometric analysis has been widely used to evaluate the scientific impact of a specific research project, programme or even research group or organization, and until not long ago, measuring the scientific impact from research was enough to evaluate a research project and programme and was the main benefit expected from research.
Current economic conditions and the increasing competition for government funding are leading to an increased focus on the impact of research outside academic environments, as a need to demonstrate that investment are effective both to improve the operation of research programmes and also to demonstrate to society that research funds are well spent. At the same time, there has been a growing willingness and enthusiasm from academia to explore the economic and societal value of research in a wide variety of forms.
The benefits and/or effects that research has beyond the creation of scientific knowledge, which may be labelled as academic impacts, can be grouped under the name of societal/social impacts or non-academic impacts. It includes impacts such as capacity building, facilitating evidence-based decision and policy making and benefits for the health system and the health of the population.
The research component of the EVAL-HEALTH project had the objective of developing and testing specific methodological models and tools to help project officers at the EC DG Research and Innovation (Health Unit), monitor and evaluate (M&E) the impact of EU funded research actions in the specific area of International Public health. The methodology had to serve the purpose of M&E Individual projects (research projects or coordinating and support actions) funded as ‘Specific International Cooperation actions’ (SICAs), which is the instrument most widely used by the EC to enable and encourage low and middle income countries to participate and get funding from the FP7. These type of projects require the participation of partners from outside the EU, mainly from low and middle income countries. They typically involve 8-10 participants, for a period of 3 years, and an overall funding of about € 2-3 million from the EC. SICAs are the continuation of the INCO actions of FP5 and FP6.
The majority of projects funded are thus international consortiums formed by a variety of organizations coming from different cultural backgrounds. This type of projects are expected to provide the means for addressing public health issues of importance in international cooperation partner countries and aim to reinforce the research and cooperative capacity of candidate, neighbourhood, developing and emerging countries.
EVAL-HEALTH will develop a methodology to better understand and measure a project 's success and help identify the impacts, including those benefits on the public health systems of partner countries participating in the projects. The methodology links with existing FPs reporting system by adding new tools for the collection of data and information, and also links with the thematic level evaluations, as it will provide important information from individual projects useful to perform these type of evaluations.
By using the methodology, EC officers should be able to:
i) monitor how the project is performing, identifying results and evidence to anticipate project impacts.
ii) assess the final results and provide an estimation of the type of impacts of individual projects
iii) help future programme evaluations.
Research work was carried out following a work plan that included the following activities
• Identify best practices in M&E of research actions from major international agencies and donors active in health research (WP1).
• Review main frameworks available for evaluating the impact of health research (WP1)
• Review and analyse frameworks, methodologies and tools currently used by the EC to M&E research projects and programmes (WP2)
• Develop specific methodological models and tools for M&E the impact of research projects (WP4).
• Field testing the suitability and applicability of the developed methodology through case studies (WP5)
• Disseminate the project results (WP7)
The methodology developed has been named as Impact Oriented Monitoring (IOM), as it tries to anticipate as much as possible the impact of projects during the monitoring process.
B. ANALYSIS OF THE ROLE OF EVIDENCE, INCLUDING RESEARCH, IN HEALTH POLICY AND STRATEGY DEVELOPMENT WITHIN, AND ACROSS, DEVELOPING COUNTRIES.
There is a growing recognition of the importance of evidence-informed health policy and planning decisions. Different types of evidence are referred to in the literature such as formal (e.g. Published research, results of monitoring and evaluation, HMIS statistics) and informal (e.g. personal experience, rumours, media reports). The above distinction is based on recognition that policies and plans are based on "something" whether it is clear-cut research findings or just the expectations that "this particular programme/approach will work". Frameworks exist in the literature to help understand, and enhance, the role of evidence in health policies and strategies. However, there is limited use of these frameworks in analysing the role of different types of evidence in health policy development in developing countries.
The Evidence Component of the EVAL-HEALTH project aimed to contribute to filling the above gap in the knowledge through identifying and analysing the role of different types of evidence in health policy and strategy development in one Asian (India) and one African (Nigeria) country. The specific research objectives were:
1. To identify different types and the key characteristics of evidence that informed health policy development.
2. To determine and analyse, within and across the case studies and the study countries, the use of different types of evidence in health policy development, including the facilitators and constraining factors;
3. To analyse the strengths and weaknesses of existing mechanisms and practices of evidence use in health policy development;
4. To recommend ways of enhancing the role of evidence in health policy development
within the study countries and beyond;
The research has been conducted in two countries: India and Nigeria, using a case study approach. For the purpose of this project, we defined ‘case study’ as an issue - for which a policy has been formulated - representing health services (HIV/AIDS, Maternal and Child Health, Non-Communicable Disease), health-related issue (Tobacco Control) or a health system’s component (Human Resources). In each country, three specific policies were selected corresponding to the three case study areas:
The rationale for selecting more than one country was that variation in contexts between countries would facilitate better understanding of the determinants of successful policymaking. Rather than enumerating just what worked, a comparative analysis of six policies helped to identify mechanisms of effective policy-making across cases and contexts.
In each of the two countries three specific policies were selected to examine the role of evidence in diverse policy areas and, to generate and disseminate lessons from the intra and inter-country comparative analysis. In India, the policies were: National AIDS Control Programme (NACP III); ii) National Tobacco Control Programme 2007 (NTCP); and the Accredited Social Health Activists programme of 2005 (ASHA). In Nigeria, the policies were: Integrated Maternal, Neonatal and Child Health strategy 2007 (IMNCH); Human Resources for
Health Policy 2006 (HRH) and Oral Health Policy 2012 (OHP).
The rationale for the choice of the above three case studies was two-fold: firstly, we wanted to avoid focusing only on areas of international prominence and, secondly, comparing findings between case studies and between countries, could allow generating and disseminating lessons learned.
Two qualitative research methods were used: document reviews and in-depth interviews. Key policy documents were reviewed and the views of policymakers, civil society organizations, health staff, development partners and academics were gathered using in-depth interviews.
The study was guided by a conceptual framework, drawn from the literature. A complete description of the conceptual framework that guided the work is presented in the project deliverable D3.1: Research methodology to analyse the role of different types of evidence (including EC-funded research) in health development within developing countries. Our conceptual interpretation of evidence was not confined to research only but included both formal (e.g. published research, policy evaluation reports, HMIS data analyses) and informal (media reports, personal experiences) types.
The role of evidence in health policies is perceived as interplay between evidence processes and policy processes. The policy actors determine this interplay through their roles in evidence and policy processes, their agendas and practices. All of above occurs within the wider context, which includes different national and international influences. Each component of the framework is complex and in-depth understanding of each component was required as part of analysis of the role of evidence in health policy development.
Framework approach was used to analyse data. In each country the role of evidence was explored in three specific policies: an area of international prominence (HIV/AIDS in India and maternal, neonatal and child health in Nigeria), a neglected area (tobacco control in India and oral health in Nigeria), and a health systems component (social health activists programme in India and human resource policy in Nigeria). The cross-country analysis (Deliverable 5.7) was performed using the country reports from India and Nigeria (Deliverables 5.6 and 5.5 respectively).
Project Results:
1) THE IMPACT ORIENTED METHODOLOGY (IOM) METHODOLOGY
The EVAL-HEALTH project has designed a methodology to monitor and assess the impacts of public health research projects funded by the European Commission through its 7th. Framework Programme. This Methodology has been named as IMPACT ORIENTED MONITORING (IOM). It has been designed with the objective of helping the EC DG Research & Innovation officers to know what the impacts of funded projects is. We have worked specifically for projects funded in the area of International Public Health, but the methodology could be used for all Public health projects, and with some minor changes, to the whole health research area.
Before describing the methodology, it is important to define some M&E terms that will be used throughout the document and that are important to better understand the IOM methodology.
RESULTS: The output, outcome and/or impact (intended or unintended, positive and/or negative) of a project.
REPORTING: Periodic submission of project documents during its development, at the end or after its completion (in the FP7 funded projects the EC reporting requirements are indicated in the Grant Agreement).
IMPACT MONITORING: A periodic identification and assessment of the impacts of a project
EX-POST EVALUATION An in-depth assessment of how a completed project has contributed to its overall expected results.
INPUTS: Resources that feed into the project.
ACTIVITIES: Actions that describe how the inputs are used to carry out the project (WPs activities).
OUTPUTS: Direct “products” of the project activities (Deliverables).
IMPACTS: Effects (changes) produced by the project activities and output. We can differentiate between:
- Immediate impacts. Those likely to occur within the project cycle.
- Short term impacts (outcomes). Impacts likely to occur within 3 years following project completion
- Long term impacts: Impacts likely to occur between 3 -10 years after project completion.
(Note. These definitions have been taken from the document: The European Evaluation Consortium . (2005). Analysis of ‘high impact’ research activities under Community Research Framework Programmes. DG RTD Framework Contract No: BUDG-02-01 L2)
IMPACT CATEGORIES (Health Research): Broad classification in different domains, where the impacts of health research projects are produced (capacity-building, knowledge production, informing policy, health, etc).
The IOM methodology can be used to identify and assess the impacts of funded projects on International Public Health research, with specific focus on:
i. Scientific outcomes and impacts: training of researchers, quality of research, constitution of international networks, etc.
ii. Impacts of the project results on the Public Health policy of the developing country and
iii. Impacts on the health of the population and other broad economic and social impacts.
1.1 CONCEPTUAL MODEL FOR THE IOM METHODOLOGY
The unit of analysis for our methodology is individual research projects.
The conceptual model designed to understand the process from project to impact in the context where the IOM methodology will be applied (The EC Framework programme) has been built upon the strengths of the Buxton-Hanney Payback model.
The IOM conceptual model uses a logic model approach to describe the process from research input to project outputs and impacts. The model illustrates the theoretical pathways underlying the research cycle, and how the different results are translated to immediate, short-term and long term impacts. The model identifies what in theory can be expected as the main outputs of the projects, the direct results (scientific publications, patents, better trained researchers, educational programmes..). These results can be measured and can be easily attributed to the funded project. The use of these results by different stakeholders can have an immediate /short term impact, a benefit of the project. If the research results are used by the scientific community then the project will have contributed to advancing knowledge, if the results or findings are used for informing policy decisions or for improving health practice, then the project will have had an impact on policy and decision making. These type of impacts could be regarded as immediate impacts, which result from the direct use of the project outputs. Further on in the model, policy decisions based on research evidence can lead to a better health policy, which in turn may have as a consequence an improvement in the population health, better treatments and use of medical resource and improved equity. Improving health practice will also have a benefit in the health of the population. All these effects, difficult to attribute exclusively to one research project and also difficult to identify and measure, normally take place long after the research project has finished and can be categorized as impacts or benefits on the health system. Finally, long term impacts expected from health research are improvements in health and well being of the population (i.e. achieving the MDGs) and economic and social prosperity. These long term impacts are the final goal of any (health) Research and Innovation programme, but can never be attributed to individual projects, but more to RTD programmes over a long period of time.
1.2 IMPACT CATEGORIES FOR THE IOM METHODOLOGY
The IOM methodology uses the impact categories for health research proposed by the Payback model to guide the operability of the conceptual model and its implementation. A short description of the impact categories and how we expect to gather evidence for each of them is provided below.
1) ADVANCING KNOWLEDGE
Producing knowledge is usually the first output from any research, and codifying this knowledge in a scientific paper, report or book chapter is vital for it to have some impact and thus contribute to advancing scientific knowledge. Research results are traditionally published in peer-review journals, but can also take the form of patent applications, public databases, research methods or even policy briefs.
In order to assess the impact on knowledge production gained by the projects, it is important to gather evidence on research outputs that could benefit the research community by entering the "pool of knowledge", mainly peer review articles published in indexed journals and if possible, try to have some indicators on the scientific impact, such as citations or other bibliometric analysis. Other indicators used to follow up the impact on knowledge are: other publications such as books, book chapters, conference proceeding; research reports, policy briefs and other "grey literature" produced and Systematic reviews.
We have included also in this category patents, as these documents also contribute to the pool of knowledge, and other research outputs such as databases or software that could also serve to advance scientific knowledge.
1) RESEARCH TARGETING AND CAPACITY BUILDING
Research targeting refers to the way the research project can contribute to a better targeting of future research, carried out both by the projects researchers as by others. In the first case, it can be tracked by gathering evidence on the existence of follow-up research projects gained by some members of the research team, research collaborations between project participants or with others. Any evidence that the results or findings from the research may have served to set up new research topics in future programmes will be valuable, but this type of evidence is difficult to follow.
Capacity building refers to both the gain of skills to undertake future research inside the project team, as well as an increase of the research capacity at the organization level (i.e. new research infrastructure), that is, how research can increase or improve the research capacity of individuals, teams and organizations. This benefit from the research project is quite straightforward to understand but should not only be based on counting the number of researchers and trainees participating in the project. It is important to also measure the number and level of higher or research degrees resulting, either totally or in part, from the research funding (how many trainees have gained a PhD, how many Post docs have acquired the category of junior researchers, etc.). Any evidence that can be captured of career development is valuable to assess the impact on capacity building. This impact is especially relevant in projects such as those funded in the Area of International Public Health, which have partners coming from low and middle income countries who can benefit with an increase of their research skills as results of their participation in the project.
Training that may take place outside the consortium as results of the project should also be considered when assessing the impact on capacity building, as many projects in the Area of International Public health have, as main goal, health training and capacity building of target groups in participating countries.
3) INFORMING DECISION MAKING, PRACTICE AND POLICY
The utilization of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. Health research can be used to inform policy and decision making in a wide range of circumstances and at many levels: Governmental national and local policies, managers within health services, health care practitioners (i.e. clinical guidelines), professional bodies, education and training, etc. It can make a contribution in at least three phases of the policy-making process: agenda-setting, policy formulation and implementation.
It is important for any research project to identify and assess impacts on policy and practice, especially in clinical, public health or health service research. These type of impacts or benefits are normally considered as a "secondary output" of the research process to distinguish them from the "primary or direct outputs" of research such as journal articles, other publications and trained researchers.
The most direct evidence for this type of impact would be direct citation of the research in a policy document, testimony of policy and decision makers or any evidence of implementation of influence, which are most of the times difficult to obtain. Evidence can be more easily collected by identifying indicators on the process for informing policy (i.e. participation of project researchers in Policy Advisory committees) or even in the way engagement with relevant policy makers has taken place during the project life (how and when), as these are considered factors of success for translating research into policy and practice.
4) POPULATION HEALTH AND HEALTH SECTOR BENEFITS
Pursuing health benefits for target population are the major final goals of any public funded research, and being able to provide evidence that this goal has been achieved is a major interest for all funding agencies to account for public funds spending.
Real evidence for this category of impact is very difficult to identify, and even more to measure and assess, especially at the level of individual research projects (as is the case for the IOM methodology). The attribution challenge that arises when trying to assess research impact is, for this type of impact, very relevant. Gains in health are normally the consequences of many research works along many years, together with many other variables (economic growth, environmental conditions, education) and circumstances (Political situation).
Nevertheless, it is possible to gain some insight on the contribution that the project may have to potential gains in population health and health sector benefits. Collecting qualitative information on how the project may contribute to issues such as improvement in clinical treatments, improvements in health service delivery and reducing the cost of health delivery can provide some insight on the project's contribution to impact. Moreover, for the specific projects funded in the area of International Public Health, it is important to know if the project may contribute to specific issues such as strengthening health care systems, better understanding and improvements in maternal and child care and reproductive health, increase knowledge and treatment of communicable and non communicable diseases, etc. We can note here that these health impacts are normally the final goals that the research funder sets (in our case the EC) when launching a specific research programme such as International Public Health research.
THE IMPORTANCE OF DISSEMINATION AND KNOWLEDGE TRANSFER ACTIVITIES FOR INCREASING THE IMPACT OF RESEARCH.
The nature and degree of impact any research may have, relies greatly on the effort put forward by the research project towards disseminating the research results and findings and actively engaging with key project stakeholders and end users. This is even more important for any research that produces evidence for policy or decision making, as the impact greatly depends on an active uptake of research findings by those who have to take decisions or implement specific health policies. Results need to be up taken and used at the level of policy and decision making to have a positive impact further on, for example for strengthening health systems and improving health.
Through the cases studies work carried out by the EVAL-HEALTH project to develop the methodology, we were able to confirm that successful projects had thoroughly worked towards actively engaging with all important stakeholders, such as Ministry of Health, local authorities or health practitioners in countries where the research could be used to improve the country's health system.
For the IOM methodology, we consider that dissemination (understood in a broad sense and including both active and passive dissemination actions) is an important factor for a project to have a wider impact. For this, we have included in the coordinators' survey and in the scoring matrix, a specific section on Dissemination to help in the assessment of project impacts.
1.3 IMPLEMENTATION TOOLS OF THE IOM METHODOLOGY.
Establishing a good system for capturing and recording appropriate project data during and after the life of the projects, can provide sufficient information to identify and assess immediate and short term impacts. The IOM methodology provides the guidance and tools to set up an Impact Monitoring System that can be used to gain information on the impacts of individual projects.
During the life cycle of the project and shortly after, impact can be monitored by collecting data and information on:
• Direct project results
• Potential and/or direct use of project results and findings by key stakeholders
• Dissemination and knowledge translation activities, (all those activities that facilitate, foster or are necessary for research to have an impact ).
These principles have been the basis to design the three specific data collection tools and the assessment tool used in the Impact Monitoring System. Each of these tools provide valuable information to help assess the impact of the projects at different stages and with a different level of understanding, and together they conform the IOM methodology. A short description of each tool is provided below:
- THE COORDINATORS' SURVEY
It is the main data collection tool for capturing project results and evidence of the impacts of the research. It is implemented as a web-based questionnaire, that should be completed by the project coordinator at the end of the projects and 3 years after the project has finished. For projects lasting 4 or more years, a questionnaire could also be used for midterm data collection. It can be used for project management (monitoring), project assessment and programme evaluation.
To facilitate the completion of the questionnaire and reduce the administrative work load for the coordinators, completion of the questionnaire should be linked, as much as possible, to reporting periods required by the Gran Agreement. We suggest the possibility of producing only one technical report, based mainly on the questionnaire, especially at midterm.
The questionnaire should be implemented as an online survey, facilitating as much as possible its completion by the coordinator. A good repository system should be implemented to store the data and information from projects as well as facilitate the aggregation of quantitative data and the analysis of qualitative information.
Coordinators should be aware from the moment they sign the Grant Agreement with the EC that they will be required to complete the questionnaire. They should also be informed on the benefits of collecting the information and its use by the EC. Completing the survey should me mandatory and part of the reporting requirements of the grant agreement.
Questions are arranged in different blocks corresponding each to the different categories of impact: knowledge production, research targeting and capacity building, policy and decision making and population health and health systems. Separating questions for each block of impacts allows projects to be assessed for each category separately, as well as facilitates comparative analysis for different types of projects and research .
A group of questions on Dissemination and knowledge transfer activities has also been included, as it is an important way to assess how far the project has worked towards achieving impact. In FP7 projects, Dissemination is a an important criteria for selection of proposals and it should be an important activity in all funded projects. Moreover, active engagement of main beneficiaries of the research results (end users) has been found to be an important means of increasing the impacts of research. This is even more important for health projects that are policy and practice oriented. The achievement of impacts relies greatly on an effective translation of knowledge from the project to the users of that knowledge, and so asking project coordinators on the way the project is disseminating and translating knowledge, and how far they are engaging the key end user, can serve as a proxy indicator of impact.
To facilitate and simplify as much as possible the questionnaire, coordinators are asked mostly on issues which lie on the sphere of the project and its activities, and for which the coordinator should be able to report and respond easily.
- THE END USERS' OPINION SURVEY
In order to complement and in some way, contrast, the information obtained from the researchers through the use of the coordinators' survey, we propose to seek the opinion from some of the real or potential end users of the research . We can define end users as the specific intended groups of "users" or beneficiaries of the project's results. For the area of health research, End-users are those who are affected by issues under study (e.g. community groups or populations affected by illness), or those positioned to act on the knowledge generated by research (e.g. clinicians, community leaders, health managers, patients, and policy makers).
Having some feedback on the usefulness and practical applicability of the results and findings of the project from the potential or actual key users is valuable for the non-academic impact assessment of the projects. But this has been found to be a challenging tasks as the end users have to be both identified and motivated to participate in the survey. In the IOM methodology, project coordinators are asked to provide the names of the organizations engaged as end users and this will facilitate its identification. It will be important to identify the person(s) from these organizations who have been contacted by the project and thus, are aware of the project, its goals and activities, and can give an opinion on the project impact.
The end user survey should be conducted at the end of the project directly by the EC project officer. It should be short and easy to answer, and only seeking the opinion of the person who answers the survey. He or she should not be asked to answer on behalf of the organization as this may hamper the response.
- THE PROJECT RESULTS FRAMEWORK
As part of the IOM methodology we propose that the project coordinator builds a PROJECT RESULTS FRAMEWORK during the preparation of the grant agreement with the EC and before starting the project.
A results framework structures the project planning process and helps communicate essential information about the specific project results as well as a valuable tool for the self assessment of project impacts.
As a difference with the coordinator's survey, which follows a common structure for all funded projects, the results frameworks should be project specific and thus, will be valuable for the individual assessment of projects and not for comparative purposes. It is important to mention here that projects funded by the EC under the specific call of the FP7 have been selected by an peer-review evaluation process that includes, as one of the criteria for evaluation, the expected impacts of the proposed research with reference to a set of expected impacts outlined in the specific work programme. That is, the EC expects from funded projects to achieve specific impacts that are mentioned in the description of the topics of the work programme. To be able to monitor if the projects are working towards achieving these impacts and moreover, to have some evidence that projects have been successful or not in producing the impacts listed in the topic to which the project proposal was submitted, we propose that a project results framework should be used. This will provide the EC officers with information on whether the projects are having the impacts foreseen for the different topics of the work programmes launched.
- THE SCORING MATRIX
The scoring matrix is an easy to use tool to facilitate a quick assessment of the level of impact each an individual project has achieved on:
i) Advancing Knowledge
ii) Capacity building and research targeting
iii) Policy and decision making
iv) Population health and health systems
The level of impact on each category, measured as low, good and high, is specific for the moment the scoring takes place, and can be compared with future scores of the project, for example, with the data from the coordinator's survey three years after. This can be useful to monitor the impact of the project. These measures give an idea of the impact for each category in comparison with other projects, and does not refer to the level of achievement of individual project objectives, and thus, it should not be used to evaluate if the project has performed good or bad, as this should take into account the project's initial objectives, the type of project (research projects, coordination and support action) and other variables.
The scoring matrix is valid to analyze the impact of a set of projects and valuable to detect high and low impact projects in individual groups, as the majority of projects will be located in the group of good, It is useful to see in which categories impact is mostly achieved and if certain types of projects achieve more impacts than others.
1.4 DISCUSSION
Further work should be done to test the tools with projects from the very beginning of the research cycle (the grant agreement phase), so that the operability of all the tools can be tested and refined based on real project data. Moreover, the operability of the IOM methodology should be analyzed in close collaboration with the EC to adapt it and integrate it in the actual project management and reporting system (The participant Portal).
The IOM methodology provides the tools to be able to have a quick vision of how individual projects have performed in terms of achieving expected impacts, as well as how well they can be rated for each of the pre-established impact categories. Data gathered through the questionnaire and the scoring approach developed could help to perform this task, but officers in charge of the assessment of the projects will need to spend some specific time to have a more general understanding the project's results and impacts, and will need to analyze the final reports as well as information about the initial goals of the project, specific issues that could have happened during the project, etc., because this information could be very important for a better interpretation of the results obtained from the survey.
The use of questionnaires has proven to be useful for assessing the benefits from health research but the use of case studies is recommended to know the quality and type of impacts produced and to advance in the knowledge of what research is likely to produce more impact, how is research used and what factor can be associated with high and low impact. Finally, the proposed IOM methodology should not be considered to be a fixed set of guidelines and tools. On the contrary, it should be refined and modified along its use, as information form projects is being retrieved and more understanding on the pathways to impacts are gained.
More information about the IOM methodology is available through the following documents:
• A report titled "The Impact Oriented Monitoring (IOM): A Methodology to Monitor and Assess the Impact of International Public Health Research Projects funded by the EC" describing the research work and the methodology has been prepared (Deliverable 4.8) available at the project website.
• A research article titled "Impact Oriented Monitoring: A new methodology for monitoring and evaluation of international public health research projects" describing the methodology has been submitted to the journal Research Evaluation and has been accepted for publishing.
• A policy brief titled "A pragmatic approach for monitoring the research projects results: The Impact Oriented Monitoring (IOM) methodology" is available at the project website.
2) THE ROLE OF EVIDENCE IN HEALTH POLICY AND STRATEGY
The country-specific results are available in the country reports from India and Nigeria (Deliverables 5.6 and 5.5 respectively). Therefore, further in this section we outline only the major results from the inter-country comparative analysis, drawing on a separate Deliverable 5.7 (inter-country comparative analysis).
Our results showed that there were two different understandings of evidence among policy actors in the two countries. Some policy actors regarded evidence only as factual, concrete and verifiable data used for decision making while others defined evidence more broadly as including both stakeholder experiences and systematically generated data such as published research, M&E and other reports documenting best practices or programme effectiveness.
Our analysis revealed that all health policies and strategies studied are perceived by the policy actors to be evidence-informed. Five broad types of evidence informed policy development in the study countries: i) policies and regulatory evidence; ii) routine programme and survey data; iii) research findings; iv) situation analysis commissioned for policy; and v) views and experiences of stakeholders. Our research did not identify any specific EC-funded research studies that informed health policy development, although some of the composite evidence (e.g. Lancet series) may not be attributed to a specific funder.
Health policy development in Nigeria was informed mainly by formal evidence (such as published academic articles and results of surveys) whereas formal and informal types (such as personal experiences) of evidence informed policy development in India. The reliance on formal evidence in Nigeria is likely to be due to the influence of the: i) prevailing government protocol for policy development or ii) the practice by international actors of funding generation of new evidence e.g. situational analysis for every new policy. Similarly, the more balanced use of formal and informal evidence in India could be resultant from a more prominent role of CSOs and their advocacy efforts.
No significant differences were found in the role of evidence in the development of policies in the three areas (prominent, neglected and health systems policies). This may reflect a methodological consistency applied by study teams in both countries; or it may be a genuine similarity of the role of evidence in policy development across the three different types of health policies. However, our analysis also shows that while there were no significant differences in the role of evidence between the case studies, each health policy development process was unique, reflecting the context-specificity of policy issues. In India, one policy development deliberately excluded development partners (i.e. the ASHA policy); whereas another policy was developed by the government behind closed doors (i.e. the NTCP policy). In Nigeria, the oral health policy was unique in the sense that it was approved following 5 previous failed attempts. The HRH policy was also unique in the sense that it was initiated by the Government of Nigeria following a human resources crisis in Nigeria while ASHA policy was initiated by the Government of India, following an apparent failure of the health system in India.
In terms of the use of evidence at different stages of policy and strategy development, the agenda-setting stage (i.e. when the issue is formally recognised on a policy agenda) was claimed to use wider variety of evidence types in both countries and policy approval was identified as an important milestone in policy processes, demarcating the start of policy implementation. In contrast with India, situational analysis was identified as an explicit stage of policy development in Nigeria. In both countries policymakers preferred evidence which is locally-produced and thus context-specific, of sufficient scale and representative of the whole country, rigorous and available. In India the reputation of policy actors producing evidence was particularly emphasised as a reflection of credibility of evidence whereas in Nigeria the issue of accessibility of evidence was identified as an important consideration.
The use of evidence in health policy development was subject to multiple facilitators and constraints. It was facilitated by four factors: i) the global movement on evidence-informed policy; ii) ease of access to evidence; iii) involvement of government and civil society in policy processes and iv) political will of governments who supported the processes. On the other hand, the use of evidence was constrained by: i) the subject nature of evidence, ii) inaccessibility of the language of research reports to policy makers, iii) counter-lobbying of powerful interest groups; and iv) perceived low quality of data generated for policy.
Development partners involved in the policy processes brought along their own evidence (e.g. WHO in HRH) or funded the generation of new evidence (e.g. DFID in HRH) through their involvement in health policy development. Perceptions, values and agendas of individual actors also appear to drive evidence use and policy processes. For example, seven characteristics of robust evidence were important to policy actors namely: availability, comprehensiveness, context specificity, credibility or rigour, relevance, timeliness, and of national scale. Three of these characteristics (comprehensiveness, context-specificity and of national scale) were to our knowledge distinct from the existing literature thus highlighting how policy actors may use specific criteria for their selection of robust evidence.
Our findings suggest a number of areas which can be targeted in order to enhance the utilisation of evidence in health policy development in India and Nigeria as well as other similar countries. More specifically, we propose nine policy implications for consideration by national and international policymakers, including the Commission:
1. Recognition of the value of different types of evidence (formal and informal) by the different policy actors, combined with structures for generating and using evidence (e.g. oral health research institute in Nigeria), are likely to facilitate generation of evidence for policy development;
2. Appropriate resource framework, combined with political commitment and dedicated institutions for generating and disseminating evidence is likely to ensure production of evidence and should enhance its utilisation health in policy development;
3. Strategic planning of evidence generation is required so that evidence is available when needed and routinely collected data such as HMIS is transformed into credible and useful evidence.
4. An explicit recognition of evidence generation, dissemination and use, for example, through including situational analysis as a separate stage of health policy process as was more apparent from cases studies in Nigeria, can catalyse the uptake of evidence;
5. Clear and shared guidance on the review, synthesis and appraisal of quality of different types of evidence can be a prerequisite for transparent and more effective policy development processes;
6. Greater awareness of different actors’ evidence preferences by key decision-makers is needed to raise awareness of different actors’ agendas and practices and possibly managing these practices within the policy environment.
7. Ensuring inclusive and participatory health policy development processes is likely to stimulate the availability of different evidence brought by different policy actors and potentially improve uptake of evidence in policy decisions; ultimately, this is likely to strengthen the legitimacy and acceptability of final document by the different policy actors;
8. A clear documentation of policy development clarifying what evidence was used (and what was considered and not utilised) is advised. It is likely to raise awareness of the importance of evidence in policy making and will improve the transparency and acceptance of the process by the different actors.
9. Greater awareness of diverse national and international influences on evidence role by the decision-makers is vital for avoiding the negative implications and for taking advantage of positive factors. This includes recognizing the potential drawbacks of: i) international agencies funding the generation and/or dissemination of certain types of evidence over others; and ii) national agencies such as CSOs and research institutes privileging or producing certain types of evidence for policy development.
Lastly, although we advanced knowledge on the role of evidence in health policy development in the two countries, this field is still limited and further studies are needed in different contexts. We propose three potential questions for consideration in future research:
1. What is the perceived versus actual use of evidence in health policymaking?
2. Is the use of evidence different across the technical versus more social policy areas and how does this compare in different contexts?
3. Which factors determine the use of evidence across the different policy issues in diverse contexts and what is the relative importance of such factors?.
Potential Impact:
The results coming out from the EVAL-HEALTH project can have an impact at the level of policy and decision-making and contribute to strengthen M&E of funded projects by the European Commission. Main end users of the results project will be EC officers, both at DG Research and Innovation and also at DG Devco. Secondary users can be other funding agencies that can adapt the IOM methodology for their health programmes.
The IOM methodology developed could be useful to improve the way research projects are managed at the DG Research and Innovation of the EC. For the EC, having information on the impact of projects is important for many reasons. First of all, it can support key management and strategic decisions by the DG Research & Innovation project officers. It can inform future funding programmes, allocating more funds to those areas/topics where it is perceived that desired impact are mostly gained. Good information on the impacts of individual projects is necessary for programme evaluations. Finally, demonstrating to the European Society the value of health research is very often done by showcasing success stories, that is, projects that have been successful, both in terms of scientific value and in the impact that they have had on society. The IOM methodology can support the identification of this success stories.
The IOM methodology will guide and support EC officers on how to identify and collect in a systematic way project results, including direct project outputs as well as any evidence of impacts. It will provide the tools to carry out a preliminary assessment of research impacts for on-going and finished projects, helpful to know in a rapid way the transversal impact of a portfolio of funded projects.
Moreover, the information and data obtained using the IOM methodology, could facilitate future individual and/or comparative evaluation of different Public Health research areas funded by the FP7 (i.e. International Public Health, Health promotion, Disease prevention, Health Systems, etc.) or even serve for the overall evaluation of the Public Health research programme. It can also help the identification of very successful projects or selection of cases studies for more in depth analysis of impacts.
The IOM methodology contributes to the wealth of knowledge on how to approach, in a realistic and pragmatic way, the impact assessment of research.
As for the results coming from the research work on the use of evidence for health policy making and strategic planning, the study has provided the key national and international policy actors (such as policymakers, civil society organisations, health managers, academics and international organisations) with a deeper understanding of what informed three specific health policies in India and Nigeria. It is expected that this deeper understanding will inform further revisions of the studied health policies as well as development of further health policies in the two study countries and other similar contexts.
The longer-term benefits of this study are envisaged to be two-fold. Firstly, the study informs better practices of evidence/knowledge communication from the EC-funded research and other donor-funded studies into policy and practice in developing countries. Secondly, the study will help strengthening evidence-informed nature of health policy development more generally, within the study countries and beyond.
In summary, EVAL-HEALTH project has contributed to:
• Strengthen monitoring and evaluation of EU-funded research programmes and projects, thus helping EC decision makers and development partner countries measure success of implemented activities.
• Improving monitoring and evaluation of activities, with an emphasis on measuring the effects (impacts) they have the participant partner countries
• New research on how to M&E R&D projects under International Cooperation Programmes and its value for providing means for increasing the partner country’s health system.
• Improve decision-making of EC programme managers with more efficient monitoring tools that can provide immediate feedback of performance-related information into management process.
• Enhance collaboration between European researchers and developing country researchers in monitoring and evaluation of research.
• To increase the knowledge on monitoring and evaluation methodologies specific for the health sector, with the aim to contribute to the establishment of future plans for further progressing in this field through long term collaboration between European and Cooperation partner countries researchers.
• Improve the understanding of the role of evidence in health policy development in the two countries (India and Nigeria), and at the same time, a better understanding of possible ways of improving health policy processes.
MAIN DISSEMINATION ACTIVITIES
The EVAL -HEALTH project has worked within two important areas related to improving the outcomes of health related intervention in low and middle income countries funded by the EC: What forms evidence in health policy making and how to better M&E the impact of health research projects. For this, one of the main and most important beneficiary of the results of the research is the European Commission itself, and there has been a direct and continuous contact between the project (through the coordinator) and the EC (through the project officer).
Research has been directed towards producing research results that can increasing the "pool of knowledge", but mostly, for policy and decision making. Recommendations and findings from the research on the role of evidence can improve the way evidence is used for health policy development, at least in countries were research has been performed: Nigeria and India. Policy briefs have been prepared with these results, as a means to better disseminate the results to Policy makers, civil society organisations, health staff, development partners and academics. Improved understanding on "what constitutes robust evidence" that informs health policy development is important because: First, it helps in identifying key facilitators and constraints to the use of evidence in health policymaking. Second, it can help to inform future practices in managing these influences, ultimately improving evidence-informed policymaking.
The IOM methodology developed by the research component of the EVAL-HEALTH project was designed specifically for M&E FP7 projects in the area of international Public Health, and taking into account the specificities of how DG Research manages projects, the availability of resources and the type of projects funded by the International Public health area. The major impact of the project would be on helping the EC officers in the way the monitor projects and thus, could support future decisions on the way funds are allocated, what areas are having better results and what projects can be highlighted as Success Stories. But for this, the IOM methodology needs to be implemented. Nevertheless, the IOM methodology serves as an example of how to approach the monitoring and preliminary assessment of research projects, and can be used by other areas in DG research as well as in other research funding agencies and organizations.
Two complete reports of all dissemination activities carried out have been presented as project deliverables (D7. 1 y D 7.9).
The project has not produced any results that can be commercially exploited, nor having any direct socio-economic impact in terms of patents or job creation.
SUMMARY OF EVAL-HEALTH DISSEMINATION ACTIVITIES
Based on the potential use of the research results and the stakeholders identified for the project, the EVAL-HEALTH project has carried out a variety of national and international dissemination activities, directed to different groups of stakeholders and audiences. (researchers, S&T and health policy makers, funders), as well as different geographical areas: Europe, Africa, Latin America and Asia.
AFRICAN SCIENCE TECHNOLOGY AND INNOVATION Below we present a summary of the main dissemination activities carried out by the project, in which there has been an active participation of projects partners and even engagement with stakeholders to make them aware of the project and the results.
- EVAL-HEALTH Expert Workshop. Madrid (Spain) 9-12 MAY 2011. Project workshop. Participants: All project partners, external experts, EC. Carried out at the beginning of the project, the workshop served the purpose of presenting and discussing the project approach with the European Commission and with experts from both donor and recipient countries.
- INDICATORS (ASTII) INITIATIVE WORKSHOP. Addis Ababa (Ethiopia). 23-25 MAY 2011. Participation of the project coordinator, presenting the project to African S&T Policymakers. The workshop was organized by the African Union.
- SECOND GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH (HSR) Beijing (China) , 31Oct-3 Nov 2012. Participation of partners from the U. Leeds and U. Mahidol, presenting the project results on the research on evidence (poster and oral presentations). The International conference was attended by Researchers, policy-makers, funders, implementers, civil society and media representatives.
- WORKSHOP ON FRAMEWORK CONDITIONS OF JOINT PROGRAMMING INITIATIVES: GOVERNANCE AND EVALUATION PRINCIPLES. Warsaw (Poland) 18-19 Dec 2012. Participation of the project coordinator to present the project. The workshop was organized by 7FP Project JPIs To Co Work, and was attended by EU S&T policymakers.
- PROJECT WORKSHOP: METHODOLOGIES FOR MONITORING AND EVALUATION OF RESULTS REGARDING PROJECTS AND PROGRAMMES ON RESEARCH AND DEVELOPMENT (R&D). A CASE STUDY: THE METHODOLOGY PROPOSED BY THE EVAL-HEALTH PROJECT FOR HEALTH RESEARCH" 7-8 Oct 2013, Bogotá, Colombia. Dissemination and Discussion of the IOM methodology. Together with the coordinator, partners from NEPAD, Mahidol University and Universidad de Rosario participated in the workshop, as well as experts and S&T policymakers from Latin America.
- IX IBERO-AMERICAN CONGRESS ON SCIENCE AND TECHNOLOGY INDICATORS. Bogota (Colombia) 9-11 October 2013. Dissemination of the project results (Research methodology) in an Oral Presentation. The Congress in a major event for Latin American, Spanish and Portuguese researchers active in ST indicators, as well as S&T Policymakers (ministries and funding agencies).
- FTEVAL CONFERENCE, "NEW HORIZONS / NEW CHALLENGES: EVALUATION OF STI POLICIES, INSTRUMENTS AND ORGANISATIONS”. Vienna (Austria), 12-13 Nov 2013. External workshop organized by the Austrian Platform for Research & Technology Policy Evaluation (FTEVAL platform). Presentation of IOM methodology (poster) to researchers and policy makers, mainly from European countries.
- HEALTH SYSTEMS IN ASIA: EQUITY, GOVERNANCE AND SOCIAL IMPACT. Singapore, 13-16 Dec 2013. External workshop for Health policy makers and researchers, mainly from Asian countries. Participation of U. Leeds and Mahidol University to present (oral presentation and poster) project results (research on evidence).
-“BUILDING PARTNERSHIPS FOR SUSTAINABLE CAPACITY DEVELOPMENT IN MEDICINES REGULATION IN AFRICA”, External workshop. 2-3 Dec 2013, Johannesburg, South Africa. Dissemination of the project results (Research methodology) in a poster presented by NEPAD. Audience: African Health Policy makers.
- THE ASEAN-EU SCIENCE, TECHNOLOGY AND INNOVATION DAYS 2014 (STI Days) External workshop 21-22 Jan 2014. Bangkok, Thailand. Dissemination of the project work (Stall with posters). The conference is attended by EC and ASEAN S&T policy makers and researchers.
- NEPAD AGENCY/NSTIH WORKSHOP ON STI INDICATORS AND MEASUREMENT OF IMPACTS OF HUMAN DEVELOPMENT PROGRAMMES. 1-2 Feb 2014. Praia, Cape Verde. Project workshop organized to discuss the IOM methodology with African S&T policymakers and M&E experts.
- CONTACTS WITH THE CANADIAN COALITION ON GLOBAL HEALTH RESEARCH (CCGHR). SKYPE CONFERENCE, May 2014. Presentation of the EVAL-HEALTH project and the IOM methodology to research group in charge of the background paper on Perspectives on Evaluating Global Health Research for Development, prepared for Canada’s Global Health Research Initiative in collaboration with ESSENCE.
- INTERNATIONAL CONFERENCE HEALTH SERVICES RESEARCH: EVIDENCE-BASED PRACTICE. 1st-3rd July 2014. London, UK.. Dissemination of the project results (research on evidence) by Oral presentation and Poster. Attended by the U. Leeds and Instituto de Higiene e Medicina Tropical de la U. Lisboa. The Conference audience includes International academicians, policy makers health practitioners and researchers.
- WEBINAR/PRESENTATION FOR PAHO - WHO, 2nd. July 2014. Online 1 hour seminar requested by PAHO -WHO regional office in Washington to present the IOM methodology to Assistant Director and Senior Staff from PAHO. The methodology was presented by Innovatec.
- DELHI SYMPOSIUM ON GLOBAL GOVERNANCE AND COMMERCIALISATION OF PUBLIC HEALTH New 7th -10th Sept 2014 Delhi. India. External workshop. Dissemination of the project results (research on evidence) by Oral presentation (U. Leeds) to Researchers and Civil Society Organizations.
- 3RD GLOBAL SYMPOSIUM ON HEALTH SYSTEM RESEARCH, 29 Sept-3 Oct 2014Cape Town, (S. Africa). Most important event to present and disseminate the project work and results. Eval-health organized two specific Satellite sessions, had a stall in the Marketplace throughout the conference, presented 4 posters and 1 oral presentation. All project partners participated in this event, which is attended by over 2000 people coming from all countries and major stakeholders: researchers, policy-makers, health funders, implementers, civil society and media representatives.
- PRESENTATION OF THE PROJECT RESULTS TO THE EC. Internal meeting organized on 8th. October 2014 in Brussels, Belgium, at the EC DG Research and Innovation. The coordinators together with team leader from the U. Leeds presented project results and discussed potential use by the EC. Officers from the EC DG Research and Innovation and DG. DEVCO attended the meeting.
List of Websites:
PROJECT WEB-SITE: www.eval-health.eu
Relevant contacts
Project coordinator and team leader for the IOM methodology
Dr. JOAQUIN GUINEA
INNOVATEC (Spain)
joaquin.guinea@innovatec.es
Team leader for the evidence research
Dr. Tolib Mirzoev
University of Leeds (UK)
t.mirzoev@leeds.ac.uk
EVAL-HEALTH ("Developing and Testing of New Methodologies to Monitor and Evaluate Health Related EU-Funded Interventions in Cooperation Partner Countries". www.eval-health.eu) is an international collaborative research project funded by the European Union 7th Framework Programme (FP7). The main goal of the project is to contribute to strengthen the monitoring and evaluation (M&E) of European Union funded interventions in developing countries in the specific area of international public health. The project had two research components focusing on different aspects of the M&E process:
• to identify and analyze the role of different types of evidence in health policy in low and middle-income countries, including research results.
• Development and testing of a new methodology to monitor and assess the impact of research projects that are funded by FP7 in the area of International Public Health.
EVAL-HEALTH has run for 46 month, carried out by a consortium of 9 organizations from Europe, Asia, Africa and Latin America. The work has been organized in 7 Work Packages:
WP1: Comparative analysis of M&E tools and methods across major international donors in health.
WP2: Analysis of EC Programmes and instruments to deliver Health aid to developing countries and EC policy framework on Health and development.
WP3: Methodology for analysis of the role of evidence in health policy and strategy development in developing countries.
WP4: Development of methodologies and tools for M&E of R&D projects in developing countries.
WP6: Management and project coordination
WP7: Dissemination of research results
The project has produced, as major results, the following :
• A new methodology to monitor and assess the impacts of research project, named Impact Oriented monitoring (IOM), to help EC officers in the identification, recording and assessment of projects. Results and impacts are classified into four dimensions or categories: Knowledge production, capacity building and research targeting, policy and decision making and health and health sector benefits. The methodology has been tested and validated with finished public health projects funded by the FP5 and FP6 and proposes the use of different complementary implementing tools: a project results framework, a coordinator’s survey and an end users' survey and an assessment tool. By using this methodology. The complete description of the methodology was presented as a major project deliverable (D4.8). A scientific paper describing the IOM methodology has also been accepted for publishing in the journal Research Evaluation.
• A better understanding the role of evidence in health policy and strategy development. The research has focused on identifying and analysing the role of evidence in health policy and strategy development in one Asian (India) and one African (Nigeria) country . As results of the research carried out, we have prepared two country reports, one for India and one for Nigeria (Deliverables 5.6 and 5.5 respectively) and The cross-country analysis (Deliverable 5.7). Research carried out has provided evidence to give new insight on:
1. The different types and the key characteristics of evidence that informs health policy development in India and Nigeria.
2. The use of different types of evidence in health policy development, including the facilitators and constraining factors, in India and Nigeria.
3. An analysis of the strengths and weaknesses of existing mechanisms and practices of evidence use in health policy development;
Finally , the project presents and recommends ways of enhancing the role of evidence in health policy development within the study countries and beyond.
Project Context and Objectives:
EVAL-HEALTH project had as major goal to contribute to strengthen monitoring and evaluation of EU funded interventions in developing countries in the specific area of public health. The project has focused on two specific areas: International Public health projects funded by DG Research and Innovation and the role of evidence in health policy and strategy development in developing countries. Each research component has had its own objectives and deliverables, and for the purpose of this document, we will describe each component separately. For this, we can say that the EVAL-HEALTH project has worked towards achieving the following objectives:
A. Developing and testing a methodology to monitor and assess the impact of research projects in the area of international public health.
B. Contributing to a better understanding of the role of different types of evidence, including research, in health policy and strategy development within, and across, developing countries.
A. DEVELOPING AND TESTING A METHODOLOGY TO MONITOR AND ASSESS THE IMPACT OF RESEARCH PROJECTS IN THE AREA OF INTERNATIONAL PUBLIC HEALTH.
How to better assess the impact of research projects in the area of international public health and health systems is an important issue for the European Commission, mainly as a means to know the impact that individual projects may be achieving but also as way to understand how far EC public health research programmes are contributing to address global health challenges. EVAL-HEALTH had the goal of developing and testing a methodology that could support EC officers in improving the way research projects are monitored and assessed, providing tools that can help answering questions such as:
• To which extent are EC funded health research project producing impacts?.
• Are funded projects contributing towards the expected impacts listed in the Work Programme?.
There is a growing interest and need within research funding councils and agencies, international organizations and the European Union to identify and evaluate the outputs, outcomes and impacts of funded research as a means to both understand the value of investment and to increase accountability. More and more, organizations that fund research are under increasing pressure to justify and demonstrate that research provides value to the community. A great deal has been already published regarding good practice in the evaluation of research, including a growing body of material concerned with the elaboration of frameworks and methods suitable to evaluate the impact of research, both in general as for specific areas of knowledge, such as health, agriculture and environmental research.
Research outputs such as journal articles and other scholarly publications are a tangible result of research projects, easy to identify and to measure. Bibliometric analysis has been widely used to evaluate the scientific impact of a specific research project, programme or even research group or organization, and until not long ago, measuring the scientific impact from research was enough to evaluate a research project and programme and was the main benefit expected from research.
Current economic conditions and the increasing competition for government funding are leading to an increased focus on the impact of research outside academic environments, as a need to demonstrate that investment are effective both to improve the operation of research programmes and also to demonstrate to society that research funds are well spent. At the same time, there has been a growing willingness and enthusiasm from academia to explore the economic and societal value of research in a wide variety of forms.
The benefits and/or effects that research has beyond the creation of scientific knowledge, which may be labelled as academic impacts, can be grouped under the name of societal/social impacts or non-academic impacts. It includes impacts such as capacity building, facilitating evidence-based decision and policy making and benefits for the health system and the health of the population.
The research component of the EVAL-HEALTH project had the objective of developing and testing specific methodological models and tools to help project officers at the EC DG Research and Innovation (Health Unit), monitor and evaluate (M&E) the impact of EU funded research actions in the specific area of International Public health. The methodology had to serve the purpose of M&E Individual projects (research projects or coordinating and support actions) funded as ‘Specific International Cooperation actions’ (SICAs), which is the instrument most widely used by the EC to enable and encourage low and middle income countries to participate and get funding from the FP7. These type of projects require the participation of partners from outside the EU, mainly from low and middle income countries. They typically involve 8-10 participants, for a period of 3 years, and an overall funding of about € 2-3 million from the EC. SICAs are the continuation of the INCO actions of FP5 and FP6.
The majority of projects funded are thus international consortiums formed by a variety of organizations coming from different cultural backgrounds. This type of projects are expected to provide the means for addressing public health issues of importance in international cooperation partner countries and aim to reinforce the research and cooperative capacity of candidate, neighbourhood, developing and emerging countries.
EVAL-HEALTH will develop a methodology to better understand and measure a project 's success and help identify the impacts, including those benefits on the public health systems of partner countries participating in the projects. The methodology links with existing FPs reporting system by adding new tools for the collection of data and information, and also links with the thematic level evaluations, as it will provide important information from individual projects useful to perform these type of evaluations.
By using the methodology, EC officers should be able to:
i) monitor how the project is performing, identifying results and evidence to anticipate project impacts.
ii) assess the final results and provide an estimation of the type of impacts of individual projects
iii) help future programme evaluations.
Research work was carried out following a work plan that included the following activities
• Identify best practices in M&E of research actions from major international agencies and donors active in health research (WP1).
• Review main frameworks available for evaluating the impact of health research (WP1)
• Review and analyse frameworks, methodologies and tools currently used by the EC to M&E research projects and programmes (WP2)
• Develop specific methodological models and tools for M&E the impact of research projects (WP4).
• Field testing the suitability and applicability of the developed methodology through case studies (WP5)
• Disseminate the project results (WP7)
The methodology developed has been named as Impact Oriented Monitoring (IOM), as it tries to anticipate as much as possible the impact of projects during the monitoring process.
B. ANALYSIS OF THE ROLE OF EVIDENCE, INCLUDING RESEARCH, IN HEALTH POLICY AND STRATEGY DEVELOPMENT WITHIN, AND ACROSS, DEVELOPING COUNTRIES.
There is a growing recognition of the importance of evidence-informed health policy and planning decisions. Different types of evidence are referred to in the literature such as formal (e.g. Published research, results of monitoring and evaluation, HMIS statistics) and informal (e.g. personal experience, rumours, media reports). The above distinction is based on recognition that policies and plans are based on "something" whether it is clear-cut research findings or just the expectations that "this particular programme/approach will work". Frameworks exist in the literature to help understand, and enhance, the role of evidence in health policies and strategies. However, there is limited use of these frameworks in analysing the role of different types of evidence in health policy development in developing countries.
The Evidence Component of the EVAL-HEALTH project aimed to contribute to filling the above gap in the knowledge through identifying and analysing the role of different types of evidence in health policy and strategy development in one Asian (India) and one African (Nigeria) country. The specific research objectives were:
1. To identify different types and the key characteristics of evidence that informed health policy development.
2. To determine and analyse, within and across the case studies and the study countries, the use of different types of evidence in health policy development, including the facilitators and constraining factors;
3. To analyse the strengths and weaknesses of existing mechanisms and practices of evidence use in health policy development;
4. To recommend ways of enhancing the role of evidence in health policy development
within the study countries and beyond;
The research has been conducted in two countries: India and Nigeria, using a case study approach. For the purpose of this project, we defined ‘case study’ as an issue - for which a policy has been formulated - representing health services (HIV/AIDS, Maternal and Child Health, Non-Communicable Disease), health-related issue (Tobacco Control) or a health system’s component (Human Resources). In each country, three specific policies were selected corresponding to the three case study areas:
The rationale for selecting more than one country was that variation in contexts between countries would facilitate better understanding of the determinants of successful policymaking. Rather than enumerating just what worked, a comparative analysis of six policies helped to identify mechanisms of effective policy-making across cases and contexts.
In each of the two countries three specific policies were selected to examine the role of evidence in diverse policy areas and, to generate and disseminate lessons from the intra and inter-country comparative analysis. In India, the policies were: National AIDS Control Programme (NACP III); ii) National Tobacco Control Programme 2007 (NTCP); and the Accredited Social Health Activists programme of 2005 (ASHA). In Nigeria, the policies were: Integrated Maternal, Neonatal and Child Health strategy 2007 (IMNCH); Human Resources for
Health Policy 2006 (HRH) and Oral Health Policy 2012 (OHP).
The rationale for the choice of the above three case studies was two-fold: firstly, we wanted to avoid focusing only on areas of international prominence and, secondly, comparing findings between case studies and between countries, could allow generating and disseminating lessons learned.
Two qualitative research methods were used: document reviews and in-depth interviews. Key policy documents were reviewed and the views of policymakers, civil society organizations, health staff, development partners and academics were gathered using in-depth interviews.
The study was guided by a conceptual framework, drawn from the literature. A complete description of the conceptual framework that guided the work is presented in the project deliverable D3.1: Research methodology to analyse the role of different types of evidence (including EC-funded research) in health development within developing countries. Our conceptual interpretation of evidence was not confined to research only but included both formal (e.g. published research, policy evaluation reports, HMIS data analyses) and informal (media reports, personal experiences) types.
The role of evidence in health policies is perceived as interplay between evidence processes and policy processes. The policy actors determine this interplay through their roles in evidence and policy processes, their agendas and practices. All of above occurs within the wider context, which includes different national and international influences. Each component of the framework is complex and in-depth understanding of each component was required as part of analysis of the role of evidence in health policy development.
Framework approach was used to analyse data. In each country the role of evidence was explored in three specific policies: an area of international prominence (HIV/AIDS in India and maternal, neonatal and child health in Nigeria), a neglected area (tobacco control in India and oral health in Nigeria), and a health systems component (social health activists programme in India and human resource policy in Nigeria). The cross-country analysis (Deliverable 5.7) was performed using the country reports from India and Nigeria (Deliverables 5.6 and 5.5 respectively).
Project Results:
1) THE IMPACT ORIENTED METHODOLOGY (IOM) METHODOLOGY
The EVAL-HEALTH project has designed a methodology to monitor and assess the impacts of public health research projects funded by the European Commission through its 7th. Framework Programme. This Methodology has been named as IMPACT ORIENTED MONITORING (IOM). It has been designed with the objective of helping the EC DG Research & Innovation officers to know what the impacts of funded projects is. We have worked specifically for projects funded in the area of International Public Health, but the methodology could be used for all Public health projects, and with some minor changes, to the whole health research area.
Before describing the methodology, it is important to define some M&E terms that will be used throughout the document and that are important to better understand the IOM methodology.
RESULTS: The output, outcome and/or impact (intended or unintended, positive and/or negative) of a project.
REPORTING: Periodic submission of project documents during its development, at the end or after its completion (in the FP7 funded projects the EC reporting requirements are indicated in the Grant Agreement).
IMPACT MONITORING: A periodic identification and assessment of the impacts of a project
EX-POST EVALUATION An in-depth assessment of how a completed project has contributed to its overall expected results.
INPUTS: Resources that feed into the project.
ACTIVITIES: Actions that describe how the inputs are used to carry out the project (WPs activities).
OUTPUTS: Direct “products” of the project activities (Deliverables).
IMPACTS: Effects (changes) produced by the project activities and output. We can differentiate between:
- Immediate impacts. Those likely to occur within the project cycle.
- Short term impacts (outcomes). Impacts likely to occur within 3 years following project completion
- Long term impacts: Impacts likely to occur between 3 -10 years after project completion.
(Note. These definitions have been taken from the document: The European Evaluation Consortium . (2005). Analysis of ‘high impact’ research activities under Community Research Framework Programmes. DG RTD Framework Contract No: BUDG-02-01 L2)
IMPACT CATEGORIES (Health Research): Broad classification in different domains, where the impacts of health research projects are produced (capacity-building, knowledge production, informing policy, health, etc).
The IOM methodology can be used to identify and assess the impacts of funded projects on International Public Health research, with specific focus on:
i. Scientific outcomes and impacts: training of researchers, quality of research, constitution of international networks, etc.
ii. Impacts of the project results on the Public Health policy of the developing country and
iii. Impacts on the health of the population and other broad economic and social impacts.
1.1 CONCEPTUAL MODEL FOR THE IOM METHODOLOGY
The unit of analysis for our methodology is individual research projects.
The conceptual model designed to understand the process from project to impact in the context where the IOM methodology will be applied (The EC Framework programme) has been built upon the strengths of the Buxton-Hanney Payback model.
The IOM conceptual model uses a logic model approach to describe the process from research input to project outputs and impacts. The model illustrates the theoretical pathways underlying the research cycle, and how the different results are translated to immediate, short-term and long term impacts. The model identifies what in theory can be expected as the main outputs of the projects, the direct results (scientific publications, patents, better trained researchers, educational programmes..). These results can be measured and can be easily attributed to the funded project. The use of these results by different stakeholders can have an immediate /short term impact, a benefit of the project. If the research results are used by the scientific community then the project will have contributed to advancing knowledge, if the results or findings are used for informing policy decisions or for improving health practice, then the project will have had an impact on policy and decision making. These type of impacts could be regarded as immediate impacts, which result from the direct use of the project outputs. Further on in the model, policy decisions based on research evidence can lead to a better health policy, which in turn may have as a consequence an improvement in the population health, better treatments and use of medical resource and improved equity. Improving health practice will also have a benefit in the health of the population. All these effects, difficult to attribute exclusively to one research project and also difficult to identify and measure, normally take place long after the research project has finished and can be categorized as impacts or benefits on the health system. Finally, long term impacts expected from health research are improvements in health and well being of the population (i.e. achieving the MDGs) and economic and social prosperity. These long term impacts are the final goal of any (health) Research and Innovation programme, but can never be attributed to individual projects, but more to RTD programmes over a long period of time.
1.2 IMPACT CATEGORIES FOR THE IOM METHODOLOGY
The IOM methodology uses the impact categories for health research proposed by the Payback model to guide the operability of the conceptual model and its implementation. A short description of the impact categories and how we expect to gather evidence for each of them is provided below.
1) ADVANCING KNOWLEDGE
Producing knowledge is usually the first output from any research, and codifying this knowledge in a scientific paper, report or book chapter is vital for it to have some impact and thus contribute to advancing scientific knowledge. Research results are traditionally published in peer-review journals, but can also take the form of patent applications, public databases, research methods or even policy briefs.
In order to assess the impact on knowledge production gained by the projects, it is important to gather evidence on research outputs that could benefit the research community by entering the "pool of knowledge", mainly peer review articles published in indexed journals and if possible, try to have some indicators on the scientific impact, such as citations or other bibliometric analysis. Other indicators used to follow up the impact on knowledge are: other publications such as books, book chapters, conference proceeding; research reports, policy briefs and other "grey literature" produced and Systematic reviews.
We have included also in this category patents, as these documents also contribute to the pool of knowledge, and other research outputs such as databases or software that could also serve to advance scientific knowledge.
1) RESEARCH TARGETING AND CAPACITY BUILDING
Research targeting refers to the way the research project can contribute to a better targeting of future research, carried out both by the projects researchers as by others. In the first case, it can be tracked by gathering evidence on the existence of follow-up research projects gained by some members of the research team, research collaborations between project participants or with others. Any evidence that the results or findings from the research may have served to set up new research topics in future programmes will be valuable, but this type of evidence is difficult to follow.
Capacity building refers to both the gain of skills to undertake future research inside the project team, as well as an increase of the research capacity at the organization level (i.e. new research infrastructure), that is, how research can increase or improve the research capacity of individuals, teams and organizations. This benefit from the research project is quite straightforward to understand but should not only be based on counting the number of researchers and trainees participating in the project. It is important to also measure the number and level of higher or research degrees resulting, either totally or in part, from the research funding (how many trainees have gained a PhD, how many Post docs have acquired the category of junior researchers, etc.). Any evidence that can be captured of career development is valuable to assess the impact on capacity building. This impact is especially relevant in projects such as those funded in the Area of International Public Health, which have partners coming from low and middle income countries who can benefit with an increase of their research skills as results of their participation in the project.
Training that may take place outside the consortium as results of the project should also be considered when assessing the impact on capacity building, as many projects in the Area of International Public health have, as main goal, health training and capacity building of target groups in participating countries.
3) INFORMING DECISION MAKING, PRACTICE AND POLICY
The utilization of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. Health research can be used to inform policy and decision making in a wide range of circumstances and at many levels: Governmental national and local policies, managers within health services, health care practitioners (i.e. clinical guidelines), professional bodies, education and training, etc. It can make a contribution in at least three phases of the policy-making process: agenda-setting, policy formulation and implementation.
It is important for any research project to identify and assess impacts on policy and practice, especially in clinical, public health or health service research. These type of impacts or benefits are normally considered as a "secondary output" of the research process to distinguish them from the "primary or direct outputs" of research such as journal articles, other publications and trained researchers.
The most direct evidence for this type of impact would be direct citation of the research in a policy document, testimony of policy and decision makers or any evidence of implementation of influence, which are most of the times difficult to obtain. Evidence can be more easily collected by identifying indicators on the process for informing policy (i.e. participation of project researchers in Policy Advisory committees) or even in the way engagement with relevant policy makers has taken place during the project life (how and when), as these are considered factors of success for translating research into policy and practice.
4) POPULATION HEALTH AND HEALTH SECTOR BENEFITS
Pursuing health benefits for target population are the major final goals of any public funded research, and being able to provide evidence that this goal has been achieved is a major interest for all funding agencies to account for public funds spending.
Real evidence for this category of impact is very difficult to identify, and even more to measure and assess, especially at the level of individual research projects (as is the case for the IOM methodology). The attribution challenge that arises when trying to assess research impact is, for this type of impact, very relevant. Gains in health are normally the consequences of many research works along many years, together with many other variables (economic growth, environmental conditions, education) and circumstances (Political situation).
Nevertheless, it is possible to gain some insight on the contribution that the project may have to potential gains in population health and health sector benefits. Collecting qualitative information on how the project may contribute to issues such as improvement in clinical treatments, improvements in health service delivery and reducing the cost of health delivery can provide some insight on the project's contribution to impact. Moreover, for the specific projects funded in the area of International Public Health, it is important to know if the project may contribute to specific issues such as strengthening health care systems, better understanding and improvements in maternal and child care and reproductive health, increase knowledge and treatment of communicable and non communicable diseases, etc. We can note here that these health impacts are normally the final goals that the research funder sets (in our case the EC) when launching a specific research programme such as International Public Health research.
THE IMPORTANCE OF DISSEMINATION AND KNOWLEDGE TRANSFER ACTIVITIES FOR INCREASING THE IMPACT OF RESEARCH.
The nature and degree of impact any research may have, relies greatly on the effort put forward by the research project towards disseminating the research results and findings and actively engaging with key project stakeholders and end users. This is even more important for any research that produces evidence for policy or decision making, as the impact greatly depends on an active uptake of research findings by those who have to take decisions or implement specific health policies. Results need to be up taken and used at the level of policy and decision making to have a positive impact further on, for example for strengthening health systems and improving health.
Through the cases studies work carried out by the EVAL-HEALTH project to develop the methodology, we were able to confirm that successful projects had thoroughly worked towards actively engaging with all important stakeholders, such as Ministry of Health, local authorities or health practitioners in countries where the research could be used to improve the country's health system.
For the IOM methodology, we consider that dissemination (understood in a broad sense and including both active and passive dissemination actions) is an important factor for a project to have a wider impact. For this, we have included in the coordinators' survey and in the scoring matrix, a specific section on Dissemination to help in the assessment of project impacts.
1.3 IMPLEMENTATION TOOLS OF THE IOM METHODOLOGY.
Establishing a good system for capturing and recording appropriate project data during and after the life of the projects, can provide sufficient information to identify and assess immediate and short term impacts. The IOM methodology provides the guidance and tools to set up an Impact Monitoring System that can be used to gain information on the impacts of individual projects.
During the life cycle of the project and shortly after, impact can be monitored by collecting data and information on:
• Direct project results
• Potential and/or direct use of project results and findings by key stakeholders
• Dissemination and knowledge translation activities, (all those activities that facilitate, foster or are necessary for research to have an impact ).
These principles have been the basis to design the three specific data collection tools and the assessment tool used in the Impact Monitoring System. Each of these tools provide valuable information to help assess the impact of the projects at different stages and with a different level of understanding, and together they conform the IOM methodology. A short description of each tool is provided below:
- THE COORDINATORS' SURVEY
It is the main data collection tool for capturing project results and evidence of the impacts of the research. It is implemented as a web-based questionnaire, that should be completed by the project coordinator at the end of the projects and 3 years after the project has finished. For projects lasting 4 or more years, a questionnaire could also be used for midterm data collection. It can be used for project management (monitoring), project assessment and programme evaluation.
To facilitate the completion of the questionnaire and reduce the administrative work load for the coordinators, completion of the questionnaire should be linked, as much as possible, to reporting periods required by the Gran Agreement. We suggest the possibility of producing only one technical report, based mainly on the questionnaire, especially at midterm.
The questionnaire should be implemented as an online survey, facilitating as much as possible its completion by the coordinator. A good repository system should be implemented to store the data and information from projects as well as facilitate the aggregation of quantitative data and the analysis of qualitative information.
Coordinators should be aware from the moment they sign the Grant Agreement with the EC that they will be required to complete the questionnaire. They should also be informed on the benefits of collecting the information and its use by the EC. Completing the survey should me mandatory and part of the reporting requirements of the grant agreement.
Questions are arranged in different blocks corresponding each to the different categories of impact: knowledge production, research targeting and capacity building, policy and decision making and population health and health systems. Separating questions for each block of impacts allows projects to be assessed for each category separately, as well as facilitates comparative analysis for different types of projects and research .
A group of questions on Dissemination and knowledge transfer activities has also been included, as it is an important way to assess how far the project has worked towards achieving impact. In FP7 projects, Dissemination is a an important criteria for selection of proposals and it should be an important activity in all funded projects. Moreover, active engagement of main beneficiaries of the research results (end users) has been found to be an important means of increasing the impacts of research. This is even more important for health projects that are policy and practice oriented. The achievement of impacts relies greatly on an effective translation of knowledge from the project to the users of that knowledge, and so asking project coordinators on the way the project is disseminating and translating knowledge, and how far they are engaging the key end user, can serve as a proxy indicator of impact.
To facilitate and simplify as much as possible the questionnaire, coordinators are asked mostly on issues which lie on the sphere of the project and its activities, and for which the coordinator should be able to report and respond easily.
- THE END USERS' OPINION SURVEY
In order to complement and in some way, contrast, the information obtained from the researchers through the use of the coordinators' survey, we propose to seek the opinion from some of the real or potential end users of the research . We can define end users as the specific intended groups of "users" or beneficiaries of the project's results. For the area of health research, End-users are those who are affected by issues under study (e.g. community groups or populations affected by illness), or those positioned to act on the knowledge generated by research (e.g. clinicians, community leaders, health managers, patients, and policy makers).
Having some feedback on the usefulness and practical applicability of the results and findings of the project from the potential or actual key users is valuable for the non-academic impact assessment of the projects. But this has been found to be a challenging tasks as the end users have to be both identified and motivated to participate in the survey. In the IOM methodology, project coordinators are asked to provide the names of the organizations engaged as end users and this will facilitate its identification. It will be important to identify the person(s) from these organizations who have been contacted by the project and thus, are aware of the project, its goals and activities, and can give an opinion on the project impact.
The end user survey should be conducted at the end of the project directly by the EC project officer. It should be short and easy to answer, and only seeking the opinion of the person who answers the survey. He or she should not be asked to answer on behalf of the organization as this may hamper the response.
- THE PROJECT RESULTS FRAMEWORK
As part of the IOM methodology we propose that the project coordinator builds a PROJECT RESULTS FRAMEWORK during the preparation of the grant agreement with the EC and before starting the project.
A results framework structures the project planning process and helps communicate essential information about the specific project results as well as a valuable tool for the self assessment of project impacts.
As a difference with the coordinator's survey, which follows a common structure for all funded projects, the results frameworks should be project specific and thus, will be valuable for the individual assessment of projects and not for comparative purposes. It is important to mention here that projects funded by the EC under the specific call of the FP7 have been selected by an peer-review evaluation process that includes, as one of the criteria for evaluation, the expected impacts of the proposed research with reference to a set of expected impacts outlined in the specific work programme. That is, the EC expects from funded projects to achieve specific impacts that are mentioned in the description of the topics of the work programme. To be able to monitor if the projects are working towards achieving these impacts and moreover, to have some evidence that projects have been successful or not in producing the impacts listed in the topic to which the project proposal was submitted, we propose that a project results framework should be used. This will provide the EC officers with information on whether the projects are having the impacts foreseen for the different topics of the work programmes launched.
- THE SCORING MATRIX
The scoring matrix is an easy to use tool to facilitate a quick assessment of the level of impact each an individual project has achieved on:
i) Advancing Knowledge
ii) Capacity building and research targeting
iii) Policy and decision making
iv) Population health and health systems
The level of impact on each category, measured as low, good and high, is specific for the moment the scoring takes place, and can be compared with future scores of the project, for example, with the data from the coordinator's survey three years after. This can be useful to monitor the impact of the project. These measures give an idea of the impact for each category in comparison with other projects, and does not refer to the level of achievement of individual project objectives, and thus, it should not be used to evaluate if the project has performed good or bad, as this should take into account the project's initial objectives, the type of project (research projects, coordination and support action) and other variables.
The scoring matrix is valid to analyze the impact of a set of projects and valuable to detect high and low impact projects in individual groups, as the majority of projects will be located in the group of good, It is useful to see in which categories impact is mostly achieved and if certain types of projects achieve more impacts than others.
1.4 DISCUSSION
Further work should be done to test the tools with projects from the very beginning of the research cycle (the grant agreement phase), so that the operability of all the tools can be tested and refined based on real project data. Moreover, the operability of the IOM methodology should be analyzed in close collaboration with the EC to adapt it and integrate it in the actual project management and reporting system (The participant Portal).
The IOM methodology provides the tools to be able to have a quick vision of how individual projects have performed in terms of achieving expected impacts, as well as how well they can be rated for each of the pre-established impact categories. Data gathered through the questionnaire and the scoring approach developed could help to perform this task, but officers in charge of the assessment of the projects will need to spend some specific time to have a more general understanding the project's results and impacts, and will need to analyze the final reports as well as information about the initial goals of the project, specific issues that could have happened during the project, etc., because this information could be very important for a better interpretation of the results obtained from the survey.
The use of questionnaires has proven to be useful for assessing the benefits from health research but the use of case studies is recommended to know the quality and type of impacts produced and to advance in the knowledge of what research is likely to produce more impact, how is research used and what factor can be associated with high and low impact. Finally, the proposed IOM methodology should not be considered to be a fixed set of guidelines and tools. On the contrary, it should be refined and modified along its use, as information form projects is being retrieved and more understanding on the pathways to impacts are gained.
More information about the IOM methodology is available through the following documents:
• A report titled "The Impact Oriented Monitoring (IOM): A Methodology to Monitor and Assess the Impact of International Public Health Research Projects funded by the EC" describing the research work and the methodology has been prepared (Deliverable 4.8) available at the project website.
• A research article titled "Impact Oriented Monitoring: A new methodology for monitoring and evaluation of international public health research projects" describing the methodology has been submitted to the journal Research Evaluation and has been accepted for publishing.
• A policy brief titled "A pragmatic approach for monitoring the research projects results: The Impact Oriented Monitoring (IOM) methodology" is available at the project website.
2) THE ROLE OF EVIDENCE IN HEALTH POLICY AND STRATEGY
The country-specific results are available in the country reports from India and Nigeria (Deliverables 5.6 and 5.5 respectively). Therefore, further in this section we outline only the major results from the inter-country comparative analysis, drawing on a separate Deliverable 5.7 (inter-country comparative analysis).
Our results showed that there were two different understandings of evidence among policy actors in the two countries. Some policy actors regarded evidence only as factual, concrete and verifiable data used for decision making while others defined evidence more broadly as including both stakeholder experiences and systematically generated data such as published research, M&E and other reports documenting best practices or programme effectiveness.
Our analysis revealed that all health policies and strategies studied are perceived by the policy actors to be evidence-informed. Five broad types of evidence informed policy development in the study countries: i) policies and regulatory evidence; ii) routine programme and survey data; iii) research findings; iv) situation analysis commissioned for policy; and v) views and experiences of stakeholders. Our research did not identify any specific EC-funded research studies that informed health policy development, although some of the composite evidence (e.g. Lancet series) may not be attributed to a specific funder.
Health policy development in Nigeria was informed mainly by formal evidence (such as published academic articles and results of surveys) whereas formal and informal types (such as personal experiences) of evidence informed policy development in India. The reliance on formal evidence in Nigeria is likely to be due to the influence of the: i) prevailing government protocol for policy development or ii) the practice by international actors of funding generation of new evidence e.g. situational analysis for every new policy. Similarly, the more balanced use of formal and informal evidence in India could be resultant from a more prominent role of CSOs and their advocacy efforts.
No significant differences were found in the role of evidence in the development of policies in the three areas (prominent, neglected and health systems policies). This may reflect a methodological consistency applied by study teams in both countries; or it may be a genuine similarity of the role of evidence in policy development across the three different types of health policies. However, our analysis also shows that while there were no significant differences in the role of evidence between the case studies, each health policy development process was unique, reflecting the context-specificity of policy issues. In India, one policy development deliberately excluded development partners (i.e. the ASHA policy); whereas another policy was developed by the government behind closed doors (i.e. the NTCP policy). In Nigeria, the oral health policy was unique in the sense that it was approved following 5 previous failed attempts. The HRH policy was also unique in the sense that it was initiated by the Government of Nigeria following a human resources crisis in Nigeria while ASHA policy was initiated by the Government of India, following an apparent failure of the health system in India.
In terms of the use of evidence at different stages of policy and strategy development, the agenda-setting stage (i.e. when the issue is formally recognised on a policy agenda) was claimed to use wider variety of evidence types in both countries and policy approval was identified as an important milestone in policy processes, demarcating the start of policy implementation. In contrast with India, situational analysis was identified as an explicit stage of policy development in Nigeria. In both countries policymakers preferred evidence which is locally-produced and thus context-specific, of sufficient scale and representative of the whole country, rigorous and available. In India the reputation of policy actors producing evidence was particularly emphasised as a reflection of credibility of evidence whereas in Nigeria the issue of accessibility of evidence was identified as an important consideration.
The use of evidence in health policy development was subject to multiple facilitators and constraints. It was facilitated by four factors: i) the global movement on evidence-informed policy; ii) ease of access to evidence; iii) involvement of government and civil society in policy processes and iv) political will of governments who supported the processes. On the other hand, the use of evidence was constrained by: i) the subject nature of evidence, ii) inaccessibility of the language of research reports to policy makers, iii) counter-lobbying of powerful interest groups; and iv) perceived low quality of data generated for policy.
Development partners involved in the policy processes brought along their own evidence (e.g. WHO in HRH) or funded the generation of new evidence (e.g. DFID in HRH) through their involvement in health policy development. Perceptions, values and agendas of individual actors also appear to drive evidence use and policy processes. For example, seven characteristics of robust evidence were important to policy actors namely: availability, comprehensiveness, context specificity, credibility or rigour, relevance, timeliness, and of national scale. Three of these characteristics (comprehensiveness, context-specificity and of national scale) were to our knowledge distinct from the existing literature thus highlighting how policy actors may use specific criteria for their selection of robust evidence.
Our findings suggest a number of areas which can be targeted in order to enhance the utilisation of evidence in health policy development in India and Nigeria as well as other similar countries. More specifically, we propose nine policy implications for consideration by national and international policymakers, including the Commission:
1. Recognition of the value of different types of evidence (formal and informal) by the different policy actors, combined with structures for generating and using evidence (e.g. oral health research institute in Nigeria), are likely to facilitate generation of evidence for policy development;
2. Appropriate resource framework, combined with political commitment and dedicated institutions for generating and disseminating evidence is likely to ensure production of evidence and should enhance its utilisation health in policy development;
3. Strategic planning of evidence generation is required so that evidence is available when needed and routinely collected data such as HMIS is transformed into credible and useful evidence.
4. An explicit recognition of evidence generation, dissemination and use, for example, through including situational analysis as a separate stage of health policy process as was more apparent from cases studies in Nigeria, can catalyse the uptake of evidence;
5. Clear and shared guidance on the review, synthesis and appraisal of quality of different types of evidence can be a prerequisite for transparent and more effective policy development processes;
6. Greater awareness of different actors’ evidence preferences by key decision-makers is needed to raise awareness of different actors’ agendas and practices and possibly managing these practices within the policy environment.
7. Ensuring inclusive and participatory health policy development processes is likely to stimulate the availability of different evidence brought by different policy actors and potentially improve uptake of evidence in policy decisions; ultimately, this is likely to strengthen the legitimacy and acceptability of final document by the different policy actors;
8. A clear documentation of policy development clarifying what evidence was used (and what was considered and not utilised) is advised. It is likely to raise awareness of the importance of evidence in policy making and will improve the transparency and acceptance of the process by the different actors.
9. Greater awareness of diverse national and international influences on evidence role by the decision-makers is vital for avoiding the negative implications and for taking advantage of positive factors. This includes recognizing the potential drawbacks of: i) international agencies funding the generation and/or dissemination of certain types of evidence over others; and ii) national agencies such as CSOs and research institutes privileging or producing certain types of evidence for policy development.
Lastly, although we advanced knowledge on the role of evidence in health policy development in the two countries, this field is still limited and further studies are needed in different contexts. We propose three potential questions for consideration in future research:
1. What is the perceived versus actual use of evidence in health policymaking?
2. Is the use of evidence different across the technical versus more social policy areas and how does this compare in different contexts?
3. Which factors determine the use of evidence across the different policy issues in diverse contexts and what is the relative importance of such factors?.
Potential Impact:
The results coming out from the EVAL-HEALTH project can have an impact at the level of policy and decision-making and contribute to strengthen M&E of funded projects by the European Commission. Main end users of the results project will be EC officers, both at DG Research and Innovation and also at DG Devco. Secondary users can be other funding agencies that can adapt the IOM methodology for their health programmes.
The IOM methodology developed could be useful to improve the way research projects are managed at the DG Research and Innovation of the EC. For the EC, having information on the impact of projects is important for many reasons. First of all, it can support key management and strategic decisions by the DG Research & Innovation project officers. It can inform future funding programmes, allocating more funds to those areas/topics where it is perceived that desired impact are mostly gained. Good information on the impacts of individual projects is necessary for programme evaluations. Finally, demonstrating to the European Society the value of health research is very often done by showcasing success stories, that is, projects that have been successful, both in terms of scientific value and in the impact that they have had on society. The IOM methodology can support the identification of this success stories.
The IOM methodology will guide and support EC officers on how to identify and collect in a systematic way project results, including direct project outputs as well as any evidence of impacts. It will provide the tools to carry out a preliminary assessment of research impacts for on-going and finished projects, helpful to know in a rapid way the transversal impact of a portfolio of funded projects.
Moreover, the information and data obtained using the IOM methodology, could facilitate future individual and/or comparative evaluation of different Public Health research areas funded by the FP7 (i.e. International Public Health, Health promotion, Disease prevention, Health Systems, etc.) or even serve for the overall evaluation of the Public Health research programme. It can also help the identification of very successful projects or selection of cases studies for more in depth analysis of impacts.
The IOM methodology contributes to the wealth of knowledge on how to approach, in a realistic and pragmatic way, the impact assessment of research.
As for the results coming from the research work on the use of evidence for health policy making and strategic planning, the study has provided the key national and international policy actors (such as policymakers, civil society organisations, health managers, academics and international organisations) with a deeper understanding of what informed three specific health policies in India and Nigeria. It is expected that this deeper understanding will inform further revisions of the studied health policies as well as development of further health policies in the two study countries and other similar contexts.
The longer-term benefits of this study are envisaged to be two-fold. Firstly, the study informs better practices of evidence/knowledge communication from the EC-funded research and other donor-funded studies into policy and practice in developing countries. Secondly, the study will help strengthening evidence-informed nature of health policy development more generally, within the study countries and beyond.
In summary, EVAL-HEALTH project has contributed to:
• Strengthen monitoring and evaluation of EU-funded research programmes and projects, thus helping EC decision makers and development partner countries measure success of implemented activities.
• Improving monitoring and evaluation of activities, with an emphasis on measuring the effects (impacts) they have the participant partner countries
• New research on how to M&E R&D projects under International Cooperation Programmes and its value for providing means for increasing the partner country’s health system.
• Improve decision-making of EC programme managers with more efficient monitoring tools that can provide immediate feedback of performance-related information into management process.
• Enhance collaboration between European researchers and developing country researchers in monitoring and evaluation of research.
• To increase the knowledge on monitoring and evaluation methodologies specific for the health sector, with the aim to contribute to the establishment of future plans for further progressing in this field through long term collaboration between European and Cooperation partner countries researchers.
• Improve the understanding of the role of evidence in health policy development in the two countries (India and Nigeria), and at the same time, a better understanding of possible ways of improving health policy processes.
MAIN DISSEMINATION ACTIVITIES
The EVAL -HEALTH project has worked within two important areas related to improving the outcomes of health related intervention in low and middle income countries funded by the EC: What forms evidence in health policy making and how to better M&E the impact of health research projects. For this, one of the main and most important beneficiary of the results of the research is the European Commission itself, and there has been a direct and continuous contact between the project (through the coordinator) and the EC (through the project officer).
Research has been directed towards producing research results that can increasing the "pool of knowledge", but mostly, for policy and decision making. Recommendations and findings from the research on the role of evidence can improve the way evidence is used for health policy development, at least in countries were research has been performed: Nigeria and India. Policy briefs have been prepared with these results, as a means to better disseminate the results to Policy makers, civil society organisations, health staff, development partners and academics. Improved understanding on "what constitutes robust evidence" that informs health policy development is important because: First, it helps in identifying key facilitators and constraints to the use of evidence in health policymaking. Second, it can help to inform future practices in managing these influences, ultimately improving evidence-informed policymaking.
The IOM methodology developed by the research component of the EVAL-HEALTH project was designed specifically for M&E FP7 projects in the area of international Public Health, and taking into account the specificities of how DG Research manages projects, the availability of resources and the type of projects funded by the International Public health area. The major impact of the project would be on helping the EC officers in the way the monitor projects and thus, could support future decisions on the way funds are allocated, what areas are having better results and what projects can be highlighted as Success Stories. But for this, the IOM methodology needs to be implemented. Nevertheless, the IOM methodology serves as an example of how to approach the monitoring and preliminary assessment of research projects, and can be used by other areas in DG research as well as in other research funding agencies and organizations.
Two complete reports of all dissemination activities carried out have been presented as project deliverables (D7. 1 y D 7.9).
The project has not produced any results that can be commercially exploited, nor having any direct socio-economic impact in terms of patents or job creation.
SUMMARY OF EVAL-HEALTH DISSEMINATION ACTIVITIES
Based on the potential use of the research results and the stakeholders identified for the project, the EVAL-HEALTH project has carried out a variety of national and international dissemination activities, directed to different groups of stakeholders and audiences. (researchers, S&T and health policy makers, funders), as well as different geographical areas: Europe, Africa, Latin America and Asia.
AFRICAN SCIENCE TECHNOLOGY AND INNOVATION Below we present a summary of the main dissemination activities carried out by the project, in which there has been an active participation of projects partners and even engagement with stakeholders to make them aware of the project and the results.
- EVAL-HEALTH Expert Workshop. Madrid (Spain) 9-12 MAY 2011. Project workshop. Participants: All project partners, external experts, EC. Carried out at the beginning of the project, the workshop served the purpose of presenting and discussing the project approach with the European Commission and with experts from both donor and recipient countries.
- INDICATORS (ASTII) INITIATIVE WORKSHOP. Addis Ababa (Ethiopia). 23-25 MAY 2011. Participation of the project coordinator, presenting the project to African S&T Policymakers. The workshop was organized by the African Union.
- SECOND GLOBAL SYMPOSIUM ON HEALTH SYSTEMS RESEARCH (HSR) Beijing (China) , 31Oct-3 Nov 2012. Participation of partners from the U. Leeds and U. Mahidol, presenting the project results on the research on evidence (poster and oral presentations). The International conference was attended by Researchers, policy-makers, funders, implementers, civil society and media representatives.
- WORKSHOP ON FRAMEWORK CONDITIONS OF JOINT PROGRAMMING INITIATIVES: GOVERNANCE AND EVALUATION PRINCIPLES. Warsaw (Poland) 18-19 Dec 2012. Participation of the project coordinator to present the project. The workshop was organized by 7FP Project JPIs To Co Work, and was attended by EU S&T policymakers.
- PROJECT WORKSHOP: METHODOLOGIES FOR MONITORING AND EVALUATION OF RESULTS REGARDING PROJECTS AND PROGRAMMES ON RESEARCH AND DEVELOPMENT (R&D). A CASE STUDY: THE METHODOLOGY PROPOSED BY THE EVAL-HEALTH PROJECT FOR HEALTH RESEARCH" 7-8 Oct 2013, Bogotá, Colombia. Dissemination and Discussion of the IOM methodology. Together with the coordinator, partners from NEPAD, Mahidol University and Universidad de Rosario participated in the workshop, as well as experts and S&T policymakers from Latin America.
- IX IBERO-AMERICAN CONGRESS ON SCIENCE AND TECHNOLOGY INDICATORS. Bogota (Colombia) 9-11 October 2013. Dissemination of the project results (Research methodology) in an Oral Presentation. The Congress in a major event for Latin American, Spanish and Portuguese researchers active in ST indicators, as well as S&T Policymakers (ministries and funding agencies).
- FTEVAL CONFERENCE, "NEW HORIZONS / NEW CHALLENGES: EVALUATION OF STI POLICIES, INSTRUMENTS AND ORGANISATIONS”. Vienna (Austria), 12-13 Nov 2013. External workshop organized by the Austrian Platform for Research & Technology Policy Evaluation (FTEVAL platform). Presentation of IOM methodology (poster) to researchers and policy makers, mainly from European countries.
- HEALTH SYSTEMS IN ASIA: EQUITY, GOVERNANCE AND SOCIAL IMPACT. Singapore, 13-16 Dec 2013. External workshop for Health policy makers and researchers, mainly from Asian countries. Participation of U. Leeds and Mahidol University to present (oral presentation and poster) project results (research on evidence).
-“BUILDING PARTNERSHIPS FOR SUSTAINABLE CAPACITY DEVELOPMENT IN MEDICINES REGULATION IN AFRICA”, External workshop. 2-3 Dec 2013, Johannesburg, South Africa. Dissemination of the project results (Research methodology) in a poster presented by NEPAD. Audience: African Health Policy makers.
- THE ASEAN-EU SCIENCE, TECHNOLOGY AND INNOVATION DAYS 2014 (STI Days) External workshop 21-22 Jan 2014. Bangkok, Thailand. Dissemination of the project work (Stall with posters). The conference is attended by EC and ASEAN S&T policy makers and researchers.
- NEPAD AGENCY/NSTIH WORKSHOP ON STI INDICATORS AND MEASUREMENT OF IMPACTS OF HUMAN DEVELOPMENT PROGRAMMES. 1-2 Feb 2014. Praia, Cape Verde. Project workshop organized to discuss the IOM methodology with African S&T policymakers and M&E experts.
- CONTACTS WITH THE CANADIAN COALITION ON GLOBAL HEALTH RESEARCH (CCGHR). SKYPE CONFERENCE, May 2014. Presentation of the EVAL-HEALTH project and the IOM methodology to research group in charge of the background paper on Perspectives on Evaluating Global Health Research for Development, prepared for Canada’s Global Health Research Initiative in collaboration with ESSENCE.
- INTERNATIONAL CONFERENCE HEALTH SERVICES RESEARCH: EVIDENCE-BASED PRACTICE. 1st-3rd July 2014. London, UK.. Dissemination of the project results (research on evidence) by Oral presentation and Poster. Attended by the U. Leeds and Instituto de Higiene e Medicina Tropical de la U. Lisboa. The Conference audience includes International academicians, policy makers health practitioners and researchers.
- WEBINAR/PRESENTATION FOR PAHO - WHO, 2nd. July 2014. Online 1 hour seminar requested by PAHO -WHO regional office in Washington to present the IOM methodology to Assistant Director and Senior Staff from PAHO. The methodology was presented by Innovatec.
- DELHI SYMPOSIUM ON GLOBAL GOVERNANCE AND COMMERCIALISATION OF PUBLIC HEALTH New 7th -10th Sept 2014 Delhi. India. External workshop. Dissemination of the project results (research on evidence) by Oral presentation (U. Leeds) to Researchers and Civil Society Organizations.
- 3RD GLOBAL SYMPOSIUM ON HEALTH SYSTEM RESEARCH, 29 Sept-3 Oct 2014Cape Town, (S. Africa). Most important event to present and disseminate the project work and results. Eval-health organized two specific Satellite sessions, had a stall in the Marketplace throughout the conference, presented 4 posters and 1 oral presentation. All project partners participated in this event, which is attended by over 2000 people coming from all countries and major stakeholders: researchers, policy-makers, health funders, implementers, civil society and media representatives.
- PRESENTATION OF THE PROJECT RESULTS TO THE EC. Internal meeting organized on 8th. October 2014 in Brussels, Belgium, at the EC DG Research and Innovation. The coordinators together with team leader from the U. Leeds presented project results and discussed potential use by the EC. Officers from the EC DG Research and Innovation and DG. DEVCO attended the meeting.
List of Websites:
PROJECT WEB-SITE: www.eval-health.eu
Relevant contacts
Project coordinator and team leader for the IOM methodology
Dr. JOAQUIN GUINEA
INNOVATEC (Spain)
joaquin.guinea@innovatec.es
Team leader for the evidence research
Dr. Tolib Mirzoev
University of Leeds (UK)
t.mirzoev@leeds.ac.uk
