Periodic Reporting for period 2 - EFFORTI (Evaluation Framework for Promoting Gender Equality in Research and Innovation)
Reporting period: 2017-09-01 to 2019-05-31
The EFFORTI team developed, tested and validated an evaluation framework that did not only draw on existing knowledge on how to measure gender equality and research and innovation results but also integrated current discourses, i.e. how traditional approaches can be extended regarding the integration of the concept of responsible research and innovation. The high number of conference presentations and scientific articles confirm the innovative approach of the EFFORTI project. The conceptual evaluation framework underpins the understanding of gender equality as a multidimensional issue, and of gender equality policies as complex processes of which the design, implementation, output, outcomes and impact depend on the interaction of a multiplicity of variables in dynamic contexts.
The evaluation framework and the online toolbox allow ministries, enterprises, research performing as well as research funding organizations to assess their measures to promote women or gender aspects in research projects efficiently and in a targeted manner. The novel contribution of EFFORTI to evaluation practice is that it‘s framework not only allows the assessment of gender equality targets but rather effects on research and innovation results in general. The outcomes of the project are available via project reports, scientific publications and an online toolbox, targeting an academic as well as a non-academic audience. The EFFORTI online toolbox proposes an innovative evaluation framework and contains relevant parameters and indicators as well as inspiration and help for how to design a theory-of-change for gender equality measures.
The evaluation framework and the online toolbox allow ministries, enterprises, research performing as well as research funding organizations to assess their measures to promote women or gender aspects in research projects efficiently and in a targeted manner. The novel contribution of EFFORTI to evaluation practice is that it‘s framework not only allows the assessment of gender equality targets but rather effects on research and innovation results in general. The outcomes of the project are available via project reports, scientific publications and an online toolbox, targeting an academic as well as a non-academic audience. The EFFORTI online toolbox proposes an innovative evaluation framework and contains relevant parameters and indicators as well as inspiration and help for how to design a theory-of-change for gender equality measures.
"The interventions and measures for which the EFFORTI toolbox aims to provide evaluation support are embedded in different contexts with regard to socio-economic and political systems and organizational settings and thus develop differently according to their context. For this purpose, seven country notes have been written (AT, DK, FR, DE, HU, ES, SWE) where the contextual framework conditions like the structure and performance of the innovation Systems and the governance and relevance of gender equality have been described. These reports also mapped existing evaluations of initiatives to promote gender equality and the dominant evaluation culture in each country.
In order to develop the conceptual evaluation framework, the study team carried out a comprehensive desk research as a basis for the collection of a preliminary list of relevant indicators. Particular emphasis was given to map existing evaluation concepts of GE measures and instruments, concepts for the measurement of research and innovation outputs, approaches for impact assessment and studies how to monitor RRI. Based on existing evidence, the project team undertook the following steps: (1) Identifying the most relevant indicators suggested by the literature review, (2) Clustering these indicators into different categories, dimensions and sub-dimensions. (3) Substantiating and conceptualising these indicators according to an evaluation logic model differentiating between input, (throughput,) output, outcome and impact aspects. (4) The indicators were illustrated at micro/individual or team level, meso/organisational level and macro/policy or country level. Contextual indicators were treated as cross-cutting indicators which are assumed to have relevant influence on indicators from input to outcome. Finally, the project carried out 19 case studies distributed over six countries. The case studies validated the evaluation framework in different empirical settings and tested its applicability for different types of equality measures in three main ways 1) theory based evaluation: theory of change, 2) validation of indicators, 3) impact stories.
The overall EFFORTI Toolkit v2.0 is deployed in the official website https://www.efforti.eu and provides (a) a detailed overview of the EFFORTI evaluation framework which is a critical aspect of the project, applied to all tools and a constant reference point for the end-user, and (b) access to the three core tools and services of the toolkit. These core tools are: The Impact Story Knowledge Base; The Programme Theory Generator and The Evaluation Framework. The toolbox captures in a systematic and visually appealing manner the work undertaken throughout the project and can be used by users to design their own evaluations based on the theoretical background and the methodology provided by the EFFORTI project. At the same time the visual illustration of the indicators allows to make connections that are not self-evident and thus might allow to further experimentation in specific cases.
Overall, the project team published eight project reports and eleven peer-reviewed scientific articles, among them ten in a special issue of the high-ranked journal ""Evaluation and Program Planning"". The team also presented EFFORTI at 17 national and international conferences. Furthermore, the EFFORTI team compiled three summary papers that contain the most important results of the context analysis, the coneptual evaluation framework and the case study work, wrote 17 so-called impact stories as ideal-type impact pathways for gender equality interventions and visualized these impact stories via log frame pictures. Finally, the EFFORTI team summarized its main lessons learned and key recommendations and gave an overview on the main performance effects of gender diversity."
In order to develop the conceptual evaluation framework, the study team carried out a comprehensive desk research as a basis for the collection of a preliminary list of relevant indicators. Particular emphasis was given to map existing evaluation concepts of GE measures and instruments, concepts for the measurement of research and innovation outputs, approaches for impact assessment and studies how to monitor RRI. Based on existing evidence, the project team undertook the following steps: (1) Identifying the most relevant indicators suggested by the literature review, (2) Clustering these indicators into different categories, dimensions and sub-dimensions. (3) Substantiating and conceptualising these indicators according to an evaluation logic model differentiating between input, (throughput,) output, outcome and impact aspects. (4) The indicators were illustrated at micro/individual or team level, meso/organisational level and macro/policy or country level. Contextual indicators were treated as cross-cutting indicators which are assumed to have relevant influence on indicators from input to outcome. Finally, the project carried out 19 case studies distributed over six countries. The case studies validated the evaluation framework in different empirical settings and tested its applicability for different types of equality measures in three main ways 1) theory based evaluation: theory of change, 2) validation of indicators, 3) impact stories.
The overall EFFORTI Toolkit v2.0 is deployed in the official website https://www.efforti.eu and provides (a) a detailed overview of the EFFORTI evaluation framework which is a critical aspect of the project, applied to all tools and a constant reference point for the end-user, and (b) access to the three core tools and services of the toolkit. These core tools are: The Impact Story Knowledge Base; The Programme Theory Generator and The Evaluation Framework. The toolbox captures in a systematic and visually appealing manner the work undertaken throughout the project and can be used by users to design their own evaluations based on the theoretical background and the methodology provided by the EFFORTI project. At the same time the visual illustration of the indicators allows to make connections that are not self-evident and thus might allow to further experimentation in specific cases.
Overall, the project team published eight project reports and eleven peer-reviewed scientific articles, among them ten in a special issue of the high-ranked journal ""Evaluation and Program Planning"". The team also presented EFFORTI at 17 national and international conferences. Furthermore, the EFFORTI team compiled three summary papers that contain the most important results of the context analysis, the coneptual evaluation framework and the case study work, wrote 17 so-called impact stories as ideal-type impact pathways for gender equality interventions and visualized these impact stories via log frame pictures. Finally, the EFFORTI team summarized its main lessons learned and key recommendations and gave an overview on the main performance effects of gender diversity."
The EFFORTI conceptual evaluation framework embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The EFFORTI framework contributes to opening the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. Finally, the EFFORTI evaluation framework takes long-term Impacts on Research and innovations processes and outcomes systematically into account. Examples of socio-economic impacts expected from GE interventions in R&I therefore comprise thus long-term socio-economic impacts that are possible to measure in a long-term perspective. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base. One of the main aims of EFFORTI was to study how gender equality interventions are contributing to society. The theory of change and impact stories demonstrate how GE intervention impacts are expected to occur and materialize. Moreover, the articles included in the Special Issue of Evaluation and Program Planning present some very concrete cases of how the results of EFFORTI can be used to demonstrate and achieve socio-economic impact.