Periodic Reporting for period 4 - FluidKnowledge (How evaluation shapes ocean science. A multi-scale ethnography of fluid knowledge.)
Reporting period: 2023-09-01 to 2025-02-28
The FluidKnowledge project investigated how evaluation practices shape ocean science through innovative multi-scale ethnographic research conducted across European marine research institutions and international policy venues. The project addressed the critical need to understand how scientific knowledge about the ocean is produced, evaluated, and mobilized across different institutional contexts. Ocean science faces unprecedented challenges requiring interdisciplinary collaboration and societal engagement, yet little is known about how evaluation practices influence knowledge production and translation in this field. The project achieved its core objectives despite significant adaptations required due to the COVID-19 pandemic. The team addressed the fundamental question of how evaluation practices shape scientific understanding of the ocean. Traditional approaches to studying science focus on either micro-level laboratory practices or macro-level policy outcomes, but fail to capture how ocean knowledge moves and transforms across different scales and contexts. This gap in understanding limits our ability to optimize how ocean science contributes to addressing critical environmental and societal challenges, particularly in areas like climate change, sustainable resource use, and environmental protection. Understanding how evaluation practices shape this knowledge is essential for improving science-policy interfaces, enhancing interdisciplinary collaboration, and ensuring that ocean research effectively serves societal needs. The findings help optimize how scientific knowledge about our oceans is produced, evaluated, and applied to real-world problems, ultimately contributing to better-informed decision-making on critical environmental issues. The project's objectives were to:
(i) develop an in-depth understanding of how research agendas are shaped in ocean science;
(ii) describe the specific values that guide concrete science policy steering efforts, in particular evaluation and accountability measures, with respect to the role of ocean science in and for society, and how these efforts take shape in EU and national policy contexts;
(iii) develop new concepts to theoretically grasp whether and how research evaluation shapes knowledge making, based on these multi-level approaches.
(i) develop an in-depth understanding of how research agendas are shaped in ocean science;
(ii) describe the specific values that guide concrete science policy steering efforts, in particular evaluation and accountability measures, with respect to the role of ocean science in and for society, and how these efforts take shape in EU and national policy contexts;
(iii) develop new concepts to theoretically grasp whether and how research evaluation shapes knowledge making, based on these multi-level approaches.
The FluidKnowledge project successfully conducted multi-scale ethnographic research across European marine research institutions and international policy venues. Severe COVID-19 challenges required significant methodological adaptations. The team pursued its core objectives by including digital ethnography, remote interviews, and the creation of the FluidKnowledge podcast series. This series featured in-depth discussions with leaders in ocean science about conducting research in turbulent times while addressing science governance and grand challenges. The project resulted in a comprehensive scientometric analysis of ocean science publications spanning four decades (1980-2020), revealing critical shifts in research priorities, methodological approaches, and institutional dynamics. The team developed specialized computational tools for analyzing ocean science, including: field delineation algorithms for mapping ocean science's evolving boundaries; term co-occurrence networks to visualize changing research priorities; co-authorship networks to map institutional collaborations; visualization techniques for representing scientific uncertainty. These tools were complemented by (virtual) ethnographic methods that examined how evaluation practices operate in diverse institutional contexts: digital ethnography during COVID and post-pandemic participant observation at marine research institutions; interviews with scientists, policymakers, and funding and NGO representatives; document analysis of evaluation protocols and policy statements; attendance at key governance and policy meetings. This methodological innovation and integration was further developed through a panel organized at the 2022 EASST Conference on "Methodological experiments in STS: Exploring digital and (quali-)quantitative methods" by project team members and external collaborators. As part of the project, the team conducted extensive fieldwork and research engagements with leading European marine research policy bodies, enabling in-depth analysis of how evaluation practices and policy frameworks interact to shape both the direction and societal relevance of ocean science. The team documented how scientists navigate competing demands to produce both fundamental knowledge and policy-relevant research. This work was complemented by field research at deep sea (mining) governance venues, revealing how evaluation criteria shape which forms of deep sea knowledge are considered legitimate in regulatory contexts. By examining the intersection of scientific uncertainty, commercial interests, and environmental governance, this work illuminates how evaluation practices privilege certain knowledge forms while marginalizing others. Key project results include: comprehensive analysis of institutional and infrastructural contexts shaping deep sea research; extensive fieldwork at high-level international policy venues including European Marine Board meetings and deep seabed mining policy forums (culminating in a new PhD project for the team's research assistant); scientometric analyses of ocean science and marine social science; the development of the theoretical concept of "brackish knowledge"; multiple peer-reviewed publications disseminating findings to the scientific community (several under review at the time of reporting).
The FluidKnowledge project moves beyond the state of the art in the following ways. It moves beyond merely providing methodological critique of evaluation. Instead, the project understands research evaluation as a thoroughly social process. It also moves beyond focusing on formal evaluation systems. Instead, the project studies how evaluation plays out on the ground, in real organisational settings. Finally, the project moves beyond the idea of research institutes as homogeneous. Instead, the project approaches research as placed between many different demands. Think of scientific values versus market logics, or of science that scores versus science that matters.
Upon completion, the project will have collected hard, empirical evidence on how evaluations affect actual scientific conduct in ocean science, by combining scientometrics with ethnography. The project will also have advanced conceptual research about combining quantitative science studies and science and technology studies. Combining these two fields and methodological traditions – which are rarely combined due to institutional and methodological reasons – is highly novel. It is also essential for developing a new, interdisciplinary field at the intersection of Evaluation Studies and Science and Technology Studies (one of the main goals of FluidKnowledge). Thirdly, the project will have developed a new theory of evaluation based on this evidence: a theory for understanding interactions between research conduct and research evaluation.
Upon completion, the project will have collected hard, empirical evidence on how evaluations affect actual scientific conduct in ocean science, by combining scientometrics with ethnography. The project will also have advanced conceptual research about combining quantitative science studies and science and technology studies. Combining these two fields and methodological traditions – which are rarely combined due to institutional and methodological reasons – is highly novel. It is also essential for developing a new, interdisciplinary field at the intersection of Evaluation Studies and Science and Technology Studies (one of the main goals of FluidKnowledge). Thirdly, the project will have developed a new theory of evaluation based on this evidence: a theory for understanding interactions between research conduct and research evaluation.