Periodic Reporting for period 1 - ILSAINC (Mapping and theorising the global International Large-Scale Assessment industry)
Berichtszeitraum: 2019-12-01 bis 2021-11-30
In the last two decades, education policy, practice and research have been transformed by the appearance and growing influence of International Large-Scale Assessments (ILSAs). But how are ILSAs shaped by the international education assessment market they have created? Scholars have extensively studied how ILSAs have impacted on education, whilst ignoring the organisations developing ILSAs and the processes of contracting ILSAs. This study investigates and theorizes the international education assessment market with the aim of understanding how its actors and workings influence ILSA tests, procedures and the data ILSAs produce (referred to as ‘ILSAs’ from here on).
The ILSAINC research project had 3 objectives: Firstly, to map the international education assessment market actors by identifying the actors to whom the development of ILSAs is outsourced. Secondly, to theorize how the global ILSA industry emerged, stabilised and operates - particularly focusing on institutions and relationships, the network functioning, forms of agency, and power practices. Thirdly, to investigate how ILSAs are shaped by the practices, values, profit interests and power dynamics of the private sector.
The ILSAINC research project had 3 objectives: Firstly, to map the international education assessment market actors by identifying the actors to whom the development of ILSAs is outsourced. Secondly, to theorize how the global ILSA industry emerged, stabilised and operates - particularly focusing on institutions and relationships, the network functioning, forms of agency, and power practices. Thirdly, to investigate how ILSAs are shaped by the practices, values, profit interests and power dynamics of the private sector.
Throughout the project implementation, the following work packages were implemented.
1.2.1 Work Package 1
In the first stage of the Action, I broadened my theoretical and methodological understanding by reviewing literature on global education policy and the privatization of education (including the Global Education Industry field), network ethnography, business studies of networks, Actor-Network Theory, and International Large-Scale Assessment research.
1.2.2 Work package 2
This package involved learning by reading and scholarly interactions with colleagues and my Fellowship supervisor, how to: carry out interdisciplinary research; use innovative research methods; strengthen research project management skills; improve my publishing skills, apply for funding and academic jobs skills. The implementation of this project allowed me to develop all these skills.
1.2.3 Work package 3
From February 2020 to June 2020, I carried out all the document analysis and interviews planned for the ILSAINC.
1.2.4 Work package 4
After an extended maternity leave and the final period of the pandemic, I coded all the data in Nvivo and analysed the emerging themes in relation to the questions and objectives of the ILSAINC.
1.2.5 Work package 5
I worked on sole-authored publications that presented the main findings from the research project. I also presented at online seminars and at conferences, though this was limited by the pandemic and challenging family issues.
1.2.6 Work package 6
As planned in my deliverable list, I have written up a paper that is aimed at non-academic audiences (still under review) and gave an online seminar to an audience that included non-academic audiences, in particular the private sector.
1.2.7 Work package 7
Together with focusing on publications, I developed two research projects and applied for tenure positions and grants that would cover the costs of the implementation of my next research projects.
1.2.1 Work Package 1
In the first stage of the Action, I broadened my theoretical and methodological understanding by reviewing literature on global education policy and the privatization of education (including the Global Education Industry field), network ethnography, business studies of networks, Actor-Network Theory, and International Large-Scale Assessment research.
1.2.2 Work package 2
This package involved learning by reading and scholarly interactions with colleagues and my Fellowship supervisor, how to: carry out interdisciplinary research; use innovative research methods; strengthen research project management skills; improve my publishing skills, apply for funding and academic jobs skills. The implementation of this project allowed me to develop all these skills.
1.2.3 Work package 3
From February 2020 to June 2020, I carried out all the document analysis and interviews planned for the ILSAINC.
1.2.4 Work package 4
After an extended maternity leave and the final period of the pandemic, I coded all the data in Nvivo and analysed the emerging themes in relation to the questions and objectives of the ILSAINC.
1.2.5 Work package 5
I worked on sole-authored publications that presented the main findings from the research project. I also presented at online seminars and at conferences, though this was limited by the pandemic and challenging family issues.
1.2.6 Work package 6
As planned in my deliverable list, I have written up a paper that is aimed at non-academic audiences (still under review) and gave an online seminar to an audience that included non-academic audiences, in particular the private sector.
1.2.7 Work package 7
Together with focusing on publications, I developed two research projects and applied for tenure positions and grants that would cover the costs of the implementation of my next research projects.
Main scientific results
1. The ILSAINC mapped the contractors involved in ILSAs between 1990 and 2020. The project implementation included the creation of visual mappings of all the OECD and IEA ILSA contractors between 1990 and 2020.
3. The ILSAINC explains why contractors compete for cut-even or loss contracts. Drawing on business studies, the ILSAINC classified and analysed the reasons why contractors seek to obtain and implement ILSA contracts. These relate to: developing improved or new products and processes, gaining access to networks, managing human capital, and creating direct business opportunities. In particular, ILSAs contracts are sought for the opportunity to develop and test new methodologies, learn about new assessment ideas, and carry out research. ILSAs are valued in terms of the network capital they provide, especially since they give access to governments as potential clients all around the world. ILSA contracts provide contractors with an international halo of credibility and are strategic in terms of opening up new markets and acquiring big contracts. The study of ILSA contracts identifies rationales that further the categories described in business studies by adding a fifth category of individual rationales. This latter category shows that contractors use ILSA contracts to further professional aspirations, lifestyles, and emotional bonds.
4. The ILSAINC demonstrated that ILSAs are presented as innovative, whilst at the same time, they are not innovative in practice. Although ILSAs are described as the golden standard in education assessment by the OECD, the IEA, and contractors, in practice, there is little space for innovation. While ILSAs are described as a methodological playground to test new theories and learn new assessment ideas, contractors ‘copy and paste’ their previous work because of economic limitations and lack of shared agreement. In sum, there is a gap between the ILSA innovation rhetoric and innovation in practice.
5. The ILSAINC analysed how ILSA contracting emerged and evolved: The ILSAINC findings captured how a community of practice at the IEA carried out the first ILSAs through personal networks and good will. This small network of actors developed social and cultural capitals (including trust) that allowed them to obtain contracts once substantial funding became available for ILSAs. The availability of funding and growing interest in the outcomes of ILSAs led to the first struggles in this space. A split in the IEA community emerged: this resulted in the birth of PISA, the most influential ILSA. The space was now populated by competing ILSAs and competing contractors, with the IEA openly relying on relationships of trust whilst the OECD used a global tendering process to conceal interpersonal and inter-organizational struggles.
6. The emergence of a global assessment market in education. Drawing on market sociology to analyse the interrelations of institutions, networks, and cognition in ILSA contracting, the project demonstrates how the OECD has stimulated the emergence of a global assessment market in education.
7. The ILSAINC contributed evidence on how International Organizations make knowledge, in particular through the selection of experts and their management. The OECD acts as a boundary organization, coordinating and easing connections among hundreds of experts, in the making of comparative knowledge about learning outcomes through PISA. What emerges is that through the management of hundreds of experts (including the contractors) through institutional boundary practices, the PISA Secretariat takes back decision-making power to steer the making of knowledge.
8. Ultimately, the ILSAINC demonstrates that the ILSA market was, and remains, a small market with tall walls and poor profit. Despite this, it is increasingly non-competitive, desirable, and exclusive. The project suggests that the rationales and dynamics in the ILSA market may not be in the best interest of education. Working in education, and in light of the far-reaching policy implications of international large-scale assessments, means all actors involved must be committed to ensuring the education they are making—the worlds they are bringing into being—are building more just and sustainable societies.
1. The ILSAINC mapped the contractors involved in ILSAs between 1990 and 2020. The project implementation included the creation of visual mappings of all the OECD and IEA ILSA contractors between 1990 and 2020.
3. The ILSAINC explains why contractors compete for cut-even or loss contracts. Drawing on business studies, the ILSAINC classified and analysed the reasons why contractors seek to obtain and implement ILSA contracts. These relate to: developing improved or new products and processes, gaining access to networks, managing human capital, and creating direct business opportunities. In particular, ILSAs contracts are sought for the opportunity to develop and test new methodologies, learn about new assessment ideas, and carry out research. ILSAs are valued in terms of the network capital they provide, especially since they give access to governments as potential clients all around the world. ILSA contracts provide contractors with an international halo of credibility and are strategic in terms of opening up new markets and acquiring big contracts. The study of ILSA contracts identifies rationales that further the categories described in business studies by adding a fifth category of individual rationales. This latter category shows that contractors use ILSA contracts to further professional aspirations, lifestyles, and emotional bonds.
4. The ILSAINC demonstrated that ILSAs are presented as innovative, whilst at the same time, they are not innovative in practice. Although ILSAs are described as the golden standard in education assessment by the OECD, the IEA, and contractors, in practice, there is little space for innovation. While ILSAs are described as a methodological playground to test new theories and learn new assessment ideas, contractors ‘copy and paste’ their previous work because of economic limitations and lack of shared agreement. In sum, there is a gap between the ILSA innovation rhetoric and innovation in practice.
5. The ILSAINC analysed how ILSA contracting emerged and evolved: The ILSAINC findings captured how a community of practice at the IEA carried out the first ILSAs through personal networks and good will. This small network of actors developed social and cultural capitals (including trust) that allowed them to obtain contracts once substantial funding became available for ILSAs. The availability of funding and growing interest in the outcomes of ILSAs led to the first struggles in this space. A split in the IEA community emerged: this resulted in the birth of PISA, the most influential ILSA. The space was now populated by competing ILSAs and competing contractors, with the IEA openly relying on relationships of trust whilst the OECD used a global tendering process to conceal interpersonal and inter-organizational struggles.
6. The emergence of a global assessment market in education. Drawing on market sociology to analyse the interrelations of institutions, networks, and cognition in ILSA contracting, the project demonstrates how the OECD has stimulated the emergence of a global assessment market in education.
7. The ILSAINC contributed evidence on how International Organizations make knowledge, in particular through the selection of experts and their management. The OECD acts as a boundary organization, coordinating and easing connections among hundreds of experts, in the making of comparative knowledge about learning outcomes through PISA. What emerges is that through the management of hundreds of experts (including the contractors) through institutional boundary practices, the PISA Secretariat takes back decision-making power to steer the making of knowledge.
8. Ultimately, the ILSAINC demonstrates that the ILSA market was, and remains, a small market with tall walls and poor profit. Despite this, it is increasingly non-competitive, desirable, and exclusive. The project suggests that the rationales and dynamics in the ILSA market may not be in the best interest of education. Working in education, and in light of the far-reaching policy implications of international large-scale assessments, means all actors involved must be committed to ensuring the education they are making—the worlds they are bringing into being—are building more just and sustainable societies.