Evaluation, considered as an integral part of the educational process in many countries, takes place at different levels with a wide range of practices. The full complexity of educational evaluation at various levels (i.e. system, school, classroom, and student) in different countries has hardly been recognized. It constitutes a tremendous barrier to consolidate theories on evaluation, and for countries to draw on successful experience and implement targeted educational policies for better quality and effectiveness. Important barriers include the insufficient methodology to soundly compare multiple student groups and multiple countries and a cultural insensitive approach to unfolding evaluation processes and outcomes. This project aims to provide a comprehensive framework for the dynamics of evaluation. It delineates the pathways in processes and outcomes of educational evaluation with data from the Programme for International Student Assessment (PISA) with a solid cross-cultural research methodology.
The substantive challenge lies in that evaluation research at various levels is not synchronized or coordinated, and results are not consistent across countries, presumably due to a lack of integration of levels and overlooking relevant moderators. In this project, a broader range of outcomes is proposed, a thorough scrutiny of evaluation practices is conducted, and various moderators from different levels are highlighted.
The methodological challenge rests upon data incomparability in large-scale assessment and surveys. In multiple cultural contexts, measurement bias from construct, method, and item levels can invalidate comparative results. In other words, whether, in different countries, (i) target constructs (e.g. student motivation to learn) are understood as having the same meaning, (ii) the instruments (e.g. Likert-scale response options) are used in the same manner, and (iii) item content has an unequivocal interpretation, need to be demonstrated before any meaningful comparisons are made. Thus, to validly compare countries or cultural groups, the comparability of data should be gauged with adapted survey designs and advanced psychometric tools.
Overall, this project aims to contribute to a better understanding of the processes and outcomes of evaluation in a variety of educational systems in two studies. It is expected to consolidate theories from a cross-cultural perspective and provide efficient policy recommendations tailored to specific contexts.
