As a first step, a systematic review and meta-analysis of the state of the art was conducted, which resulted in the publication of a review article on the plattform Open Research Europe. The review examined how writing quality is operationalised and measured in research, and how reliability and validity are ensured in writing assessment. The results show that writing quality is most commonly operationalised through holistic or analytic rating scales applied by trained evaluators. Reliability is typically ensured through rater training and the calculation of inter-rater agreement, while intra-rater reliability is rarely examined systematically. Moreover, the analysis suggests that the validity of evaluation instruments is often assumed rather than empirically demonstrated.
These findings informed the methodological design of the project. Building on the review, the project developed a quantitative modelling framework to investigate the relationship between linguistic features of texts and evaluators’ holistic judgments of text quality. The project combined expert-based holistic evaluation with multivariate statistical modelling in order to analyse how multiple linguistic dimensions jointly contribute to the perception of writing quality.
An important component of the fellowship concerned the training of the researcher. Training activities focused primarily on acquiring advanced methodological expertise in quantitative data analysis, in particular on learning how to apply the complex statistical technique of partial least squares structural equation modelling (PLS-SEM). At the same time, the researcher deepened her theoretical knowledge of text linguistics through collaboration with Prof. Angela Ferrar, which provided in-depth training on the Basel model of text analysis and its application to written texts. This combination of methodological and theoretical training significantly strengthened the researcher’s interdisciplinary expertise at the intersection of linguistics, language aeducation and educational assessment.
The project also created opportunities for knowledge transfer and professional development through collaborations with teachers of Italian in Switzerland and Italy. These interactions fostered dialogue between research and educational practice and contributed to exploring how linguistic insights can inform the teaching and assessment of writing.
The results of the project were disseminated through academic publications, conference presentations, and invited lectures at international venues, including prestigious events such as the Hermann Paul School of Linguistics. Through these activities, the project contributed to ongoing discussions on writing assessment and the linguistic foundations of text quality.
