What is a well written text? Holistic and analytic assessment of text quality in L1 and L2 Italian

Project Information

WeWriTe

Grant agreement ID: 101066322

DOI

10.3030/101066322

EC signature date 10 August 2022

Start date 1 February 2023

End date 31 January 2028

Funded under

• Marie Skłodowska-Curie Actions (MSCA)

Total cost

No data

EU contribution

€ 368 071,68

Coordinated by

UNIVERSITA DEGLI STUDI DI MODENA E REGGIO EMILIA
Italy

Periodic Reporting for period 1 - WeWriTe (What is a well written text? Holistic and analytic assessment of text quality in L1 and L2 Italian)

Reporting period: 2023-02-01 to 2026-07-31

Summary of the context and overall objectives of the project

The project WeWriTe aimed to investigate the linguistic features of texts perceived as “well-written”, addressing a central challenge in writing research and education. Although writing skills are widely recognised as crucial for academic success, professional development, and participation in society, there is still limited consensus on how writing quality should be defined, measured, and explained. Evaluators often rely on holistic impressions or metaphorical descriptions such as smoothness and flow, which represent perceptions deriving from the convergence of multiple features; the difficulty to differentiate and caputre these features makes the evaluation of written texts particularly challenging for educators, especially in multilingual contexts.
Against this background, the project addressed two interrelated research questions: how holistic text quality can be assessed in a valid and reliable way, and which linguistic features predict evaluators’ holistic judgments of text quality.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

As a first step, a systematic review and meta-analysis of the state of the art was conducted, which resulted in the publication of a review article on the plattform Open Research Europe. The review examined how writing quality is operationalised and measured in research, and how reliability and validity are ensured in writing assessment. The results show that writing quality is most commonly operationalised through holistic or analytic rating scales applied by trained evaluators. Reliability is typically ensured through rater training and the calculation of inter-rater agreement, while intra-rater reliability is rarely examined systematically. Moreover, the analysis suggests that the validity of evaluation instruments is often assumed rather than empirically demonstrated.
These findings informed the methodological design of the project. Building on the review, the project developed a quantitative modelling framework to investigate the relationship between linguistic features of texts and evaluators’ holistic judgments of text quality. The project combined expert-based holistic evaluation with multivariate statistical modelling in order to analyse how multiple linguistic dimensions jointly contribute to the perception of writing quality.
An important component of the fellowship concerned the training of the researcher. Training activities focused primarily on acquiring advanced methodological expertise in quantitative data analysis, in particular on learning how to apply the complex statistical technique of partial least squares structural equation modelling (PLS-SEM). At the same time, the researcher deepened her theoretical knowledge of text linguistics through collaboration with Prof. Angela Ferrar, which provided in-depth training on the Basel model of text analysis and its application to written texts. This combination of methodological and theoretical training significantly strengthened the researcher’s interdisciplinary expertise at the intersection of linguistics, language aeducation and educational assessment.
The project also created opportunities for knowledge transfer and professional development through collaborations with teachers of Italian in Switzerland and Italy. These interactions fostered dialogue between research and educational practice and contributed to exploring how linguistic insights can inform the teaching and assessment of writing.
The results of the project were disseminated through academic publications, conference presentations, and invited lectures at international venues, including prestigious events such as the Hermann Paul School of Linguistics. Through these activities, the project contributed to ongoing discussions on writing assessment and the linguistic foundations of text quality.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Finally, the project had a strong impact on the researcher’s career development. The expertise acquired during the fellowship, the international collaborations established, and the dissemination of results through publications and invited talks significantly strengthened the researcher’s academic profile. During the fellowship, the researcher obtained a tenure-track position at the University of Bologna and therefore concluded the project after 18 months instead of the originally planned 42 months. Nevertheless, the fellowship played a decisive role in enabling this career progression, and the research agenda initiated within the project will continue to be developed in the researcher’s current position.

wewrite-pic.png