Periodic Reporting for period 1 - NNLPP (Non-Native Language Processing and Production: Evidence for an Integrated Grammar)
Reporting period: 2023-01-01 to 2025-06-30
Summary of the context and overall objectives of the project
Across today’s Europe, it is not surprising to see a family consisting of a Spanish father, an Irish mother and children speaking German natively in a restaurant, say, in the Netherlands. Parents move around for work, children change schools, and all of them acquire new languages every time they move. Despite a significant body of research on multilingualism, a few key questions have not been fully addressed yet: What psycholinguistic mechanisms ensure the successful addition of a new language to the existing mental grammar? How can people, both children and adults, understand complex sentences in a new language when their general proficiency in this language is quite low? The proposed project Non-Native Language Processing and Production: Evidence for an Integrated Grammar (NNLPP) addresses these questions from a novel perspective.
NNLPP is an integrated study which compares the strategies multilingual speakers use across their languages. The project will investigate the role of syntactic structure in comprehension and production of unilingual sentences and sentences which switch from one language to another. A series of experiments will test 250-300 adult non-balanced multilinguals whose languages belong to different linguistic families: Norwegian, Russian, and English. Focusing on code-switched sentences, NNLPP will compare the participants’ reading times, eye movements and intonation patterns and provide a well-rounded description of psycholinguistic mechanisms used by a multilingual mind.
The target sentences in NNLPP are code-switched at different places (1) manipulating the length of the code-switched fragment. I assume that code-switching in (1a) and (1b) forces a prosodic break at the places where the language change occurs, and that this prosodic break shapes sentence parsing and interpretation.
(1a) Bill saw the friend of the neighbor [language change: English > Russian] that was talking about football in the yard.
(1b) Bill saw the friend [language change: English > Russian] of the neighbor that was talking about football in the yard.
(1c) Bill saw [language change: English > Russian] the friend of the neighbor that was talking about football in the yard.
(1d). Bill saw the friend of the neighbor that was talking about football [language change: English > Russian] in the yard.
The places of code-switching in (1a) and (1b) follow the Implicit Prosody Hypothesis (Fodor, 2002), and I anticipate (1a) to be interpreted as 'the friend was talking' and (1b) as 'the neighbor was talking'. If this assumption is confirmed in both Norwegian-English and Russian-Norwegian code-switching, it will mean that a certain prosodic structure forced by code-switching is a universal processing cue that informs mental structure building.
The places of code-switching in the target sentences (1c) and (1d) manipulate the lengths of the code-switched fragment. In (1c) it keeps the bigger part of the sentence in one language, e.g. Russian, in (1d) in English. Typically, Russian speakers prefer to interpret sentences like in (1) as 'the friend was talking', and English speakers prefer the interpretation 'the neighbor was talking' (see Fodor, 2002, 1998 for full review). The preferred interpretations in (1 c, d) will show whether the sentence is parsed in a language-specific manner and follows the parsing preference of the language that supplies most of the lexical items. Thus, the participants are expected to prefer the interpretation 'the friend was talking' in (1c) and 'the neighbor was talking' in (1d).
An alternative prediction is also available for sentences (1c) and (1d). The language of the matrix verb provides the matrix language for code-switching (Myers-Scotton & Jake, 2009). Both (1c) and (1d) begin with English and the matrix verb is given in English. Therefore, the participants may show a preference for the interpretation 'the neighbor was talking' in both (3c) and (3d). As the last alternative, the participants may choose an L1-like parse in all the experimental items in (3).
NNLPP will use sentences like 1(a – d) in the processing experiments in language comprehension and production to test two hypotheses:
1. non-native sentence comprehension and production is an integrated process which follows universal parsing strategies (ex., prosodic cues)
2. in the absence of a universal parsing prompt, sentence comprehension and production are language-specific
The results of NNLPP will offer a new perspective on mental grammars to researchers in the field of multilingualism and multilingual education, psycholinguistics, computational linguistics, and human language processing. Besides, the NNLPP findings can be used in computer modelling by professionals in technology, computer scientists working in the field of natural language processing and artificial intelligence, and designers of language-learning platforms (ex., DuoLingo, Buzuu, ELT-online). The results will provide insights into cognitive processing necessary for professionals in language education: educators, speech pathologists, language policy makers, foreign language teachers and trainee teachers, general learners of foreign languages, parents, students of modern languages and linguistics at different universities. As a scholarly initiative, NNLPP supports the EU efforts to promote linguistic diversity, raise public awareness of multilingualism and inform language-related policies and linguistic education.
NNLPP is an integrated study which compares the strategies multilingual speakers use across their languages. The project will investigate the role of syntactic structure in comprehension and production of unilingual sentences and sentences which switch from one language to another. A series of experiments will test 250-300 adult non-balanced multilinguals whose languages belong to different linguistic families: Norwegian, Russian, and English. Focusing on code-switched sentences, NNLPP will compare the participants’ reading times, eye movements and intonation patterns and provide a well-rounded description of psycholinguistic mechanisms used by a multilingual mind.
The target sentences in NNLPP are code-switched at different places (1) manipulating the length of the code-switched fragment. I assume that code-switching in (1a) and (1b) forces a prosodic break at the places where the language change occurs, and that this prosodic break shapes sentence parsing and interpretation.
(1a) Bill saw the friend of the neighbor [language change: English > Russian] that was talking about football in the yard.
(1b) Bill saw the friend [language change: English > Russian] of the neighbor that was talking about football in the yard.
(1c) Bill saw [language change: English > Russian] the friend of the neighbor that was talking about football in the yard.
(1d). Bill saw the friend of the neighbor that was talking about football [language change: English > Russian] in the yard.
The places of code-switching in (1a) and (1b) follow the Implicit Prosody Hypothesis (Fodor, 2002), and I anticipate (1a) to be interpreted as 'the friend was talking' and (1b) as 'the neighbor was talking'. If this assumption is confirmed in both Norwegian-English and Russian-Norwegian code-switching, it will mean that a certain prosodic structure forced by code-switching is a universal processing cue that informs mental structure building.
The places of code-switching in the target sentences (1c) and (1d) manipulate the lengths of the code-switched fragment. In (1c) it keeps the bigger part of the sentence in one language, e.g. Russian, in (1d) in English. Typically, Russian speakers prefer to interpret sentences like in (1) as 'the friend was talking', and English speakers prefer the interpretation 'the neighbor was talking' (see Fodor, 2002, 1998 for full review). The preferred interpretations in (1 c, d) will show whether the sentence is parsed in a language-specific manner and follows the parsing preference of the language that supplies most of the lexical items. Thus, the participants are expected to prefer the interpretation 'the friend was talking' in (1c) and 'the neighbor was talking' in (1d).
An alternative prediction is also available for sentences (1c) and (1d). The language of the matrix verb provides the matrix language for code-switching (Myers-Scotton & Jake, 2009). Both (1c) and (1d) begin with English and the matrix verb is given in English. Therefore, the participants may show a preference for the interpretation 'the neighbor was talking' in both (3c) and (3d). As the last alternative, the participants may choose an L1-like parse in all the experimental items in (3).
NNLPP will use sentences like 1(a – d) in the processing experiments in language comprehension and production to test two hypotheses:
1. non-native sentence comprehension and production is an integrated process which follows universal parsing strategies (ex., prosodic cues)
2. in the absence of a universal parsing prompt, sentence comprehension and production are language-specific
The results of NNLPP will offer a new perspective on mental grammars to researchers in the field of multilingualism and multilingual education, psycholinguistics, computational linguistics, and human language processing. Besides, the NNLPP findings can be used in computer modelling by professionals in technology, computer scientists working in the field of natural language processing and artificial intelligence, and designers of language-learning platforms (ex., DuoLingo, Buzuu, ELT-online). The results will provide insights into cognitive processing necessary for professionals in language education: educators, speech pathologists, language policy makers, foreign language teachers and trainee teachers, general learners of foreign languages, parents, students of modern languages and linguistics at different universities. As a scholarly initiative, NNLPP supports the EU efforts to promote linguistic diversity, raise public awareness of multilingualism and inform language-related policies and linguistic education.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
The project and my work in it was organized around the following objectives:
1. Conduct quality research on language processing and production and provide a well-rounded description of the linguistic mechanisms underlying them.
This objective was achieved successfully. First, I designed a multi-level research project consisting of 6 experiments: 4 on reading for comprehension (3 self-paced reading and 1 eye-tracking) and 2 on oral production (1 eliciting the intonation patterns in independent sentences and 1 investigating the same sentences in context). I expanded the original proposal and designed more fine-grained experiments eliciting the role of syntax in multilingual processing for comprehension and production. I checked the default structural preference typical for a given language as initially planned. I also added a specific variable – a perception matrix verb – that is a strong syntactic clue for processing, but it works differently in the target languages. This extension allowed me to measure the relative impact of a syntactic clue against the effect of the place of code-switching in both processing for comprehension and production.
As a result of my MSCA project, a detailed description of how a multilingual mind operates in comprehension and production will be provided. Data analysis is largely in progress, but it is already evident that languages integrate smoothly in reading for comprehension. There is little to no increase in reading time when one language changes into another within a sentence. At the same time, a language change is marked with lengthening of the vowels, and a slight intonation change in oral production. One can argue that there is a definitive prosodic contour for the fragments with code-switching. However, the distinctive patterns (preliminarily) attested in the production tasks do not affect the interpretation of the target sentence, they are just physiological accommodation of the language change.
The architecture of the project was intended to address serious methodological questions: 1) a word-by-word presentation in self-paced reading experiments; 2) presented as a full sentence in the reading with eye-tracking task; 3) written on the paper, as part of a text or separately, for oral production. It allows me to compare across experiments and see whether the results are different because of the type of experiment or the mode of data presentation.
2. Develop professionally as a researcher and a professor.
The second objective was also completed successfully. I mastered eye-tracking, and in doing so I completed my profile as a researcher in behavioral psycholinguistic experiments. I also updated and improved my knowledge of Praat, the application for phonological analysis.
I presented at multiple conferences and improved my presentation skills for talks and poster presentations. Poster presentations were particularly challenging given the large volume of data in my project because the presentation should be 2-3 minutes long.
Through participation in the meeting of the ForMAAL research group at NTNU, I had a chance to try my hand at advising younger colleagues on research design and data analysis. It was particularly important to be part of the group whose researchers were at different levels of seniority, as I could see mature and mid-career advisers.
Lastly, I tried my hand at organizing an international scholarly event – a workshop ‘Multilingual processing in psycholinguistics and technology’. I view this event as my first step in creating a platform for processing experiments in multilingualism where language will take central stage, and where psycholinguistic research can motivate advances in technology. I hope to make this event biannual.
1. Conduct quality research on language processing and production and provide a well-rounded description of the linguistic mechanisms underlying them.
This objective was achieved successfully. First, I designed a multi-level research project consisting of 6 experiments: 4 on reading for comprehension (3 self-paced reading and 1 eye-tracking) and 2 on oral production (1 eliciting the intonation patterns in independent sentences and 1 investigating the same sentences in context). I expanded the original proposal and designed more fine-grained experiments eliciting the role of syntax in multilingual processing for comprehension and production. I checked the default structural preference typical for a given language as initially planned. I also added a specific variable – a perception matrix verb – that is a strong syntactic clue for processing, but it works differently in the target languages. This extension allowed me to measure the relative impact of a syntactic clue against the effect of the place of code-switching in both processing for comprehension and production.
As a result of my MSCA project, a detailed description of how a multilingual mind operates in comprehension and production will be provided. Data analysis is largely in progress, but it is already evident that languages integrate smoothly in reading for comprehension. There is little to no increase in reading time when one language changes into another within a sentence. At the same time, a language change is marked with lengthening of the vowels, and a slight intonation change in oral production. One can argue that there is a definitive prosodic contour for the fragments with code-switching. However, the distinctive patterns (preliminarily) attested in the production tasks do not affect the interpretation of the target sentence, they are just physiological accommodation of the language change.
The architecture of the project was intended to address serious methodological questions: 1) a word-by-word presentation in self-paced reading experiments; 2) presented as a full sentence in the reading with eye-tracking task; 3) written on the paper, as part of a text or separately, for oral production. It allows me to compare across experiments and see whether the results are different because of the type of experiment or the mode of data presentation.
2. Develop professionally as a researcher and a professor.
The second objective was also completed successfully. I mastered eye-tracking, and in doing so I completed my profile as a researcher in behavioral psycholinguistic experiments. I also updated and improved my knowledge of Praat, the application for phonological analysis.
I presented at multiple conferences and improved my presentation skills for talks and poster presentations. Poster presentations were particularly challenging given the large volume of data in my project because the presentation should be 2-3 minutes long.
Through participation in the meeting of the ForMAAL research group at NTNU, I had a chance to try my hand at advising younger colleagues on research design and data analysis. It was particularly important to be part of the group whose researchers were at different levels of seniority, as I could see mature and mid-career advisers.
Lastly, I tried my hand at organizing an international scholarly event – a workshop ‘Multilingual processing in psycholinguistics and technology’. I view this event as my first step in creating a platform for processing experiments in multilingualism where language will take central stage, and where psycholinguistic research can motivate advances in technology. I hope to make this event biannual.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
The project is a multi-layered account of how a multilingual mind achieves sentence comprehension in silent reading and how oral production is performed. Its main achievement is in measuring different human reactions to such a salient cue as a language change in the middle of the sentence as well a subtle cue, such as a perception verb (e.g. saw) at the beginning of the sentence. The reactions are compared across the following dimensions: the milliseconds taken to read each word in the target sentences, when only one word is visible at a time; fixation patterns and eye-movements when a full sentence is present on the screen; prosodic patterns in production, specifically syllable duration and shifts in intonation contour. Very preliminary data analysis suggests that slowdowns in reading time and eye-movements are caused by the presence of the verb ‘saw’ in a sentence rather than by a language change (code-switching). This finding has the potential to significantly impact the field of human language processing because it demonstrates that unilingual sentences are sentences where two languages change (sentences with code-switching) are processed in the similar way. To give a brief example, my data show that people make certain anticipations in the course of sentence processing. For example, a certain continuation is anticipated after the verb ‘saw’ – an event needs to be introduced, saw -> what? If this anticipation is not confirmed by the incoming information, the human brain slows down its processing time to deal with the issue. Crucially, if the sentence uses two languages, the brain is not disrupted by the language change; it continues to monitor whether the anticipated event is present, or a totally different sentence is unfolding. Knowing that two languages are integrated smoothly in silent reading for comprehension will affect the entire approach to the use of multiple languages in everyday life, as well as in the classroom. These findings can also benefit the world of technology and machine learning, where the focus is on creating multilingual language models.
Another major scientific contribution of the project lies in the seeming mismatch between the reading and the production data. Unlike the reading comprehension studies, the experiments in oral production demonstrate a change in prosodic pattern caused by the language change (according to very preliminary data analysis). At the same time, the observed acoustic shift does not alter comprehension patterns of the target sentence. Therefore, it could be explained by the physiological need to accommodate certain sounds in the speech production rather than by the difference in application of the deep psycholinguistic mechanisms responsible for multilingual processing. If supported by the further analysis, these findings have a potential to revolutionize the field of human language processing: They demonstrate how minimal behavioural differences observed in a single experiment may not be representative enough to claim psycholinguistic differences in multilingual processing. The latter will set higher standards for behavioural research, thereby enhancing its rigor and effectiveness.
Another major scientific contribution of the project lies in the seeming mismatch between the reading and the production data. Unlike the reading comprehension studies, the experiments in oral production demonstrate a change in prosodic pattern caused by the language change (according to very preliminary data analysis). At the same time, the observed acoustic shift does not alter comprehension patterns of the target sentence. Therefore, it could be explained by the physiological need to accommodate certain sounds in the speech production rather than by the difference in application of the deep psycholinguistic mechanisms responsible for multilingual processing. If supported by the further analysis, these findings have a potential to revolutionize the field of human language processing: They demonstrate how minimal behavioural differences observed in a single experiment may not be representative enough to claim psycholinguistic differences in multilingual processing. The latter will set higher standards for behavioural research, thereby enhancing its rigor and effectiveness.