Objective
When humans process language, they rarely do so in isolation. Linguistic input often occurs synchronously with visual input, e.g. in everyday activities such as attending a lecture or following directions on a map. The visual context constrains the interpretation of the linguistic input, and vice versa, making processing more efficient and less ambiguous. Given the ubiquity of synchronous linguistic and visual processing, it is surprising that there is only a sparse experimental literature that deals with this topic, while virtually no computational models exist that capture the synchronous interpretation process. We propose an experimental research program that will investigate key features of synchronous processing by tracking participants' eye movements when they view a naturalistic scene and listen to a speech stimulus at the same time. The aim is to understand synchronous processing better by studying the interaction of saliency and ambiguity, and the role of incrementality, object context, and task factors. These experimental results will feed into a series of computational models that predict the eye-movement patterns that humans exhibit when they view a scene and listen to speech at the same time. The key modeling idea is to treat synchronous processing as an alignment problem, for which a rich literature exists in computational linguistics. Building on this literature, we will develop models that incrementally construct aligned linguistic and visual representations, and that can be evaluated against eye-tracking data.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
- humanities languages and literature linguistics
- social sciences media and communications graphic design
You need to log in or register to use this function
We are sorry... an unexpected error occurred during execution.
You need to be authenticated. Your session might have expired.
Thank you for your feedback. You will soon receive an email to confirm the submission. If you have selected to be notified about the reporting status, you will also be contacted when the reporting status will change.
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
ERC-2007-StG
See other projects for this call
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Host institution
EH8 9YL Edinburgh
United Kingdom
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.