Visual narratives as a window into language and cognition

Descripción del proyecto

Imágenes y lengua en la comunicación de los humanos

Las secuencias de imágenes siempre se han considerado fundamentales en la comunicación de los humanos. Sin embargo, solo recientemente los académicos han empezado a examinar dichas narrativas visuales, a la vez que reconocen que representan una valiosa fuente para el estudio de la cognición de la creación de significado secuencial. En consecuencia, los académicos han identificado similitudes entre la lengua y las imágenes secuenciales, lo que conducirá a indagaciones sobre las similitudes entre la estructura y el tratamiento de la lengua y las narrativas visuales. El proyecto TINTIN, financiado con fondos europeos, propone abordar la cuestión mediante la ciencia de la lingüística y de la psicología. Dado que no existe ninguna base de datos para sistemas de narrativas visuales, el proyecto creará instrumentos de anotación visual y construirá una colección con 1 500 cómics de todo el mundo. Además, examinará si las narrativas visuales varían en sus propiedades, si dichas variaciones influyen en su comprensión y si están conectadas con los lenguajes verbales.

Objetivo

Drawn sequences of images are a fundamental aspect of human communication, appearing from instruction manuals and educational material to comics. Despite this, only recently have scholars begun to examine these visual narratives, making this an untapped resource to study the cognition of sequential meaning-making. The emerging field analysing this work has implicated similarities between sequential images and language, which raises the question: Just how similar is the structure and processing of visual narratives and language? I propose to explore this query by drawing on interdisciplinary methods from the psychological and linguistic sciences. First, in order to examine the structural properties of visual narratives, we need a large-scale corpus of the type that has benefited language research. Yet, no such databases exist for visual narrative systems. I will thus create innovative visual annotation tools to build a corpus of 1,500 annotated comics from around the world (Stage 1). With such a corpus, I will then ask, do visual narratives differ in their properties around the world, and does such variance influence their comprehension (Stage 2)? Next, we might ask why such variation appears, particularly: might differences between visual narratives be motivated by patterns in spoken languages, thereby implicating cognitive processes across modalities (Stage 3)? Thus, this proposal aims to investigate the domain-specific (Stage 2) and domain-general (Stage 3) properties of visual narratives, particularly in relation to language, by analysing both production (corpus analyses) and comprehension (experimentation). This research will be ground-breaking by challenging our knowledge about the relations between drawing, sequential images, and language. The goal is not simply to create tools to explore a limited set of questions, but to provide resources to jumpstart a budding research field for visual and multimodal communication in the linguistic and cognitive sciences.