Periodic Reporting for period 4 - TINTIN (Visual narratives as a window into language and cognition)
Okres sprawozdawczy: 2024-08-01 do 2025-07-31
Often, people think about pictures as reflecting what people see, rather than about patterns. Indeed, pictures are often thought to be “universal” to understand, despite there being recognizable styles associated with graphics that come from different cultures, contexts, or uses. The TINTIN Project, officially known as “Visual narratives as a window into language and cognition,” explores this issue of diversity: Are there patterns in the visual languages used in comics of the world, and do they vary based on cultures, styles, or the spoken languages of the comic creators? Are there regularities that span across all comics and/or languages? Do people’s languages or comic reading experience influence how they comprehend comics?
To investigate these questions, the TINTIN Project has created the Multimodal Annotation Software Tool (MAST) that allows for the graphic analysis of visual and multimodal documents. Using MAST, we created the TINTIN Corpus, a database of 1,030 annotated comics from 144 countries spanning 55 languages with over 1 million annotations. Many of these comics came from generous donations from over 80 comic creators and companies around the world.
With this data, we are able to analyze details about the structure of the graphics and storytelling of comics, giving us insight into the patterns that persist across comics. We have shown that these patterns indeed extend beyond cultural boundaries and cluster into distinctive “visual languages.” At the same time, aspects of languages permeate these graphic systems, such as the directionality of writing systems affecting comics layouts, or the word order of languages aligning with variation in visual narrative constructions. Yet, despite this variation we also observe regularities that persist across all types of comics, representing statistical regularities consistent with universal “linguistic laws” found across languages.
Altogether, the TINTIN Project has uncovered systematic regularities that persist across patterns in graphic communication, dispelling myths about the universality of pictures and demonstrating how visual languages use the same principles as other languages. Such results reveal how different linguistic systems of the verbal and visual might tap into common structures stemming from the ways our minds organize information more generally.
To investigate these questions, the TINTIN Project has created the Multimodal Annotation Software Tool (MAST) that allows for the graphic analysis of visual and multimodal documents. Using MAST, we created the TINTIN Corpus, a database of 1,030 annotated comics from 144 countries spanning 55 languages with over 1 million annotations. Many of these comics came from generous donations from over 80 comic creators and companies around the world.
With this data, we are able to analyze details about the structure of the graphics and storytelling of comics, giving us insight into the patterns that persist across comics. We have shown that these patterns indeed extend beyond cultural boundaries and cluster into distinctive “visual languages.” At the same time, aspects of languages permeate these graphic systems, such as the directionality of writing systems affecting comics layouts, or the word order of languages aligning with variation in visual narrative constructions. Yet, despite this variation we also observe regularities that persist across all types of comics, representing statistical regularities consistent with universal “linguistic laws” found across languages.
Altogether, the TINTIN Project has uncovered systematic regularities that persist across patterns in graphic communication, dispelling myths about the universality of pictures and demonstrating how visual languages use the same principles as other languages. Such results reveal how different linguistic systems of the verbal and visual might tap into common structures stemming from the ways our minds organize information more generally.
The TINTIN Project began with the creation of an innovative computer program for analyzing the structure of visual and multimodal media: the Multimodal Annotation Software Tool (MAST). With it, we have analyzed the properties of 1,030 comics across 144 countries in the TINTIN Corpus, combining with the finalization of a previous Visual Language Research Corpus of 360 comics from 9 countries. These corpora provide unique resources for studying the structure of graphic systems, combined with a book describing our rich annotation procedures so that others can follow our lead.
Analysis of these corpora have revealed that systematic regularities persist across graphic systems that transcend cultural boundaries, implying variation across different visual languages. These systems maintain distinctive properties of their styles, graphic conventions, layouts, visual storytelling, and other properties. These structures of graphic systems appear to further be aligned in subtle ways with structures of spoken languages of their creators, while also demonstrating statistical regularities common to the universals that span across languages.
While analysis of corpus data has constituted the primary focus of the TINTIN Project, it has been complemented by theory development and psychological experimentation. Theoretical work has provided careful grounding of the principles of sequential image comprehension and motion event understanding. Meanwhile, our psychological experiments have revealed the processes that comprehenders use in understanding discontinuities of visual storytelling (like depictions of imagination), and in the interpretations of motion in different visual cues. These psychological effects are further modulated by participants’ frequency of reading comics, and sometimes the specific types of comics they read, reinforcing the necessity of proficiency in visual languages is essential for interpreting graphics.
These findings have been shared across 16 publications (with more under review), 12 conferences, and over 15 invited talks. We have further promoted our work directly to an interested public audience with popular press articles, videos, and a booth at the Dutch Comic-Con, while have sponsored community-building of our academic field by hosting the Visual Language Conference 2024 at Tilburg University.
Altogether, the TINTIN Project has revealed insights into the fundamental building blocks of graphic systems and their interpretation, while also providing resources for an emerging field examining visual languages with the same tools as we study spoken and signed languages.
Analysis of these corpora have revealed that systematic regularities persist across graphic systems that transcend cultural boundaries, implying variation across different visual languages. These systems maintain distinctive properties of their styles, graphic conventions, layouts, visual storytelling, and other properties. These structures of graphic systems appear to further be aligned in subtle ways with structures of spoken languages of their creators, while also demonstrating statistical regularities common to the universals that span across languages.
While analysis of corpus data has constituted the primary focus of the TINTIN Project, it has been complemented by theory development and psychological experimentation. Theoretical work has provided careful grounding of the principles of sequential image comprehension and motion event understanding. Meanwhile, our psychological experiments have revealed the processes that comprehenders use in understanding discontinuities of visual storytelling (like depictions of imagination), and in the interpretations of motion in different visual cues. These psychological effects are further modulated by participants’ frequency of reading comics, and sometimes the specific types of comics they read, reinforcing the necessity of proficiency in visual languages is essential for interpreting graphics.
These findings have been shared across 16 publications (with more under review), 12 conferences, and over 15 invited talks. We have further promoted our work directly to an interested public audience with popular press articles, videos, and a booth at the Dutch Comic-Con, while have sponsored community-building of our academic field by hosting the Visual Language Conference 2024 at Tilburg University.
Altogether, the TINTIN Project has revealed insights into the fundamental building blocks of graphic systems and their interpretation, while also providing resources for an emerging field examining visual languages with the same tools as we study spoken and signed languages.
The TINTIN Project has provided innovative analysis tools and annotation data that greatly progresses our understanding of graphic communication, language, comics, and the general properties of how the mind organizes information. Our Multimodal Annotation Software Tool (MAST) provides a cutting-edge system for researchers as individuals or teams to annotate the properties of visual and multimodal media. This software facilitates the sharing of documents, projects, theories, and data, and can hopefully provide a boon to the growing efforts for empirical research on multimodal media.
We have used MAST to create two datasets detailing the structural properties of comics from around the world: the Visual Language Research Corpus (VLRC, 360 comics, 9 countries, 48K panels) and the TINTIN Corpus (1,030 comics, 144 countries, 76K panels). These corpora constitute the first global datasets to investigate the structural properties of the visual languages of comics, providing a unique resource for further understanding the properties of visual storytelling around the world.
Our analyses of these data have so far revealed patterns in graphic systems that transcend cultural boundaries to reflect “visual languages.” In many analyses, we also find that variation in how spoken languages pattern information “permeate” the structures used by the visual languages in comics, showing alignment in their properties suggesting that different expressive systems may interact with each other in cognition. Yet, across these visual languages we find statistical regularities consistent with universal “linguistic laws” found in spoken languages, implying how the mind organizes information transcends expressive modalities.
Altogether, these results show that the visual languages used in comics display more complexity than is typically believed, and such richness can be used as a window to better understand language and cognition.
We have used MAST to create two datasets detailing the structural properties of comics from around the world: the Visual Language Research Corpus (VLRC, 360 comics, 9 countries, 48K panels) and the TINTIN Corpus (1,030 comics, 144 countries, 76K panels). These corpora constitute the first global datasets to investigate the structural properties of the visual languages of comics, providing a unique resource for further understanding the properties of visual storytelling around the world.
Our analyses of these data have so far revealed patterns in graphic systems that transcend cultural boundaries to reflect “visual languages.” In many analyses, we also find that variation in how spoken languages pattern information “permeate” the structures used by the visual languages in comics, showing alignment in their properties suggesting that different expressive systems may interact with each other in cognition. Yet, across these visual languages we find statistical regularities consistent with universal “linguistic laws” found in spoken languages, implying how the mind organizes information transcends expressive modalities.
Altogether, these results show that the visual languages used in comics display more complexity than is typically believed, and such richness can be used as a window to better understand language and cognition.