Periodic Reporting for period 3 - TINTIN (Visual narratives as a window into language and cognition)
Reporting period: 2023-02-01 to 2024-07-31
Often, people think about graphic representations as being more about what we see than about cultural patterns. Indeed, the visual languages used in pictures are often thought to be “universal” to understand, despite there being recognizable styles associated with graphics that come from different cultures, contexts, or uses. The “TINTIN Project”, officially known as “Visual narratives as a window into language and cognition,” explores this issue of cross-cultural diversity: Are there cross-cultural patterns in the visual languages used in comics of the world? Do those patterns connect to the spoken languages of the comic creators? Do people’s languages or comic reading experience influence how they comprehend comics?
To investigate these questions, the TINTIN Project has created the Multimodal Annotation Software Tool (MAST), a downloadable application that allows for the graphic analysis of visual and multimodal documents. We are using MAST to create a corpus of annotated comics from roughly 100 countries from around the world, many coming from generous donations from over 70 comic creators and companies. We are analyzing extensive details about the structure of the graphics and visual storytelling of comics. With this data we hope to discover the degree to which cross-cultural patterns persist across comics, whether those patterns are connected to properties of spoken languages, and whether those patterns affect the way people comprehend comics. Along with this analysis, we will then conduct psychology experiments to see if experience with these cross-cultural patterns and/or people’s spoken languages influence the ways they comprehend comics.
Altogether, the TINTIN Project aims to discover the extent and limits of cross-cultural patterns in graphic communication, dispelling myths about the universality of pictures and demonstrating how visual languages use the same principles as our other languages. In addition, this work can reveal how these different linguistic systems of the verbal and visual might tap into common structures stemming from the ways our minds organize information more generally.
To investigate these questions, the TINTIN Project has created the Multimodal Annotation Software Tool (MAST), a downloadable application that allows for the graphic analysis of visual and multimodal documents. We are using MAST to create a corpus of annotated comics from roughly 100 countries from around the world, many coming from generous donations from over 70 comic creators and companies. We are analyzing extensive details about the structure of the graphics and visual storytelling of comics. With this data we hope to discover the degree to which cross-cultural patterns persist across comics, whether those patterns are connected to properties of spoken languages, and whether those patterns affect the way people comprehend comics. Along with this analysis, we will then conduct psychology experiments to see if experience with these cross-cultural patterns and/or people’s spoken languages influence the ways they comprehend comics.
Altogether, the TINTIN Project aims to discover the extent and limits of cross-cultural patterns in graphic communication, dispelling myths about the universality of pictures and demonstrating how visual languages use the same principles as our other languages. In addition, this work can reveal how these different linguistic systems of the verbal and visual might tap into common structures stemming from the ways our minds organize information more generally.
Through the first half of the TINTIN Project, we have created an innovative computer program for analyzing the structure of visual and multimodal media: the Multimodal Annotation Software Tool (MAST). With it, we have analyzed the properties of 450 comics across 40 countries, on our way to analyze works from almost 100 countries that will constitute our final dataset. This combines with our finalization of a prior corpus which analyzed over 300 comics from 9 countries.
Analysis of data so far has shown that comics from Europe, Asia, and America differ in the length of continuous stretches of consistent time, space, and characters in their visual storytelling, suggesting that understanding these different types of comics warrants culturally specific fluencies. In addition, we have shown that motion events in comics are influenced by the ways different languages encode motion, showing an interaction between the structures of spoken and visual languages. These types of findings exemplify the intertwining aspects of cultural patterns, language, and cognition explored by the TINTIN Project.
Analysis of data so far has shown that comics from Europe, Asia, and America differ in the length of continuous stretches of consistent time, space, and characters in their visual storytelling, suggesting that understanding these different types of comics warrants culturally specific fluencies. In addition, we have shown that motion events in comics are influenced by the ways different languages encode motion, showing an interaction between the structures of spoken and visual languages. These types of findings exemplify the intertwining aspects of cultural patterns, language, and cognition explored by the TINTIN Project.
The TINTIN Project will provide innovative analysis tools and data that will greatly progress our understanding of graphic communication, language, comics, and the general properties of how the mind organizes information. Our Multimodal Annotation Software Tool (MAST) provides a cutting-edge system for researchers as individuals or teams to annotate the properties of visual and multimodal media. This software facilitates the sharing of documents, projects, theories, and data, and can hopefully provide a boon to the growing efforts for empirical research on multimodal media.
Within the TINTIN Project, MAST has provided us with an innovative tool to gather extensive data about the cross-cultural diversity of comics around the world. These corpus data constitute the first cross-cultural datasets to investigate the structural properties of the visual languages of comics, providing a unique resource for further understanding the properties of visual storytelling around the world.
Following from our preliminary findings, we expect to find three types of results persisting through our analysis of comics around the world. First, we predict to find consistent cross-cultural patterns which reflect different visual languages of the world, such as corresponding to the Japanese Visual Language originating from manga in Japan. Second, we predict to find trends that transcend cross-cultural variation and are consistent with those in spoken languages, such trade-offs in the complexity of units and sequences, or trade-offs in the lengths of sequences and the frequency that they occur. These “universal” tendencies would reflect something broader about how the mind organizes information. Finally, we posit that certain ways that spoken languages pattern information might “permeate” into the structures used by the visual languages in comics. This would show that different expressive systems may interact with each other.
Altogether, these results would show that the visual languages used in comics display more complexity than is typically believed, and such richness can be used as a window to better understand language and cognition.
Within the TINTIN Project, MAST has provided us with an innovative tool to gather extensive data about the cross-cultural diversity of comics around the world. These corpus data constitute the first cross-cultural datasets to investigate the structural properties of the visual languages of comics, providing a unique resource for further understanding the properties of visual storytelling around the world.
Following from our preliminary findings, we expect to find three types of results persisting through our analysis of comics around the world. First, we predict to find consistent cross-cultural patterns which reflect different visual languages of the world, such as corresponding to the Japanese Visual Language originating from manga in Japan. Second, we predict to find trends that transcend cross-cultural variation and are consistent with those in spoken languages, such trade-offs in the complexity of units and sequences, or trade-offs in the lengths of sequences and the frequency that they occur. These “universal” tendencies would reflect something broader about how the mind organizes information. Finally, we posit that certain ways that spoken languages pattern information might “permeate” into the structures used by the visual languages in comics. This would show that different expressive systems may interact with each other.
Altogether, these results would show that the visual languages used in comics display more complexity than is typically believed, and such richness can be used as a window to better understand language and cognition.