Often, people think about pictures as reflecting what people see, rather than about patterns. Indeed, pictures are often thought to be “universal” to understand, despite there being recognizable styles associated with graphics that come from different cultures, contexts, or uses. The TINTIN Project, officially known as “Visual narratives as a window into language and cognition,” explores this issue of diversity: Are there patterns in the visual languages used in comics of the world, and do they vary based on cultures, styles, or the spoken languages of the comic creators? Are there regularities that span across all comics and/or languages? Do people’s languages or comic reading experience influence how they comprehend comics?
To investigate these questions, the TINTIN Project has created the Multimodal Annotation Software Tool (MAST) that allows for the graphic analysis of visual and multimodal documents. Using MAST, we created the TINTIN Corpus, a database of 1,030 annotated comics from 144 countries spanning 55 languages with over 1 million annotations. Many of these comics came from generous donations from over 80 comic creators and companies around the world.
With this data, we are able to analyze details about the structure of the graphics and storytelling of comics, giving us insight into the patterns that persist across comics. We have shown that these patterns indeed extend beyond cultural boundaries and cluster into distinctive “visual languages.” At the same time, aspects of languages permeate these graphic systems, such as the directionality of writing systems affecting comics layouts, or the word order of languages aligning with variation in visual narrative constructions. Yet, despite this variation we also observe regularities that persist across all types of comics, representing statistical regularities consistent with universal “linguistic laws” found across languages.
Altogether, the TINTIN Project has uncovered systematic regularities that persist across patterns in graphic communication, dispelling myths about the universality of pictures and demonstrating how visual languages use the same principles as other languages. Such results reveal how different linguistic systems of the verbal and visual might tap into common structures stemming from the ways our minds organize information more generally.
