Community Research and Development Information Service - CORDIS


DANCE Report Summary

Project ID: 645553
Funded under: H2020-EU.2.1.1.

Periodic Reporting for period 1 - DANCE (Dancing in the Dark)

Reporting period: 2015-01-01 to 2016-06-30

Summary of the context and overall objectives of the project

Close your eyes to see: the ancient mystical suggestion of Persian poetry intends to find images that open to sensations coming from the least explored places in the brain, exploring the immeasurable spaces of the inner dreams. Close your eyes to see: at the core of DANCE the historical humanistic idea becomes the scientific and technological challenge of experimenting the translatability of visible quality through different sensory configurations.
DANCE is engaged in deeply understanding how (and how much) the affective and relational quality of movement (generally perceived only by view) can be represented, analysed and understood within an audible space (of sound or music). DANCE aims to understand what does it mean, closing the eyes, the expressiveness of a gesture (specifically in the symbolic, affective and hyper-semantic context of a dance choreography), the emotions conveyed by a series of movements in space, the nonverbal language used by bodies to communicate affects and feelings. The two main strictly interlaced objectives of the project – better understanding the qualitative dimension of movement and elaborating strategies of sensorial substitution (from visual into auditory channels) – could have a strong impact on society. Questioning theoretically and experiment in concrete artistic situation effective paradigms for listening to a choreography, feeling a ballet without seeing it (or deepening the expressive effect of what we see through the impressions of what we hear) pursue not only the obvious direction of a best inclusion for subjects with sensorial diminutions but also the possibility of a stimulating re-activation of the general perceptual sensibility (faded by an excessive multimodal exposition to data with a consequent loss of capacity of concentration and comprehension) via translation of qualitative contents from a specific ‘correct’ sensorial experience into another one less adequate in se and furthermore stimulating imagination and activity of the subjects.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The understanding of the full-body expressive movement is a prerequisite for creation of interactive sonification systems, i.e. systems in which one can hear and perceive the movements of the other. Thus, during the first half of the Project, significant effort was dedicated to study full-body expressive movements. Our work can be divided into several steps. First, we defined Multi-Layered Computational Framework of Expressive Qualities, which became a ground for consecutive systematic studies on expressive movement. In particular, we dedicated a lot of effort to build a multimodal dataset containing short choreographies performed with different expressive qualities, e.g., fluid, fragmented, or impulsive movements. The dataset contains multimodal data (motion capture, wearable sensors, audio, video) of several professional dancers. We also developed a software platform to manage recording, synchronization and playback of the multimodal data. Next, we proposed several algorithms to compute different expressive qualities of the dancers’ movements.
The DANCE Project research results were published in international journals, conferences and workshops. At least 18 publications were submitted or published so far. The papers were presented on biggest conferences in human-computer interactions such as CHI 2016, as well as on related to dance such as International Symposium on Movement & Computing (MOCO 2016). Additionally, the Dance Partners co-organized some scientific (e.g., SIG meeting at CHI 2016, EyesWeb Week in Genoa) and wide-audience events (e.g., An Unveiled Dance Worshop at Genoa, Genoa Science Festival) to promote the results of the Project. The DANCE Project participated in several cultural events. Among, others, in collaboration with the Goethe Institut of Genoa, the contemporary dance performance “Blind Sight” by Sagi Gross, devoted to the impact that technological and social developments have on human relationships, was presented at Genoa. The same artist in collaboration with the blind pianist Bert van der Brink proposed the performance titled Embodies at Maastricht Jazz Festival. It exploits technology developed in DANCE Project: wearable sensors were used to capture the dancer movements, expressive movement detection algorithms were used to recognize the dancer’s movement quality, which then was sonified using sonification models. . In the performance, the dancer dances his choreography with sensors sensors attached to his limbs, and the pianist hears the sonification of the dancer’s movements through the headphones. The audience can hear the pianist improvising to the sonification of the dancer movements without seeing him.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

DANCE moves on three different but strongly integrated and interacting dimensions:
1. Scientific and technological research. The first result is a fundamental conceptual framework for the analysis of expressive qualities of movement, modeling an observer of a dance performance. This conceptual framework is made of four layers, ranging from the physical signals that sensors capture to the qualities that movement communicate (e.g., in terms of emotions). The framework aims to provide a conceptual background the development of computational systems can build upon, with a particular reference to systems analyzing a vocabulary of expressive movement qualities, in order to elaborate and to experiment different theoretical paradigms and applicative models of translation of these visual qualities to the auditory modality. Contextually, the conceptual frame and the prototypes of sonifications based on it, are neuro-physiologically tested contributing to new approaches devoted to understand the complex relationship between perception and cognition (the investigation of the plasticity of the brain in the form of sensory substitution).
2. Inclusion and rehabilitation. The sonification of the main qualities of movement based on the conceptual frame generates new way of experiencing space and movement even without the use of the sight. The automatic systems (seeing the qualities of the movement and translating them in sounds) enable their users to listen to a choreography or to feel, by hearing, a ballet. This kind of cross-modal mediated experience implying the possibility of conveying a different, deeper awareness of the (affective and social) meaning of movement for people with diminished visual ability and to share this embodied knowledge of the relation between spaces and emotions with other people (with or without loss of visual capacity).
3. Artistic production. The sound, instead of preceding or inspiring movement (as in the classical conception of European choreography), is created by the movement itself (by its qualitative characteristics, its intrinsic affective and relational meanings). In the active collaboration with the choreographer Virgilio Sieni, director of Biennale Dance Festival, we are programming international artistic events exploring the possibility of creating, via DANCE paradigms of sensorial substitution, aesthetic and socially inclusive community based on the aware sharing of emotions and feeling embodied in archetypes of gestures. The gestures, transformed from an aesthetic object, into a perceptive sound object, become a social and affective catalyst.
These complex perspectives (social, scientific and artistic) make DANCE a project with a strong cultural vocation. The constant dialogue among scientist, artists, musicians, choreographers who have reflected - theoretically and practically - on the major issues of the relationship between space, movement and emotion, between listening and vision, is opening further research perspectives demonstrating the importance of cross-fertilizing humanistic and technological methodologies.

Related information

Record Number: 191384 / Last updated on: 2016-11-16