Playing by feeling: an investigation of the role of haptic cues in the perceived quality of a musical instrument

Project Information

HAPTIFEEL

Grant agreement ID: 659645

DOI 10.3030/659645

Project terminated on 4 October 2016

EC signature date 27 March 2015

Start date 1 March 2016

End date 28 February 2017

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost € 84 138,60

EU contribution € 84 138,60

Coordinated by

UNIVERSITA DEGLI STUDI DI PADOVA
Italy

Periodic Reporting for period 1 - HAPTIFEEL (Playing by feeling: an investigation of the role of haptic cues in the perceived quality of a musical instrument)

Reporting period: 2016-03-01 to 2017-02-28

Summary of the context and overall objectives of the project

The assessment of the quality of a musical instrument, both from a physical and a perceptual point of view, has been the object of much research in the past and in more recent years. These studies have tried to associate peculiarities in the instrument's mechanical behavior (i.e. its vibrations) to its sound properties, and to how these are perceived by a performer or audience.
A key aspect in the interaction with a musical instrument that is increasingly being investigated is the haptic feedback that results from the mechanical interaction between performer and instrument. Haptic perception is an umbrella term that encompasses both the tactile and proprioceptive sense, and it is known to be a crucial element in expert music performance.
Even though haptic perception plays such a critical role in musical interaction, relatively little effort has been invested so far into characterizing the haptic behavior of a musical instrument and into how this affects musicians' perception.
The goal of HAPTIFEEL is to develop a methodological framework to characterize the haptic behavior of a piano keyboard, and later of several traditional musical instruments as perceived by a player. Our methodology will be based on several perceptually-informed physical measurements together with user-based studies which will make use of tactile-augmented digital musical interfaces.

There are several points of novelty:
- building of a comprehensive database of tactile cues recorded on several piano keyboards of different qualities. The perceptual effect of these cues can be simultaneously assessed in user-based tests. This can provide, for the first time, a perceptual projection of the mechanical behavior of the instrument.
- The specific effect of haptic feedback on the perceived properties of an instrument has been speculated at length, but so far uncovered only to a limited extent. This multisensory centered methodology can significantly advance the understanding of the role of haptics in musical interaction.
- A tactile-augmented keyboard can be developed based on our results. This would be the first-of-a-kind device, allowing dynamic tactile feedback rendering to simulate the behavior and quality of several piano keyboards.
- Our findings can enable innovation in the context of haptic and specifically tactile rendering for augmented musical interfaces. Research and instrument builders will be able to design haptic feedback so to achieve their desired qualities in the behavior of the instrument as perceived by the player.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The work performed during the 2.3 months of project activity (March 1st to May 8th, 2016) was devoted to the start-up of the project and can be summarized in two main points:
A) Development of an experimental set-up: a digital piano augmented with vibrotactile actuators.
B) Planning of subjective experiments with pianists, to be performed both on real and on digital piano

Concerning point A), the developed prototype is as follows (see also attached pictures).

The keyboard of a digital piano was detached from its metal casing, containing also the electric and electronic hardware, and then screwed to a thick plywood board.
Two Tactile Transducers were attached to the bottom of the wooden board, respectively in correspondence of the lower and middle octaves, in this way enabling to convey vibrations at the most relevant areas of the keyboard 2. The transducers were driven by signals provided by an audio interface communicating with an computer.
Synthetic sound and vibrotactile feedback were generated via software. The piano synthesizer was configured to match the sound of a Yamaha Disklavier grand piano available in our labs, and previously used for recording vibration samples.
The computer was also used to conduct the tests and collect experimental data.

Concerning point B), the planned experiments are concern.

- Multifinger interaction:
How is vibrotactile sensitivity enhanced by multi-finger stimulation?
To what extent is one sensitive to tactile consonance delivered at different fingers?
Short-term tactile memory vs. spatial integration: which one prevails?

- Bimodally perceived quality
Is there bone conduction that really affects audition?
Having ruled out bone conduction, do vibrations really affect the perceived sound?
Do pianists rate the perceived quality differently depending on vibrations?

- Performance and long-term effects
Do vibrations affect performance (e.g. help in performing a given task or passage)?
Do vibrations affect the sense of fatigue after a long (1 hour?) traning session?
Do vibrations affect ease/speed of learning in the mid- or long-term (weeks or months)?

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project underwent an early termination. The termination date was 8th May 2016.
The project start was extremely positive, however shortly after the start the Experienced Researcher received an offer for a permanent position as Haptics Engineer in a high-tech start-up company based in the United Kingdom. Although it was a very hard decision to take he finally resolved to accept this offer and therefore resigned from the MC Individual Fellowship.

This shows that the project topic, haptic interaction, is extremely relevant for EU research and applications in the field of haptic interaction. The reintegration process of the Experienced Researcher into the EU Research Community has been extremely successful, as he now has a permanent R&D position in the EU area.
On the other hand, such an early project termination implies that activities have not developed enough to produce advances in the state of the art, or socio-economic impact. The project activities are currently continuing at a slower pace with local funding and human resources, and further funding applications are being prepared.

Detail of the vibroactuator mounting at the bottom of the digital piano

Prototype of a digital piano augmented with vibrotactile actuators

Lab setup for measurements of sound and vibrations from a grand piano