Periodic Reporting for period 1 - Safe and Sound Drive (Design of a serious game for cars to help increase driver skills and lower fuel consumption)
Reporting period: 2015-09-01 to 2016-08-31
Summary of the context and overall objectives of the project
New in-vehicle information systems and advanced driver assistance systems are currently being introduced in cars in order to help driving safely and energy efficiently. These systems help the driver to pay attention to critical events (e.g. collision warnings and blind spot information systems) or deliver important information (e.g. speed limits and navigation instructions). However, safety and environmental impact are most of all dependent on the behaviour and attitude of the driver. Mobile apps and games have been suggested as means to affect behaviour and attitudes, and there are already some mobile applications available for Android and Apple platforms, such as FuelLog, EcoDrive, Efficiency, and GoDriveGreen. However, none of these are designed for interaction during driving. Sound based interaction is a way to facilitate continuous interaction with the driver during driving, without preventing the driver from keeping the gaze on the road. The Safe and Sound Drive project concerned the design of an audio-only serious game for cars that helps drivers to increase eco-driving skills and encourage safe and environmentally friendly approaches to driving. The design process was influenced by industrial design methodology and user-centred agile methods. Game engine technology and audio middleware were used for rapid prototyping. Sounds were made interactive by actively controlling them based on driving behaviour, and were designed to help and encourage a stable and suitable speed. User feedback was collected at an early stage through contextual enquiry sessions during real driving. Later, the system was showcased at two science festivals, where over 400 attendees test drove a game simulator. The agile design process provided continuous feedback to the project team, and guided the design of a version of the game that was evaluated in a simulator experiment. The results from the user studies suggest that opinions about beeps and audio signals vary among subjects whereas music and podcast based contents were generally better received. Alteration of media content by actively adjusting spectral balance or music mix formed working mechanisms for providing cues easily understood by participants. However, the effects on energy efficient behaviour were moderate, suggesting that visual information dominated the determination of the participants’ behaviour in the experiment. The project demonstrated that continuous interactive sounds can be used for influencing driver behaviour. It was further shown that sounds can be designed to be intuitively understood and easily explained. Even though the participants found the game interesting and fun, there was a reluctance to use this kind of media based interactive sounds during real driving. This highlights one of the major challenges for designing the future interactive sounds in cars – to design sounds and interfaces that are appealing and desired by future drivers.
Although the demonstrated effects were moderate and the concepts developed in the project generally were not preferred over only visual feedback, the results provide valuable knowledge for further development of interactive sounds for user interfaces in cars in general, and for eco-coaching in particular. The ideas will be further explored in future studies of interactive sounds, and a study of interactive sounds for eco-coaching based on active noise control in trucks is already planned in cooperation between Scania, Luleå University of Technology and Interactive Institute Swedish ICT, financed by the Swedish Energy Agency. Software developed in the Safe and Sound Drive project will be used for further development and evaluation of sound based and multisensory user interfaces in vehicles. Contacts between Luleå University of Technology and Anglia Ruskin University are well established, and further collaboration is planned.
The results were disseminated through participation in conferences, publication of conference papers, demonstrations at two science festivals (Cambridge Festival of Ideas 2015 and Cambridge Science Festival 2016), a demonstration at school event (My Smarter Essex, Chelmsford 2016), workshops with Volvo Car Corporation and discussions with Scania trucks. Outcomes of the project have also been disseminated through regular blog, Twitter and Facebook posts. The demonstration software can and will be used for future demonstrations at science festivals, science centres and other popular science events. Parts of the methods developed in the project are already implemented in Volvo Car Corporation’s design procedures for interactive sounds and will be implemented at Scania through the planned continuation project.
Although the demonstrated effects were moderate and the concepts developed in the project generally were not preferred over only visual feedback, the results provide valuable knowledge for further development of interactive sounds for user interfaces in cars in general, and for eco-coaching in particular. The ideas will be further explored in future studies of interactive sounds, and a study of interactive sounds for eco-coaching based on active noise control in trucks is already planned in cooperation between Scania, Luleå University of Technology and Interactive Institute Swedish ICT, financed by the Swedish Energy Agency. Software developed in the Safe and Sound Drive project will be used for further development and evaluation of sound based and multisensory user interfaces in vehicles. Contacts between Luleå University of Technology and Anglia Ruskin University are well established, and further collaboration is planned.
The results were disseminated through participation in conferences, publication of conference papers, demonstrations at two science festivals (Cambridge Festival of Ideas 2015 and Cambridge Science Festival 2016), a demonstration at school event (My Smarter Essex, Chelmsford 2016), workshops with Volvo Car Corporation and discussions with Scania trucks. Outcomes of the project have also been disseminated through regular blog, Twitter and Facebook posts. The demonstration software can and will be used for future demonstrations at science festivals, science centres and other popular science events. Parts of the methods developed in the project are already implemented in Volvo Car Corporation’s design procedures for interactive sounds and will be implemented at Scania through the planned continuation project.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
Safe and Sound Drive was a Marie Sklodowska Curie Individual Fellowship for Dr Arne Nykänen. The project was carried out at Anglia Ruskin University in Cambridge, UK, during the period 1 September 2015 to 31 August 2016. The aim of the project was to study how serious gaming based on sonic interaction could be used for increasing safety and lowering energy consumption in cars. The project consisted of a training program for Dr Arne Nykänen, a research program developing and assessing sound based driver assistance systems for eco-coaching, and a dissemination and communication program for spreading knowledge about design of interactive sounds to companies and organisations working with vehicles and informing the public about the potential of reducing energy consumption and environmental impact by changing driving behaviour. The objectives were:
Objective 1: Design a demonstrator of a serious game supporting the driver to increase driver skills, lower fuel consumption and allowing safe and environmentally friendly competition with friends.
Objective 2: Develop and release a mobile application version of the game for iOS, available for download from App Store (this will at a later stage be implemented also for Android and Windows).
Objective 3: Measure how the designed game affects fuel consumption, speed and safety.
Objective 4: Training in computer gaming art and technology.
Objective 5: Training in creative use of sound design.
Objective 6: Digital dissemination training – maximising impact of research.
Objective 7: Training in mobile application programming and technology.
Objective 8: Training in commercialisation of mobile applications.
Objective 9: Public dissemination by demonstrations at science centres in the UK and Sweden, and presentations and demonstrations at festivals and events hosted by Anglia Ruskin University and its networks.
The project followed a user-centred research through design scheme. A demonstrator of an audio-only serious game supporting the driver to lower fuel consumption was design. Volvo Car Corporation supported the project with access to a car, full access to sensors and systems in the car. The goal was to develop a mobile applications version of the game for iOS. Because of practical reasons, mainly Nykänen’s previous knowledge in development for the Windows and Android platforms, and because of new knowledge in game engine technology and audio middleware for games gained during the project, the demonstrator was developed using Unreal Engine. The demonstrator was implemented for Windows and demonstrated in a driving simulator and in a car. Future work includes implementation for Android and Apple platforms for public download.
User-Centred Agile Methods
Design of user interfaces is a multidisciplinary task, requiring deep knowledge in psychophysics (or in this particular case, psychoacoustics) and psychology, as well as methods and procedures for design and product development. Knowledge from psychoacoustics and psychology aids the development process, and enables early decisions on potentially successful design routes, avoiding known pitfalls. Examples in the Safe and Sound Drive case are the choice of modality (hearing over vision), choice of frequency content of the sounds to avoid masking, spatial positioning of sound sources to help the driver to keep attention on the road, awareness of the risk of cognitively overloading the driver with a too complex task, etc. However, the design options are still vast, and knowledge is built along the path of the project. A good final design is crucially dependent on how this knowledge is built and taken care of all the way through. A user-centred agile approach was used. In particular, agile game development was explored, as game design is a field where both graphics and sounds are essential parts of the product. The core idea is to organise the work into short iterations (sprints), listening to the evolving product frequently and repeatedly throughout the design process, and get users into the loop early on. Game engine technology was used for rapid prototyping of concepts.
The Sound Design and Research Process
A user-centred agile method was applied to the Safe and Sound Drive project, including rapid and numerous iterations, and supported by the use of a large number of sound sketches. The process was inspired by Scrum, and major elements from the Scrum framework were used, for example: the division of work into time-boxed sprints; User Stories, Product Backlog and Sprint backlogs; prioritisation of backlog items; and the general aim for always delivering a working result from a sprint. Repeatedly during process, qualitative studies and analyses were performed based on diary notes by the participating designers, Contextual Inquiry Interviews, and observation of participants. This approach increases the chances of success in identifying a design that significantly affects fuel consumption and driver behaviour. Finally, the design was evaluated in a simulator study where driver behaviour was measured.
In the start-up/release planning meeting a number of user stories were determined. In agile methods, user stories are typically one sentence descriptions of what a user wants to do with the product, written on the form: “As [kind of user] I would like [a feature] so that I can [reach a goal]”. The following user stories were defined in the start-up/release planning phase:
(i) As a driver I would like to hear when the fuel consumption is low.
(ii) As a driver I would like to hear when the fuel consumption is high.
(iii) As a driver I would like to hear when I am driving too fast or too slow.
(iv) As a driver I would like to hear when I am using too high or too low torque (too much or too little throttle).
(v) As a driver I would like to hear if I am driving at an even or fluctuating speed.
The available input from the car to the system was: speed, torque, speed limit, acceleration, and rate of change of torque. At the time of the project start, the car did not deliver momentary fuel consumption. Therefore, the two first user stories were not implemented. In Sprints 3 to 5 the design work was focussed on sounds to encourage an even speed as this is a key factor for fuel efficient driving and easily measurable in lab settings.
Design Sprint 1 – Prototype implemented in racing game and in car
The delivery from Design Sprint 1 was a fully working prototype, first implemented in a computer racing game and then in a real car. The objective was to allow the designers to develop and test-drive a number of sound concepts. It was assumed that experiencing the sounds in the right context is essential for making the right decisions.
Design Sprint 2 – Contextual Inquiry Interviews
The planned outputs from Sprint 2 were prototypes of sound concepts working flawlessly with all technical and perceptual issues raised in the review of Design Sprint 1 corrected. Nine concepts were developed and assessed in Contextual Inquiry interviews with three drivers in a real car on a test track. These interviews showed that:
- Opinions on beeps, and tones varied among the participants.
- Speech messages were generally not appreciated. The human nature of the voice messages may give rise to expectations on the system of performing and reacting like a human.
- Concepts based on music or podcast contents were generally well received. Alteration of music and podcasts, for example by adjusting BPM, music mix, volume or spectral content, could be used for providing drivers with continuous information, with a high level of acceptability.
The designers’ reflections can be summarised into the following points:
- Immediate responses to driver actions were generally perceived as logical and desirable.
- Individualisation and possibility to switch the system on and off are probably necessary for acceptance, as preferences for sounds and music varies. Altering a play list or radio program chosen by the driver and elaborating the content could be a way of dealing with the large differences in preferences.
- Addition of sounds, for example drums, when approaching the target speed created a feeling of flow. Therefore, concepts based on composition of sounds enhancing the feeling of flow at the target speed were developed.
- Simulated driving in the designers’ studios was useful for experiencing the interactive sounds. However, the experience varied much between the studio and the real car. It was crucial to do final adjustments, e.g. adjusting levels, equalisation, responses to events, etc., in the car while driving.
Design Sprint 3 – Game Demonstrated at Cambridge Festival of Ideas 2015
Gaming elements were introduced into the racing game/simulator. The game was intended for testing design concepts through demonstrations to larger audiences at science festivals. The game prototype included four sound concepts developed from the most successful designs identified in Sprint 2:
1. Speech only podcast. Treble or bass were gradually decreased as the deviation from the target speed increased, treble when speed was too low and bass when speed was too high. A buzzing sound was played when speed started to deviate from the target speed, and three dings indicated that the target speed had been approached.
2. Music, free choice from Spotify. Treble and bass were adjusted as in Concept 1, and the same buzzing sound and dings were used.
3. Loop based music. Composed for the project. Three states were used: too high, too low and correct speed. The character of the music and the lyrics changed with state.
4. Music mix. Circus by Daisy and the Dark (from the Red Planet album app released for iOS at the App Store, 2015). Remixed depending on speed. Drums removed when deviating from target speed. At too high speed the mix was made intense and at too low speed calm.
The interactive sounds provided support to maintain a constant speed. Measures of simulated fuel consumption and integrated deviation from a given target speed were used to calculate an “eco-score”. The game prototype was based on an open source racing simulator (CORS based on TORCS). OBD-II messages were sent from the racing simulator to a Pure Data patch that interpreted them and controlled software synthesizers in Ableton Live through MIDI.
The prototype was launched at Cambridge Festival of Ideas 2015, where approximately 100 participants tried the game. Feedback to the project was based on one of the designers observing and interacting with the participants. It was felt that the opportunity to interact with such a large number of people gave a deeper understanding of user behaviour in general, and advantages and drawbacks with the design concepts in particular. This experience is supported by literature showing that having the development team taking part in user testing increases their understanding of the usability problems and makes them more sensitive to problems encountered by users.
Design Sprint 4 – Game Demonstrated at Cambridge Science Festival 2016
The four concepts developed in Sprint 1-3 were implemented in a driving game developed using the Unreal Engine 4 game engine. The FMOD game audio middleware was used to allow rapid prototyping of the interactive sounds. The main reason for this change of development platform was to allow for more flexibility in the design of scenarios and game ideas, as well as easy integration with game audio middleware. The new version was demonstrated at Cambridge Science Festival 2016, where more than 300 participants completed a test drive and eco-scores were calculated and presented. A very simple environment and car asset were used (see Figure 3), giving the impression of work in progress and making participants more aware of the objective of the demonstration, helping them pay less attention to details in the game play that were not under investigation.
Design Sprint 5 – Experiment Based on Simulated Driving
Three concepts were developed based on concepts from Sprint 4. In addition, a baseline condition with only simulated interior car sound was included. A driving scenario suitable for a lab experiment was designed. It consisted of a road resembling a rural road without other traffic, with 30 mph and 60 mph speed limit sections alternating during the test drive. The participants were asked to try to drive as close to the speed limit as possible. The driving task was intentionally made quite difficult, with a car simulation that was sensitive to hills and to mistakes in the control of the vehicle. The car asset visible in the game used in Sprint 4 was hidden and a plain graphical presentation of speed and gear was added. 16 subjects participated, 7 female and 9 male, mean age 28 years (SD 14 years), all with a valid driving licence. The experiment started with a training session where all four conditions were driven for approximately 3.5 minutes each.
After the training session the test session started. In the test session the four conditions were driven on the same track, but the track was driven two laps (i.e. approx. 7 min. per condition). After having finished driving one condition the participants were asked to rate how they felt while driving using the 9-point Self-Assessment Manikin (SAM) scales for measuring pleasure, arousal and dominance. In addition, average speed and integrated deviation from the speed limit (target speed) were measured.
Finally, the participants were given the opportunity to give their opinions on the different driver assistance systems. This was made by first driving each system for a short distance, just to remind them about the system. They could abort driving at any time when they felt ready to indicate their preferences and give comments on an assessment sheet. Assessments were made on two 9-point scales: “How did you like this driver assistance?” on a scale ranging from “Disliked very much” to “Liked very much”, and: “Would you use this driver assistance system if it was installed in your car?” on a scale ranging from “Not likely at all” to “Very likely”. In addition, an open box for free comments was provided.
A significant difference in average speed between the baseline condition (no audio-based driver assistance) and the condition with treble and bass adjustments of music was found. Participants drove in average 1.1±1.0 mph slower when the music based driver assistance system was used compared to the baseline condition (p <.05). No other speed related measures differed significantly between the conditions. The results suggest that this kind of continuous audio-based assistance can have some effect on people’s ability to keep an even and suitable speed during a short drive. A likely explanation for the limited effect is that visual information dominated the determination of the participants’ behaviour and performance in this experiment.
Further, a significant difference in estimated arousal was found between the baseline condition (no audio-based driver assistance) and the condition with treble and bass adjustments of music. Participants were less excited/more relaxed when the music based driver assistance system was used compared to the baseline condition (p < .05). No other feelings were significantly different between the conditions.
Significant differences in the likelihood for a subject to use the systems in his/her own car were found (p<.05). Post-hoc tests showed that participants rated the likelihood that they would use the system based on music lower than the baseline condition (p<.05), and the system based on podcasts lower than the baseline condition. It is apparent from the results that the participants preferred driving without the audio based driver assistance systems, despite the positive effect the systems had on driver behaviour. It also indicated that the alteration of music mix was more likely to be used in real driving than th
e other two audio based assistance systems.
Conclusion
The design process used in the Safe and Sound Drive project was based on user-centred agile methods. It facilitated early and continuous feedback throughout the project. The use of game technology enabled designers to experience the sounds in a simulated environment in the studio and made it possible to interact with large numbers of end-user representatives in early phases of the project. The framework was found to be useful and suitable for the design of interactive sounds, and the data collected provides insight into the response to sounds used in advanced driver assistance systems.
The effects on driver behaviour were moderate. A significant effect on average speed could only be seen for one of the conditions. The speed fluctuation was lower for all conditions compared with the baseline condition, but none of these differences were significant (p>.05). The weak effect on driver performance was unexpected, as the qualitative studies in Sprints 1-4 had shown that drivers/players intuitively understood and often appreciated the cues given. A possible explanation could be that subjects were focused on the graphical speedometer during the relatively short lab experiment. Future research should therefore examine if continuous feedback through sounds can have an effect on long distance driving, for example by reducing mistakes when graphical information is missed.
The preference assessments indicated that the alteration of music mix was more likely to be used in real driving than the other two concepts. This is important for future development of continuous interactive sounds, as it shows that a well thought through design can affect the acceptance of the sounds. A major challenge for future audio based user interfaces is to design sounds with a high level of acceptability. The user-centred agile methods provide a toolbox for designing and improving future audio based user interfaces.
The training program
Training in computer gaming art and technology was achieved by Nykänen attending to two taught modules that are part of the BSc in Computer Gaming Technology at Anglia Ruskin University: Introduction to Game Engine Technology and Introduction to Computer Gaming. Training in creative use of sound design consisted of one-to-one sessions with Dr Mariana Lopez and consisted of creative use of sound design, sound design for film and media and training on the Pro Tools application. The digital dissemination training consisted of two one-day training courses run by Anglia Ruskin’s HR team on the subjects of “Presenting for Power and Impact” and “Media Training”. Further, training on digital dissemination was achieved through one-to-one tutorials with Dr Rob Toulson. Website design, podcast and video production and use of social media for research projects were covered. Training in mobile application programming and technology, and Training in commercialisation of mobile applications were covered simultaneously through weekly one-to-one tutorials with Dr Rob Toulson. The Daisy and the Dark Red Planet developed by Dr Toulson was used as a case study, as it technically has many similarities with the Safe and Sound Drive app. Programming techniques as well as commercialisation of mobile apps were discussed.
Public dissemination program
We believe strongly in the importance of making research findings available to the general public as well as communicating broadly about the role of research in society. Further, we find it essential in human-centred research through design to interact with people early on and repeatedly throughout the research process. This is especially important for this kind of research, as the impact on environment and economy is heavily dependent on the spread of knowledge to a large audience and widespread use of the developed applications. Therefore, a number of public engagement activities were carried out.
Science festivals and popular science events:
- Demonstration at the Cambridge Festival of Ideas, October 2015 (approximately 100 participants tested the Safe and Sound Drive demonstrator).
- Demonstration at the Cambridge Science Festival, March 2016 (more than 300 participants tested the Safe and Sound Drive demonstrator).
- Demonstration at My Smarter Essex, Chelmsford, October 2016.
Media spots:
- Radio spot in Swedish Radio P4 Norrbotten 8/9/2016.
Web and social media:
- Presentation on CoDE’s web site.
- Regular posts on Nykänen’s Twitter account and blog.
- Regular posts on CoDE’s Twitter and Facebook accounts.
- Web page summarising the project findings.
Scientific conferences:
- AES Audio for Games, London, February 2016. Paper presentation.
- CoDE Conference 2016, Creative Communication, May 2016. Demonstration.
- AES Convention, Paris, June 2016. Tutorial and demonstration.
- Interactive Audio Systems Symposium, York, September 2016. Paper presentation.
Workshops:
- Brains Eden Gaming Festival, Cambridge, June 2016. Tutorial on audio for games. Arne Nykänen presented user-centred agile game sound design together with Jim Croft, chief sound designer at Frontier.
- AES Cambridge Section talk series, June 2016. Tutorial on user-centred agile sound design.
Journal publications:
The results will be published in scientific journals. However, due to the short project time, no journal papers have been published yet. Manuscripts and material for journal publication have been written. As the project has led to a grant for further studies of the topic, future journal publication of the results is planned. The contribution from the EU and the Safe and Sound Drive project will be acknowledged in all future publications using the material.
Dissemination for the automotive industry
Regular teleconference meetings were held with Volvo Car Corporation. A one-day sound design workshop was held at Volvo Cars, Gothenburg, Sweden. The workshop both served the purpose of getting feedback from Volvo Cars into the Safe and Sound Drive project and to disseminate the project results.
Teleconference meetings were held regularly with Scania throughout the project. The meetings both served the purpose of getting feedback from Scania into the project, and to disseminate project results. The meetings resulted in a grant application for a project developing the ideas from Safe and Sound Drive further, with an emphasis on manipulating the sound sources in the truck to better convey information about the energy consumption and efficiency of the truck. A pre-study has been granted by the Swedish Energy Agency, and will be conducted from September 2016 to February 2017 as a collaborative project with Luleå University of Technology, Scania and Interactive Institute Swedish ICT.
Objective 1: Design a demonstrator of a serious game supporting the driver to increase driver skills, lower fuel consumption and allowing safe and environmentally friendly competition with friends.
Objective 2: Develop and release a mobile application version of the game for iOS, available for download from App Store (this will at a later stage be implemented also for Android and Windows).
Objective 3: Measure how the designed game affects fuel consumption, speed and safety.
Objective 4: Training in computer gaming art and technology.
Objective 5: Training in creative use of sound design.
Objective 6: Digital dissemination training – maximising impact of research.
Objective 7: Training in mobile application programming and technology.
Objective 8: Training in commercialisation of mobile applications.
Objective 9: Public dissemination by demonstrations at science centres in the UK and Sweden, and presentations and demonstrations at festivals and events hosted by Anglia Ruskin University and its networks.
The project followed a user-centred research through design scheme. A demonstrator of an audio-only serious game supporting the driver to lower fuel consumption was design. Volvo Car Corporation supported the project with access to a car, full access to sensors and systems in the car. The goal was to develop a mobile applications version of the game for iOS. Because of practical reasons, mainly Nykänen’s previous knowledge in development for the Windows and Android platforms, and because of new knowledge in game engine technology and audio middleware for games gained during the project, the demonstrator was developed using Unreal Engine. The demonstrator was implemented for Windows and demonstrated in a driving simulator and in a car. Future work includes implementation for Android and Apple platforms for public download.
User-Centred Agile Methods
Design of user interfaces is a multidisciplinary task, requiring deep knowledge in psychophysics (or in this particular case, psychoacoustics) and psychology, as well as methods and procedures for design and product development. Knowledge from psychoacoustics and psychology aids the development process, and enables early decisions on potentially successful design routes, avoiding known pitfalls. Examples in the Safe and Sound Drive case are the choice of modality (hearing over vision), choice of frequency content of the sounds to avoid masking, spatial positioning of sound sources to help the driver to keep attention on the road, awareness of the risk of cognitively overloading the driver with a too complex task, etc. However, the design options are still vast, and knowledge is built along the path of the project. A good final design is crucially dependent on how this knowledge is built and taken care of all the way through. A user-centred agile approach was used. In particular, agile game development was explored, as game design is a field where both graphics and sounds are essential parts of the product. The core idea is to organise the work into short iterations (sprints), listening to the evolving product frequently and repeatedly throughout the design process, and get users into the loop early on. Game engine technology was used for rapid prototyping of concepts.
The Sound Design and Research Process
A user-centred agile method was applied to the Safe and Sound Drive project, including rapid and numerous iterations, and supported by the use of a large number of sound sketches. The process was inspired by Scrum, and major elements from the Scrum framework were used, for example: the division of work into time-boxed sprints; User Stories, Product Backlog and Sprint backlogs; prioritisation of backlog items; and the general aim for always delivering a working result from a sprint. Repeatedly during process, qualitative studies and analyses were performed based on diary notes by the participating designers, Contextual Inquiry Interviews, and observation of participants. This approach increases the chances of success in identifying a design that significantly affects fuel consumption and driver behaviour. Finally, the design was evaluated in a simulator study where driver behaviour was measured.
In the start-up/release planning meeting a number of user stories were determined. In agile methods, user stories are typically one sentence descriptions of what a user wants to do with the product, written on the form: “As [kind of user] I would like [a feature] so that I can [reach a goal]”. The following user stories were defined in the start-up/release planning phase:
(i) As a driver I would like to hear when the fuel consumption is low.
(ii) As a driver I would like to hear when the fuel consumption is high.
(iii) As a driver I would like to hear when I am driving too fast or too slow.
(iv) As a driver I would like to hear when I am using too high or too low torque (too much or too little throttle).
(v) As a driver I would like to hear if I am driving at an even or fluctuating speed.
The available input from the car to the system was: speed, torque, speed limit, acceleration, and rate of change of torque. At the time of the project start, the car did not deliver momentary fuel consumption. Therefore, the two first user stories were not implemented. In Sprints 3 to 5 the design work was focussed on sounds to encourage an even speed as this is a key factor for fuel efficient driving and easily measurable in lab settings.
Design Sprint 1 – Prototype implemented in racing game and in car
The delivery from Design Sprint 1 was a fully working prototype, first implemented in a computer racing game and then in a real car. The objective was to allow the designers to develop and test-drive a number of sound concepts. It was assumed that experiencing the sounds in the right context is essential for making the right decisions.
Design Sprint 2 – Contextual Inquiry Interviews
The planned outputs from Sprint 2 were prototypes of sound concepts working flawlessly with all technical and perceptual issues raised in the review of Design Sprint 1 corrected. Nine concepts were developed and assessed in Contextual Inquiry interviews with three drivers in a real car on a test track. These interviews showed that:
- Opinions on beeps, and tones varied among the participants.
- Speech messages were generally not appreciated. The human nature of the voice messages may give rise to expectations on the system of performing and reacting like a human.
- Concepts based on music or podcast contents were generally well received. Alteration of music and podcasts, for example by adjusting BPM, music mix, volume or spectral content, could be used for providing drivers with continuous information, with a high level of acceptability.
The designers’ reflections can be summarised into the following points:
- Immediate responses to driver actions were generally perceived as logical and desirable.
- Individualisation and possibility to switch the system on and off are probably necessary for acceptance, as preferences for sounds and music varies. Altering a play list or radio program chosen by the driver and elaborating the content could be a way of dealing with the large differences in preferences.
- Addition of sounds, for example drums, when approaching the target speed created a feeling of flow. Therefore, concepts based on composition of sounds enhancing the feeling of flow at the target speed were developed.
- Simulated driving in the designers’ studios was useful for experiencing the interactive sounds. However, the experience varied much between the studio and the real car. It was crucial to do final adjustments, e.g. adjusting levels, equalisation, responses to events, etc., in the car while driving.
Design Sprint 3 – Game Demonstrated at Cambridge Festival of Ideas 2015
Gaming elements were introduced into the racing game/simulator. The game was intended for testing design concepts through demonstrations to larger audiences at science festivals. The game prototype included four sound concepts developed from the most successful designs identified in Sprint 2:
1. Speech only podcast. Treble or bass were gradually decreased as the deviation from the target speed increased, treble when speed was too low and bass when speed was too high. A buzzing sound was played when speed started to deviate from the target speed, and three dings indicated that the target speed had been approached.
2. Music, free choice from Spotify. Treble and bass were adjusted as in Concept 1, and the same buzzing sound and dings were used.
3. Loop based music. Composed for the project. Three states were used: too high, too low and correct speed. The character of the music and the lyrics changed with state.
4. Music mix. Circus by Daisy and the Dark (from the Red Planet album app released for iOS at the App Store, 2015). Remixed depending on speed. Drums removed when deviating from target speed. At too high speed the mix was made intense and at too low speed calm.
The interactive sounds provided support to maintain a constant speed. Measures of simulated fuel consumption and integrated deviation from a given target speed were used to calculate an “eco-score”. The game prototype was based on an open source racing simulator (CORS based on TORCS). OBD-II messages were sent from the racing simulator to a Pure Data patch that interpreted them and controlled software synthesizers in Ableton Live through MIDI.
The prototype was launched at Cambridge Festival of Ideas 2015, where approximately 100 participants tried the game. Feedback to the project was based on one of the designers observing and interacting with the participants. It was felt that the opportunity to interact with such a large number of people gave a deeper understanding of user behaviour in general, and advantages and drawbacks with the design concepts in particular. This experience is supported by literature showing that having the development team taking part in user testing increases their understanding of the usability problems and makes them more sensitive to problems encountered by users.
Design Sprint 4 – Game Demonstrated at Cambridge Science Festival 2016
The four concepts developed in Sprint 1-3 were implemented in a driving game developed using the Unreal Engine 4 game engine. The FMOD game audio middleware was used to allow rapid prototyping of the interactive sounds. The main reason for this change of development platform was to allow for more flexibility in the design of scenarios and game ideas, as well as easy integration with game audio middleware. The new version was demonstrated at Cambridge Science Festival 2016, where more than 300 participants completed a test drive and eco-scores were calculated and presented. A very simple environment and car asset were used (see Figure 3), giving the impression of work in progress and making participants more aware of the objective of the demonstration, helping them pay less attention to details in the game play that were not under investigation.
Design Sprint 5 – Experiment Based on Simulated Driving
Three concepts were developed based on concepts from Sprint 4. In addition, a baseline condition with only simulated interior car sound was included. A driving scenario suitable for a lab experiment was designed. It consisted of a road resembling a rural road without other traffic, with 30 mph and 60 mph speed limit sections alternating during the test drive. The participants were asked to try to drive as close to the speed limit as possible. The driving task was intentionally made quite difficult, with a car simulation that was sensitive to hills and to mistakes in the control of the vehicle. The car asset visible in the game used in Sprint 4 was hidden and a plain graphical presentation of speed and gear was added. 16 subjects participated, 7 female and 9 male, mean age 28 years (SD 14 years), all with a valid driving licence. The experiment started with a training session where all four conditions were driven for approximately 3.5 minutes each.
After the training session the test session started. In the test session the four conditions were driven on the same track, but the track was driven two laps (i.e. approx. 7 min. per condition). After having finished driving one condition the participants were asked to rate how they felt while driving using the 9-point Self-Assessment Manikin (SAM) scales for measuring pleasure, arousal and dominance. In addition, average speed and integrated deviation from the speed limit (target speed) were measured.
Finally, the participants were given the opportunity to give their opinions on the different driver assistance systems. This was made by first driving each system for a short distance, just to remind them about the system. They could abort driving at any time when they felt ready to indicate their preferences and give comments on an assessment sheet. Assessments were made on two 9-point scales: “How did you like this driver assistance?” on a scale ranging from “Disliked very much” to “Liked very much”, and: “Would you use this driver assistance system if it was installed in your car?” on a scale ranging from “Not likely at all” to “Very likely”. In addition, an open box for free comments was provided.
A significant difference in average speed between the baseline condition (no audio-based driver assistance) and the condition with treble and bass adjustments of music was found. Participants drove in average 1.1±1.0 mph slower when the music based driver assistance system was used compared to the baseline condition (p <.05). No other speed related measures differed significantly between the conditions. The results suggest that this kind of continuous audio-based assistance can have some effect on people’s ability to keep an even and suitable speed during a short drive. A likely explanation for the limited effect is that visual information dominated the determination of the participants’ behaviour and performance in this experiment.
Further, a significant difference in estimated arousal was found between the baseline condition (no audio-based driver assistance) and the condition with treble and bass adjustments of music. Participants were less excited/more relaxed when the music based driver assistance system was used compared to the baseline condition (p < .05). No other feelings were significantly different between the conditions.
Significant differences in the likelihood for a subject to use the systems in his/her own car were found (p<.05). Post-hoc tests showed that participants rated the likelihood that they would use the system based on music lower than the baseline condition (p<.05), and the system based on podcasts lower than the baseline condition. It is apparent from the results that the participants preferred driving without the audio based driver assistance systems, despite the positive effect the systems had on driver behaviour. It also indicated that the alteration of music mix was more likely to be used in real driving than th
e other two audio based assistance systems.
Conclusion
The design process used in the Safe and Sound Drive project was based on user-centred agile methods. It facilitated early and continuous feedback throughout the project. The use of game technology enabled designers to experience the sounds in a simulated environment in the studio and made it possible to interact with large numbers of end-user representatives in early phases of the project. The framework was found to be useful and suitable for the design of interactive sounds, and the data collected provides insight into the response to sounds used in advanced driver assistance systems.
The effects on driver behaviour were moderate. A significant effect on average speed could only be seen for one of the conditions. The speed fluctuation was lower for all conditions compared with the baseline condition, but none of these differences were significant (p>.05). The weak effect on driver performance was unexpected, as the qualitative studies in Sprints 1-4 had shown that drivers/players intuitively understood and often appreciated the cues given. A possible explanation could be that subjects were focused on the graphical speedometer during the relatively short lab experiment. Future research should therefore examine if continuous feedback through sounds can have an effect on long distance driving, for example by reducing mistakes when graphical information is missed.
The preference assessments indicated that the alteration of music mix was more likely to be used in real driving than the other two concepts. This is important for future development of continuous interactive sounds, as it shows that a well thought through design can affect the acceptance of the sounds. A major challenge for future audio based user interfaces is to design sounds with a high level of acceptability. The user-centred agile methods provide a toolbox for designing and improving future audio based user interfaces.
The training program
Training in computer gaming art and technology was achieved by Nykänen attending to two taught modules that are part of the BSc in Computer Gaming Technology at Anglia Ruskin University: Introduction to Game Engine Technology and Introduction to Computer Gaming. Training in creative use of sound design consisted of one-to-one sessions with Dr Mariana Lopez and consisted of creative use of sound design, sound design for film and media and training on the Pro Tools application. The digital dissemination training consisted of two one-day training courses run by Anglia Ruskin’s HR team on the subjects of “Presenting for Power and Impact” and “Media Training”. Further, training on digital dissemination was achieved through one-to-one tutorials with Dr Rob Toulson. Website design, podcast and video production and use of social media for research projects were covered. Training in mobile application programming and technology, and Training in commercialisation of mobile applications were covered simultaneously through weekly one-to-one tutorials with Dr Rob Toulson. The Daisy and the Dark Red Planet developed by Dr Toulson was used as a case study, as it technically has many similarities with the Safe and Sound Drive app. Programming techniques as well as commercialisation of mobile apps were discussed.
Public dissemination program
We believe strongly in the importance of making research findings available to the general public as well as communicating broadly about the role of research in society. Further, we find it essential in human-centred research through design to interact with people early on and repeatedly throughout the research process. This is especially important for this kind of research, as the impact on environment and economy is heavily dependent on the spread of knowledge to a large audience and widespread use of the developed applications. Therefore, a number of public engagement activities were carried out.
Science festivals and popular science events:
- Demonstration at the Cambridge Festival of Ideas, October 2015 (approximately 100 participants tested the Safe and Sound Drive demonstrator).
- Demonstration at the Cambridge Science Festival, March 2016 (more than 300 participants tested the Safe and Sound Drive demonstrator).
- Demonstration at My Smarter Essex, Chelmsford, October 2016.
Media spots:
- Radio spot in Swedish Radio P4 Norrbotten 8/9/2016.
Web and social media:
- Presentation on CoDE’s web site.
- Regular posts on Nykänen’s Twitter account and blog.
- Regular posts on CoDE’s Twitter and Facebook accounts.
- Web page summarising the project findings.
Scientific conferences:
- AES Audio for Games, London, February 2016. Paper presentation.
- CoDE Conference 2016, Creative Communication, May 2016. Demonstration.
- AES Convention, Paris, June 2016. Tutorial and demonstration.
- Interactive Audio Systems Symposium, York, September 2016. Paper presentation.
Workshops:
- Brains Eden Gaming Festival, Cambridge, June 2016. Tutorial on audio for games. Arne Nykänen presented user-centred agile game sound design together with Jim Croft, chief sound designer at Frontier.
- AES Cambridge Section talk series, June 2016. Tutorial on user-centred agile sound design.
Journal publications:
The results will be published in scientific journals. However, due to the short project time, no journal papers have been published yet. Manuscripts and material for journal publication have been written. As the project has led to a grant for further studies of the topic, future journal publication of the results is planned. The contribution from the EU and the Safe and Sound Drive project will be acknowledged in all future publications using the material.
Dissemination for the automotive industry
Regular teleconference meetings were held with Volvo Car Corporation. A one-day sound design workshop was held at Volvo Cars, Gothenburg, Sweden. The workshop both served the purpose of getting feedback from Volvo Cars into the Safe and Sound Drive project and to disseminate the project results.
Teleconference meetings were held regularly with Scania throughout the project. The meetings both served the purpose of getting feedback from Scania into the project, and to disseminate project results. The meetings resulted in a grant application for a project developing the ideas from Safe and Sound Drive further, with an emphasis on manipulating the sound sources in the truck to better convey information about the energy consumption and efficiency of the truck. A pre-study has been granted by the Swedish Energy Agency, and will be conducted from September 2016 to February 2017 as a collaborative project with Luleå University of Technology, Scania and Interactive Institute Swedish ICT.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
The project has developed state of the art multi-disciplinary research involving sound design, media technology, computer game design and technology, and engineering acoustics. Nykänen has developed skills in computer game design and technology and applied them in the field of sound design for automotive applications, leading to novel ideas. The ideas have been developed through interaction with professional sound designers and media producers, game designers, automotive engineers, psychologists, and with the public through interaction at two large science festivals in Cambridge. Finally, the developed game was evaluated in a simulator study showing that effects on driving behaviour can be achieved under certain circumstances. The simulator study resulted in extensive material for further development of the concept. The methodology used, based on user-centred agile methods was shown to be useful and effective. The simulator developed, based on game engine technology and audio middleware from the games industry, was effective for getting users into the design loop, and allowed the involved designers to experience and demonstrate ideas early on and repeatedly throughout the project. The methodology and the tools have been demonstrated for automotive manufacturers (Volvo Cars and Scania), and have resulted in serious interest from the manufacturers as well as funding bodies. A pre-study for further development of user interfaces utilising interactive sounds has been granted by the Swedish Energy Agency. The project will run from September 2016 to February 2017, and involves Luleå University of Technology, Scania and Interactive Institute Swedish ICT. The developed simulator will be used and will be further developed in that project. Ideas will also be developed and further explored. Safe and Sound Drive demonstrated that continuous interactive sounds can be used for influencing driver behaviour. It was further shown that sounds can be designed to be intuitively understood and easily explained. However, the study showed that even though people found the game interesting and fun, there was a reluctance to use this kind of media based interactive sounds during real driving. This highlights one of the major challenges for designing the future interactive sounds in cars - to design sound and interfaces that are appealing and desired by future drivers. The continuation project will focus on the use of active noise control to use the interior sound of trucks as interactive sounds. Active noise control allows for the interior sound character to be changed depending on driving state. This will first be studied in a simulator/computer game setting, using the Safe and Sound Drive simulator. The work will follow a user-centred agile approach, and the best ideas from Safe and Sound Drive will be used. This ensures that knowledge and technology developed in Safe and Sound Drive will be developed further.