Solutions for voice interaction towards natural crew assistant

Project Information

VOICI

Grant agreement ID: 785401

DOI

10.3030/785401

Project closed

EC signature date 20 February 2018

Start date 1 March 2018

End date 29 February 2020

Funded under

• SOCIETAL CHALLENGES - Smart, Green And Integrated Transport

Total cost

€ 1 250 750,00

EU contribution

€ 1 111 250,00

Coordinated by

SINTEF AS
Norway

Periodic Reporting for period 2 - VOICI (Solutions for voice interaction towards natural crew assistant)

Reporting period: 2019-03-01 to 2020-02-29

Summary of the context and overall objectives of the project

"VOICI has aimed to develop a natural crew assistant – designed to process natural spoken requests from the crew members and help them perform complex actions. Such an assistant can reduce the cognitive workload associated with the increasingly complex human-machine interfaces in modern aircraft cockpits.
VOICI's main goal has been to demonstrate the natural crew assistant at Technology Readiness Level 3, i.e. experimental proof of concept. This includes speech capture, speech recognition and interpretation, and speech synthesis for dialog with the crew. An Audio Evaluation Environment has been established, for developing and testing the crew assistant under realistic audio/noise conditions.
The project is part of the ""extended cockpit"" work package under Clean Sky 2's ""Systems"" technology demonstration. By reducing crew workload VOICI contributes to optimization of operations; flight safety, crew awareness, better maintenance, reduced cost of operations and generally higher efficiency and lower stress.
The consortium comprises 4 partners: SINTEF (Norway, R&D institute, coordinator), sensiBel (Norway, SME), Multitel (Belgium, R&D institute), Acapela (Belgium, SME).
Thales (France, Clean Sky 2 partner) is Topic Manager.
The project has run 24 months, ending February 2020. By the project end the main goal has been achieved - providing proof of concept demonstration of the crew assistant for two use cases selected by the Topic Manager. The project has contributed to innovation and job creation by improving component technologies, demonstrating the concept and establishing new cooperation within the consortium and with the Topic Manager."

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

VOICI has established an Audio Evaluation Environment, emulating a case selected by the Topic Manager: a Dassault Falcon 2000 cockpit. This includes a geometrical cockpit model and calibrated noise recording, split into multiple channels so as to represent multiple noise sources. Hence, a realistic noise and reverberation environment has been created, including its directional properties. A digital version of the environment has been established by measuring impulse responses between multiple loudspeakers and multiple microphones.

Two methods of speech capture have been investigated: A pilot's headset using a close-talk microphone and an ambient microphone array. Captured speech has been processed by stand-alone speech/dialog processing functions, i.e. suitable for embedding in a cockpit without connection to external, e.g. cloud-based services. The processing includes speech enhancement, voice activity detection, context dependent speech recognition, dialog system and text-to-speech for the noisy environment.

Due to the high noise levels, the close-talk microphone has been found to be the preferred technology for speech capture. Still, arrays of SME partner sensiBel's very low noise optical microphones were shown to produce significant noise suppression when used with modern beamforming algorithms. VOICI has contributed to sensiBel's microphone technology development, representing a significant project exploitation potential both in aviation/transport and in a much wider market.

The speech processing engine has demonstrated 5% Word Error Rate for cruising flight conditions, relying on an context -dependent language model and deep learning approaches for training. The dialog system was built to additionally include touch-screen input, together with external aviation information such as airport and weather data. It has been designed for easy inclusion of extended skills in terms of new stories, intent, entities and actions. Speech based feedback to the pilot was implemented by SME partner Acapela's text-to-speech function optimized for the noisy environment.
The total crew assistant has been called SOLO. The Topic Manager has signalled that an Implementation agreement will be negotiated. Acapela has elaborated plans for exploitation of both the total concept and the optimized text-to-speech function.

Dissemination items of the project have been identified. Due to the relatively short duration of the project, most of the dissemination-plan will be implemented after the end of the project. This will take place in close dialog with the Topic Manager.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

"VOICI's total concept has contributed to the state-of-the-art in cockpit-embedded speech processing and dialog systems – tailored to the aviation scenario and high noise levels. Similarly, the project has contributed to the state-of-the-art in component technologies of the SME partners; very low noise optical microphones and noise-optimized text-to-speech technology.

Through a reduced crew workload VOICI will contribute to Clean Sky 2's ""extended cockpit"" goals; optimization of operations, flight safety, crew awareness, better maintenance, reduced cost of operations and generally higher efficiency and lower stress."

SOLO architecture

Embedded, standalone natural crew assistant

Microphone array with front end electronics. Endfire configuration example.

Audio evaluation environment built from sheet plywood and transparent acrylic