Objective
Goal:
To develop multilingual technology for a speech-derived synthetic face that gives essential visual speech information to hearing impaired users of telephone and other voice channels. This group is significantly handicapped in access to spoken information. Visible face movements are an effective means of enhancing speech intelligibility especially for hearing-impaired persons, and for all in noise. These developments will be assisted by the rapid development of multi-modal speech technology and PC processing power. Background from KTH in this area is unique in Europe. Use will be made of existing multilingual speech databases. Outcomes will include a market definition, a specification of visual information delivery and speech recognition requirements, a complete multilingual prototype, user-based trials of the prototype and a final market assessment based on these trials and a review of implementation in the context of European Telecoms.
DESCRIPTION OF WORK
The project will be based on background technology from the KTH multi-modal speech group, whose achievements in this area are unique in Europe, and existing multilingual speech databases.
The KTH technology comprises two main components.
1/ a system for generating a computer image of a talking 3D head, currently controlled from Swedish phoneme strings
2/ a phoneme level speech recogniser already developed to provide control data of proven utility for the talking head.
Project tasks:
1. Commercial Evaluation (Telia, RNID)
1a. Initial review of market for this technology in relation to such alternatives as videophone, computer-based video links, text displays driven by speech recognition
1b. Final market assessment drawing on outcomes of multi-lingual trials of project prototype (see 4.)
2. Research to specify speech recognition requirements in terms of quality of visual face information (UCL, KTH, IvD, RNID)
2a. definition of speech (viseme) features that need to be accurately recognised
2b. assessment of user tolerance of class and frequency of recognition error
2c. review of existing knowledge of tolerable audio-visual delay and audio-visual asynchrony
3. Development of speech recognition software for a complete demonstrator for the target languages, Dutch, English and Swedish, including assessment of language-independence of recogniser performance and need for language and speaker specific adaptation (KTH, UCL, IvD)
4. Assembly (KTH, Telia) and multilingual testing of full prototype designed for home and workplace use (all partners).
MILESTONES
1. Preliminary definition of user requirements and likely market in Europe - feeding into system design
2. Specification of visual information requirements to determine target performance for speech recognition
3. Speech recognition system optimised for this application & spin-offs to other face image generation applications.
4. Multilingual tested prototype for generation of talking head from live telephone speech.
5. Final full market assessment based on user-evaluations of the prototype.
Fields of science
Call for proposal
Data not availableFunding Scheme
CSC - Cost-sharing contractsCoordinator
100 44 STOCKHOLM
Sweden