Project factsheets will no longer be updated. All information relevant to the project can be found on the CORDIS factsheet . This is updated on a regular basis with public deliverables, etc.
SignSpeak - Scientific understanding and vision-based technological development for continuous sign language recognition and translation
At a glance
ICT-2007.2.2 - Cognitive Systems, Interaction, Robotics
231424 - STREP
Deaf communities revolve around sign languages as they are their natural means of communication. Although deaf, hard of hearing and hearing signers can communicate without boundaries amongst themselves, there is a serious challenge for the deaf community in trying to integrate into educational, social and work environments, as the vast majority of Europeans do not have signing skills. The overall goal of SignSpeak is to develop a new vision-based technology for translating continuous sign language to text, in order to provide new e-Services to the deaf community and to improve their communication with the hearing people, and the other way around.
Deaf communities revolve around sign languages as they are their natural means of communication. Although deaf, hard of hearing and hearing signers can communicate without boundaries amongst themselves, there is a serious challenge for the deaf community in trying to integrate into educational, social and work environments, as the vast majority of Europeans do not have signing skills.
The overall goal of SignSpeak is to develop a new vision-based technology for translating continuous sign language to text, in order to improve the communication between deaf and hearing communities. Databases have now been created in Dutch and German, demonstrating that SignSpeak can work with different sign languages and context domains.
The breakthrough in the understanding of sign language will allow a subsequent breakthrough in the development of a new vision-based technology for continuous sign language recognition and translation to text. The scientific objective of this project is to enrich the linguistic understanding of several “phonetic” markers, to identify the sign and sentence boundaries needed for parsing continuous sign language discourse by different signers, and also to find out what types of assimilation and coarticulation occur when signs are realised in a string of multiple signs (so going beyond signs in isolation), with across-word and within-word coarticulation effects similar to spoken language. These questions have only received limited attention in the linguistic literature, and have never been studied from the perspective of machine recognition of continuous sign language. Although the study will be carried out in Sing Language of the Netherlands (NGT), these grounding concepts will also be valid for the other sign languages because they refer to the natural and intuitive basement of the signing.
The SignSpeak system tracks the dominant and non-dominant hand, as well as facial expressions and body posture, taking into account the signs performed before and after or, in other words, taking into account the context in which a sign has been realised; the new technology will be also signer and ambient-independent. SignSpeak will be the first step to approach sign language recognition and translation to the levels already obtained in similar technologies like translating from text-to-speech and speech-to-text.
The impact of SignSpeak’s results is much broader than the unique application in sign languages, because the results have also important applications in the industry for improving human-machine communication by gesture and for an automatic object and body part recognition and tracking in video streams.
Where will the project be present?
SignSpeak partners have co-organised four workshops: two of them regarding the construction and best exploitation of sign language corpora, and the other two regarding sign recognition and translation; these two technical workshops have been co-organised with Dicta-Sign, which is another EU funded project working on Sign Recognition, www.dictasign.eu .
Name: Georg Heigold
Organisation: Rheinisch-Westfälische Technische Hochschule Aachen
This page is maintained by: Susan Fraser (email removed)