When you do not speak or sign the same language
Since its launch in 2021, the EU-funded SignON project has been working on the development of a mobile app that automatically translates between signed and oral languages. The aim is to facilitate information exchange among deaf, hard of hearing and hearing individuals across Europe using Irish, British, Dutch, Flemish and Spanish sign languages, and the English, Irish, Dutch and Spanish spoken languages. SignON’s focus is making communication easier in hospitality settings. This is illustrated in a video presented in International Sign, available on the SignON website: “A deaf signer from Ireland is travelling to Spain for work. She needs to check into her hotel, but she’s in a hurry to get to a conference. The hotel clerk on duty only speaks Spanish. The deaf guest doesn’t know Spanish, and doesn’t have time to go back and forth writing in English.” How will the app currently being developed help solve the problem? The video goes on to explain: “The deaf signer can sign in Irish Sign Language, and the application produces text in Spanish. Then the hotel clerk can speak in Spanish and the application provides a translation to Irish Sign Language via a sign language avatar.” The AI technologies used by the app include sign language and speech recognition, and the automatic translation and synthesis of 3D virtual characters. The biggest challenge here is the enormous amount of high-quality data needed to generate well-accepted translations between the different languages. Another challenge involves the type of sign language data used to train AI models. For example, has the data been obtained from non-native interpreters who learned a sign language as a foreign language, as is the case with news broadcasts from hearing interpreters? “These often show bias in their sign language use, leading to the possibility that the deaf people using the application would need to adapt their own (native) sign language to that of the non-native users in order to be understood by the application. Unfortunately, sign language data of an acceptable quality is very scarce and scattered.”
Co-creation is the key
SignON is tackling such issues through its strong commitment to the co-creation principle. Throughout the project, researchers have been collaborating with European deaf and hard of hearing communities to define use cases and to co-design and co-develop the SignON service and app. On 22 September 2023, one day before the annual celebration of the International Day of Sign Languages, SignON will be hosting a workshop titled ‘Navigating the Future of Sign Language Machine Translation: Insights and Challenges from the SignON Project’. The workshop will include presentations on SignON’s outcomes and challenges as the project approaches its conclusion. The research team will also present a new white paper outlining policy recommendations. The workshop will be held in Bruges, Belgium. SignON (SignON - Sign Language Translation Mobile Application and Open Communications Framework) ends in December 2023. For more information, please see: SignON project website
Keywords
SignON, language, sign language, deaf, hard of hearing, hearing, spoken language, communication