First system to automatically translate sign language
Around 5 % of the global population, or 466 million people, have disabling hearing loss. Around 800 000 people in the European Union use some form of sign language to communicate. Despite the widespread prevalence of deaf people in societies around the world, sign language is poorly understood by those who can hear. Suitable technologies able to bridge this semantic gap in human connectivity are lacking. As a result, many deaf people struggle to integrate fully – both in work, and in the use of public and private services. SignAll is an innovative translation system based on artificial intelligence and designed to translate sign language into other languages and start improving the quality of life for the hard-of-hearing. The Horizon 2020 grant allowed for localising the SIGNALL project for the EU market. The SignAll system allows for direct, seamless and automated translation between deaf and hearing people, without the need for a human translator. It is the first software-hardware solution developed in the world and is already heading to market. The first version of SignAll was developed for use in the United States, piloting a translation system for American Sign Language (ASL). The SignAll team now plans to adapt the technology for European deaf communities and aims to bring it to market across the EU.
Listening and learning
The SignAll system uses hundreds of tailor-made machine learning algorithms to analyse information gathered by a system of cameras and sensors, including everything from large skeletal movements to tiny micro-expressions in the face. Hand gestures are picked up on specially-developed gloves, while the state-of-the-art computer vision system monitors facial and body expressions – vital yet often ignored aspects of sign languages. “One of the distinct solutions of the SignAll system is recognition of facial expressions, like ‘knowing’ if someone raises his or her eyebrows,” says Zsolt Robotka, CEO of SignAll Technologies and project coordinator. Aside from the difficulties faced with multiple sign languages, there are regional dialects, as well as context-specific information which can change the meaning of signs. “We have a team of well-known linguists who understand the nuances of sign languages from various countries, and work to inform the technology on how to translate these combinations of movements in a way that is grammatically correct,” says Robotka. SignAll is also developing and refining a complementary system – an interactive, customisable software tool to teach ASL. The tool gamifies the learning process, teaching and testing users on sign language. “Our Learn ASL tool is the most recent development. It motivates more people to learn and therefore results in more ASL speakers. Ultimately, this leads to better opportunities for social inclusion for the deaf in daily life,” says Robotka.
Supported success
The SignAll team are extremely grateful for the grant they received from the EU, which gave them the time to develop the technology while also learning European legislation and country-specific sign languages. “We could also better understand the technical complexity and the business potential related to every sign language in the region. Ultimately, the grant equipped us with a roadmap for our European market entry,” says Robotka. “For us, there is nothing more important than how it performs and how people like it. We are happy more and more people share their excitement about it. The more people use it, the closer we are to our social mission – equal opportunities for the deaf,” Robotka says.
Keywords
SIGNALL, deaf, difficulty, translation, teach, learn, ASL, algorithm, cameras
