Skip to main content
Web oficial de la Unión EuropeaWeb oficial de la UE
European Commission logo
español español
CORDIS - Resultados de investigaciones de la UE
CORDIS
CORDIS Web 30th anniversary CORDIS Web 30th anniversary

Intelligent Automatic Sign Language Translation

Periodic Reporting for period 2 - EASIER (Intelligent Automatic Sign Language Translation)

Período documentado: 2022-07-01 hasta 2023-12-31

EASIER is a Horizon 2020 co-funded project of a duration of 36 months that aimed to design, develop, and validate a multilingual machine translation system that will act as a framework for barrier-free communication among deaf and hearing individuals and for a larger variety of specific contexts and languages.
EASIER addressed the existing challenges by developing an ecosystem which incorporates a translation system surrounded by numerous tools and services not only to support equal participation of deaf individuals to the whole range of everyday-life activities within an inclusive community, but also to accelerate the incorporation of less-resourced Sign Languages (SL) into SL technologies and leverage the SL content creation industry.
EASIER leverages user feedback in all the phases of the project and by promoting inclusions of deaf people in all the relevant activities as not only EUD is a partner of the project, but also other partners employ deaf personnel to conduct the project work e.g. DCAL, STXT, INT.
The main scientific and technical pillars of the project were:
• Exploitation of a robust data-driven SL (video) recognition engine to support processing of large quantity of data
• Utilisation of a signing avatar engine that produces accurate signing, easy to comprehend by the deaf community and that integrates augmented information like affective expressions (e.g. emotions) and coherent prosody
• Incorporation of state-of-the-art Machine Translation (MT) technology in a wide range of languages and communication scenarios, with the possibility of introducing a post-editing workflow to enable corrections and language variations.
• Consideration of affective and gender aspect in data processing and visualisation.
• Creation of a Linguistic framework derived from the analysis of the existing European Sign Languages (SL) to formalise similarities and describe commonalities to boost the adoption of SL processing, translation and visualisation technologies.
During the 36 months of the project, the EASIER project partners have meticulously and collaboratively developed and enhanced a diverse range of assets, reflecting the partners’ shared commitment to improving accessibility and inclusivity. These assets include:
- Mobile application and the corresponding desktop platform that serve as the central point for bridging linguistic gaps between spoken and sign languages, providing a user-centered interface and the architecture for seamless translation across several spoken and sign languages.
- Multilingual translation models that support high-quality translation in low-resource language settings.
- Advanced signing avatar technology that provides superior visualization of sign languages. The project’s signing avatar, “Paula,” has undergone significant enhancements during the project’s lifespan.
- Cutting-edge technology for affect extraction, encompassing voice, text, and video inputs. The incorporation of these innovative features enabled the recognition and portrayal of emotions, enhancing the depth of communication experiences
- Sign language recognition models that provide a bottom-up phonetic representation. EASIER employed cutting-edge 2D to 3D lifting technology, enhancing the recognition models.
- Multi-sign language lexical resource with an interlingual index as the backbone – a machine-readable and semantics-based lexicon. The project added multiple European sign languages to the Open Multilingual Wordnet with the aim to establish connections between signs with shared meanings across languages.
- Web applications that allow post-editing of machine translation output, including NERstar which includes automated processes during import, editing, and export to enforce limits and optimize subtitles and glosses; AZDV which provides corrections as graphical diagrams interpretable by software, enabling avatar animations as replacements for erroneous portions of machine translation output;
- Best practices and harmonized guidelines for generation and annotation of new sign language resources and training resources. EASIER has emphasized best practices for data generation, storage, and reuse and provided guidelines that play a pivotal role in ensuring the integrity and accessibility of sign language resources, fostering consistency and quality. Additionally, EASIER actively participated in several standardization development organizations, contributing to the establishment of industry standards in the field of accessible communication.

Besides, EASIER has greatly expanded the network of stakeholders to understand and align with user needs and preferences, boosted by 40+ scientific publications. One of the key aspects of the project was the end user involvement, achieved through active engagement and gathering end user feedback from sign language communities in France, Germany, Greece, the Netherlands, Switzerland, and the UK. This work was coordinated by the European Union of the Deaf (EUD) who led the team responsible for providing the definition of evaluation metrics and the evaluation protocol, assessing user requirements, needs, and gaps, recruitment strategies of end user groups, as well as the performance of project evaluation studies. All together, EASIER engaged 152 deaf and hearing individuals across seven language communities in evaluation studies. Not only did this valuable community feedback significantly influenced the technology development process by shaping the design of the mobile application, avatar, and translation models, it also increased awareness and insights into these advanced technologies within deaf communities through their direct engagement.
The project introduced and improved three fundamental translation directions: Sign-to-Spoken, Spoken-to-Sign, and Spoken-to-Spoken. These models formed the backbone of EASIER’s translation capabilities, ensuring a comprehensive approach to language accessibility. In additointo that, the avatar Paula now possesses advanced abilities to convey emotions. Additionally, language-specific strategies for mouthing have been integrated, enhancing the avatar’s expressiveness. Notably, Paula can process elements of the EASIER notation, resulting in improved prosody and clearer affect within signed sentence animations.

The EASIER project identified key improvement areas and laid groundwork for future advancements with its sister project, SignON. The project emphasized effective communication with deaf communities and the importance of involving deaf professionals in outreach. EASIER's outreach efforts raised awareness about deaf culture and aimed to reduce communication barriers, winning recognition for its inclusive and creative approach. By fostering a collaborative environment for the reuse and improvement of its outputs, EASIER encouraged innovation and set a foundation for sustained progress in accessible communication, underscoring the socio-economic benefits of inclusivity and technological advancements in sign language translation.
Easier Teaser Image with Paula the Avatar, Website and Video Screenshots
Easier Project Logo Horizontal Colored