Periodic Reporting for period 2 - EASIER (Intelligent Automatic Sign Language Translation)
Período documentado: 2022-07-01 hasta 2023-12-31
EASIER addressed the existing challenges by developing an ecosystem which incorporates a translation system surrounded by numerous tools and services not only to support equal participation of deaf individuals to the whole range of everyday-life activities within an inclusive community, but also to accelerate the incorporation of less-resourced Sign Languages (SL) into SL technologies and leverage the SL content creation industry.
EASIER leverages user feedback in all the phases of the project and by promoting inclusions of deaf people in all the relevant activities as not only EUD is a partner of the project, but also other partners employ deaf personnel to conduct the project work e.g. DCAL, STXT, INT.
The main scientific and technical pillars of the project were:
• Exploitation of a robust data-driven SL (video) recognition engine to support processing of large quantity of data
• Utilisation of a signing avatar engine that produces accurate signing, easy to comprehend by the deaf community and that integrates augmented information like affective expressions (e.g. emotions) and coherent prosody
• Incorporation of state-of-the-art Machine Translation (MT) technology in a wide range of languages and communication scenarios, with the possibility of introducing a post-editing workflow to enable corrections and language variations.
• Consideration of affective and gender aspect in data processing and visualisation.
• Creation of a Linguistic framework derived from the analysis of the existing European Sign Languages (SL) to formalise similarities and describe commonalities to boost the adoption of SL processing, translation and visualisation technologies.
- Mobile application and the corresponding desktop platform that serve as the central point for bridging linguistic gaps between spoken and sign languages, providing a user-centered interface and the architecture for seamless translation across several spoken and sign languages.
- Multilingual translation models that support high-quality translation in low-resource language settings.
- Advanced signing avatar technology that provides superior visualization of sign languages. The project’s signing avatar, “Paula,” has undergone significant enhancements during the project’s lifespan.
- Cutting-edge technology for affect extraction, encompassing voice, text, and video inputs. The incorporation of these innovative features enabled the recognition and portrayal of emotions, enhancing the depth of communication experiences
- Sign language recognition models that provide a bottom-up phonetic representation. EASIER employed cutting-edge 2D to 3D lifting technology, enhancing the recognition models.
- Multi-sign language lexical resource with an interlingual index as the backbone – a machine-readable and semantics-based lexicon. The project added multiple European sign languages to the Open Multilingual Wordnet with the aim to establish connections between signs with shared meanings across languages.
- Web applications that allow post-editing of machine translation output, including NERstar which includes automated processes during import, editing, and export to enforce limits and optimize subtitles and glosses; AZDV which provides corrections as graphical diagrams interpretable by software, enabling avatar animations as replacements for erroneous portions of machine translation output;
- Best practices and harmonized guidelines for generation and annotation of new sign language resources and training resources. EASIER has emphasized best practices for data generation, storage, and reuse and provided guidelines that play a pivotal role in ensuring the integrity and accessibility of sign language resources, fostering consistency and quality. Additionally, EASIER actively participated in several standardization development organizations, contributing to the establishment of industry standards in the field of accessible communication.
Besides, EASIER has greatly expanded the network of stakeholders to understand and align with user needs and preferences, boosted by 40+ scientific publications. One of the key aspects of the project was the end user involvement, achieved through active engagement and gathering end user feedback from sign language communities in France, Germany, Greece, the Netherlands, Switzerland, and the UK. This work was coordinated by the European Union of the Deaf (EUD) who led the team responsible for providing the definition of evaluation metrics and the evaluation protocol, assessing user requirements, needs, and gaps, recruitment strategies of end user groups, as well as the performance of project evaluation studies. All together, EASIER engaged 152 deaf and hearing individuals across seven language communities in evaluation studies. Not only did this valuable community feedback significantly influenced the technology development process by shaping the design of the mobile application, avatar, and translation models, it also increased awareness and insights into these advanced technologies within deaf communities through their direct engagement.
The EASIER project identified key improvement areas and laid groundwork for future advancements with its sister project, SignON. The project emphasized effective communication with deaf communities and the importance of involving deaf professionals in outreach. EASIER's outreach efforts raised awareness about deaf culture and aimed to reduce communication barriers, winning recognition for its inclusive and creative approach. By fostering a collaborative environment for the reuse and improvement of its outputs, EASIER encouraged innovation and set a foundation for sustained progress in accessible communication, underscoring the socio-economic benefits of inclusivity and technological advancements in sign language translation.