Periodic Reporting for period 1 - EASIER (Intelligent Automatic Sign Language Translation)
Reporting period: 2021-01-01 to 2022-06-30
EASIER is a Horizon 2020 co-funded project of a duration of 36 months that aims to design, develop, and validate a complete multilingual machine translation system that will act as a framework for barrier-free communication among deaf and hearing individuals and for a larger variety of specific contexts and languages.
EASIER addresses the existing challenges by developing an ecosystem which incorporates a robust translation system surrounded by numerous tools and services not only to support equal participation of deaf individuals to the whole range of everyday-life activities within an inclusive community, but also to accelerate the incorporation of less-resourced Sign Languages (SL) into SL technologies and leverage the SL content creation industry.
EASIER leverages user feedback in all the phases of the project and by promoting inclusions of deaf people in all the relevant activities as not only EUD is a partner of the project, but also other partners employ deaf personnel to conduct the project work e.g. RU, DCAL, STXT, INT.
The main scientific and technical pillars of the project are:
• Exploitation of a robust data-driven SL (video) recognition engine to support processing of large quantity of data
• Utilisation of a signing avatar engine that produces accurate signing, easy to comprehend by the deaf community and that integrates augmented information like affective expressions (e.g. emotions) and coherent prosody
• Incorporation of state-of-the-art Machine Translation (MT) technology in a wide range of languages and communication scenarios, with the possibility of introducing a post-editing workflow to enable corrections and language variations.
• Consideration of affective and gender aspect in data processing and visualisation.
• Creation of a Linguistic framework derived from the analysis of the existing European Sign Languages (SL) to formalise similarities and describe commonalities to boost the adoption of SL processing, translation and visualisation technologies.
EASIER addresses the existing challenges by developing an ecosystem which incorporates a robust translation system surrounded by numerous tools and services not only to support equal participation of deaf individuals to the whole range of everyday-life activities within an inclusive community, but also to accelerate the incorporation of less-resourced Sign Languages (SL) into SL technologies and leverage the SL content creation industry.
EASIER leverages user feedback in all the phases of the project and by promoting inclusions of deaf people in all the relevant activities as not only EUD is a partner of the project, but also other partners employ deaf personnel to conduct the project work e.g. RU, DCAL, STXT, INT.
The main scientific and technical pillars of the project are:
• Exploitation of a robust data-driven SL (video) recognition engine to support processing of large quantity of data
• Utilisation of a signing avatar engine that produces accurate signing, easy to comprehend by the deaf community and that integrates augmented information like affective expressions (e.g. emotions) and coherent prosody
• Incorporation of state-of-the-art Machine Translation (MT) technology in a wide range of languages and communication scenarios, with the possibility of introducing a post-editing workflow to enable corrections and language variations.
• Consideration of affective and gender aspect in data processing and visualisation.
• Creation of a Linguistic framework derived from the analysis of the existing European Sign Languages (SL) to formalise similarities and describe commonalities to boost the adoption of SL processing, translation and visualisation technologies.
The work achieved so far is presented per work package to make it easier to link to public deliverables available on the project website.
WP1: The involvement of end-users (e.g. deaf individuals, deaf and hearing sign language professionals) in workshops to collect requirements and expectations for the project and its technical development e.g. the definition of the application User Interface (UI).
WP2: The improvement of the basic version of the avatar incorporating better mouthing and prosody and investigating avatar standards, as well as the deployment of a standalone service to be used in combination with a questionnaire for user evaluation.
WP3: The development of components for 3D skeletal extraction, detailed hand pose estimation and facial expression, mouth gestures and mouthing, as well as affect to support data analysis, translation, and production. A data processing pipeline was created for data collection and annotation/alignment and the release of a preliminary version of EMSL (European Meta Sign Language).
WP4: The collection of large amounts of parallel data with the enrichment for selected dataset of biological sex of the signers and speakers involved and a preliminary release of models for the spoken to spoken and sign to spoken language and vice-versa translation, employing also the preliminary version of EMSL from WP3.
WP5: The extension of the existing SILAS system for human interpretation to integrate the automatic translation and representation (avatar), the post-editing corrections of signing represented in a video allowing a human to record this section with a new, correct signing of a word/phrase and the development of the basic system for the sign language representation and editing via AZee.
WP6: The release of an overview of datasets for European sign languages listing 26 different corpora and tasks and 41 lexical resource and the comparison of annotation formats used in 17 of these corpora for over 20 aspects (e.g. segmentation, compounds, repetition, name signs, directional verbs) to exploit commonalities and reach the definition of a detailed machine-readable interchange format. The processing of broadcast data including not only video and audio streams but also various types of captions and subtitles to a common format.
WP7: The first release of an individual set of tools to detect emotional arousal in the user’s speech utterances, to recognise affect from text (e.g. subtitles), to recognise and classify facial expression and the development of a software that fuses the output of the text, the video and audio-based emotion to predict pleasure, arousal, and dominance.
WP8: The creation of the UI and User eXperience (UX) design for the user interface (mobile and web) via a set of mock-ups and the development of an alpha version of the application to be released for the first user evaluation. The backend architecture and data exchange formats formalization and the initial development of the whole end-to-end implementation of external and internal APIs leveraging microservices for all the individual components.
WP9: The release of a study to make visible the limited scope of documentation of sign languages and their correspondent spoken languages in many countries in the EU and the planning of a workshop on neologism incorporation also ethical matters of language policy and the role of different parties in generating and distributing neologisms in signed.
WP10: The release and animation of all the project communication channels (website, social media, newsletter, etc.); definition and release of targeted communication campaigns, in particular, to make the hearing audience aware about the subject of deafness, sign languages, and Deaf culture; participation as finalists to the .eu Web Awards competition with the EASIER website, which triggered the creation of a dedicated project video; organization of two workshops collocated with the LREC 2022 conference and the first Shared Task on Sign Language Translation (WMT-SLT) at WMT 2022, in collaboration with the sister project SignON.
WP11: Daily project management, including contacts with the SignON coordinator, the creation of a Technical Board and the appointment of the Advisory Board, the release of two versions of the Data Management Plan.
WP1: The involvement of end-users (e.g. deaf individuals, deaf and hearing sign language professionals) in workshops to collect requirements and expectations for the project and its technical development e.g. the definition of the application User Interface (UI).
WP2: The improvement of the basic version of the avatar incorporating better mouthing and prosody and investigating avatar standards, as well as the deployment of a standalone service to be used in combination with a questionnaire for user evaluation.
WP3: The development of components for 3D skeletal extraction, detailed hand pose estimation and facial expression, mouth gestures and mouthing, as well as affect to support data analysis, translation, and production. A data processing pipeline was created for data collection and annotation/alignment and the release of a preliminary version of EMSL (European Meta Sign Language).
WP4: The collection of large amounts of parallel data with the enrichment for selected dataset of biological sex of the signers and speakers involved and a preliminary release of models for the spoken to spoken and sign to spoken language and vice-versa translation, employing also the preliminary version of EMSL from WP3.
WP5: The extension of the existing SILAS system for human interpretation to integrate the automatic translation and representation (avatar), the post-editing corrections of signing represented in a video allowing a human to record this section with a new, correct signing of a word/phrase and the development of the basic system for the sign language representation and editing via AZee.
WP6: The release of an overview of datasets for European sign languages listing 26 different corpora and tasks and 41 lexical resource and the comparison of annotation formats used in 17 of these corpora for over 20 aspects (e.g. segmentation, compounds, repetition, name signs, directional verbs) to exploit commonalities and reach the definition of a detailed machine-readable interchange format. The processing of broadcast data including not only video and audio streams but also various types of captions and subtitles to a common format.
WP7: The first release of an individual set of tools to detect emotional arousal in the user’s speech utterances, to recognise affect from text (e.g. subtitles), to recognise and classify facial expression and the development of a software that fuses the output of the text, the video and audio-based emotion to predict pleasure, arousal, and dominance.
WP8: The creation of the UI and User eXperience (UX) design for the user interface (mobile and web) via a set of mock-ups and the development of an alpha version of the application to be released for the first user evaluation. The backend architecture and data exchange formats formalization and the initial development of the whole end-to-end implementation of external and internal APIs leveraging microservices for all the individual components.
WP9: The release of a study to make visible the limited scope of documentation of sign languages and their correspondent spoken languages in many countries in the EU and the planning of a workshop on neologism incorporation also ethical matters of language policy and the role of different parties in generating and distributing neologisms in signed.
WP10: The release and animation of all the project communication channels (website, social media, newsletter, etc.); definition and release of targeted communication campaigns, in particular, to make the hearing audience aware about the subject of deafness, sign languages, and Deaf culture; participation as finalists to the .eu Web Awards competition with the EASIER website, which triggered the creation of a dedicated project video; organization of two workshops collocated with the LREC 2022 conference and the first Shared Task on Sign Language Translation (WMT-SLT) at WMT 2022, in collaboration with the sister project SignON.
WP11: Daily project management, including contacts with the SignON coordinator, the creation of a Technical Board and the appointment of the Advisory Board, the release of two versions of the Data Management Plan.
EASIER will contribute to facilitating the communication between sign language and spoken language communities in the EU, with direct impact on deaf and hearing individuals’ lives and everyday interaction. Since communication barriers will decrease, deaf people will be able to expand their social interaction and culture transfer to their hearing environment, including work environment. Thus, EASIER will further attract new Sign Language learners from the hearing populations and have impacts on sectors such as policy making, content creation industry including broadcasters or deaf and hearing sign language professional interpreters.EASIER’s scientific impact in the domains of Neural Machine Translation, video recognition and Sign Language animation technologies will change the current landscape and boost their industrialization potential, while sign language researchers will benefit from EASIER tools and methodologies as well as contributions to standards. Actions entailed within EASIER will enable technological uptake by many other sign languages and will build new scientific collaborations with research groups and the industry in Europe and globally.