Periodic Reporting for period 1 - SignON (SignON - Sign Language Translation Mobile Application and Open Communications Framework)
Berichtszeitraum: 2021-01-01 bis 2022-06-30
Crossing language barriers is essential for global information exchange and unobstructed, fair communication. According to the World Federation of the Deaf (WFD), over 70 million people are deaf and communicate primarily via sign language (SL). The amount of content generated daily, mainly in spoken language, and the increase in social interactions, e.g. via social media, poses a challenge when it comes to unobstructed and direct communication between deaf, hearing and hard of hearing people. Currently, human interpreters are the main medium for sign-to-spoken, spoken-to-sign and sign-to-sign language translation. The availability and cost of these professionals is often a limiting factor in communication between signers and non-signers.
SignON is a Horizon 2020 project which aims to develop a communication service that translates between signed and spoken (in both text and audio modalities) languages and caters for the communication needs between deaf, hard of hearing and hearing individuals.
Reducing this communication gap is the overall objective of this project. To achieve this objective SignON aims to provide a communication service delivered through a mobile application, which has been co-designed and co-developed with the relevant user communities. This co-creation approach, rigidly followed by the SignON partners, where potential deaf, hard of hearing and hearing users are involved in the research, design and development activities makes SignON a unique endeavour with strong social and societal impact.
SignON is a Horizon 2020 project which aims to develop a communication service that translates between signed and spoken (in both text and audio modalities) languages and caters for the communication needs between deaf, hard of hearing and hearing individuals.
Reducing this communication gap is the overall objective of this project. To achieve this objective SignON aims to provide a communication service delivered through a mobile application, which has been co-designed and co-developed with the relevant user communities. This co-creation approach, rigidly followed by the SignON partners, where potential deaf, hard of hearing and hearing users are involved in the research, design and development activities makes SignON a unique endeavour with strong social and societal impact.
The main focus of our research and development activities is driven by our co-creation process. This engagement with our user communities fuels the ideas that further the development of the project. This work aims to assess the gaps in communication between researchers, industry and the user communities, and in particular, deaf and hard of hearing communities. During these processes we have conducted interviews and surveys, have looked at case-studies and user-oriented design to ensure that users have been involved at every step of the design and development process and that the SignON services (including the App) meet users' expectations. Through open communication with potential users, we also manage their expectations, demonstrating the realistic capabilities of current AI. Our co-creation approach is iterative to ensure that both users and the SignON team advance steadily and together.
To ensure that all activities within the consortium as well as towards the community follow correct ethical principles and respect personal privacy and legislations, the SignON Research Ethics Committee (REC) provides guidance for all of these activities and is on hand to offer support when needed.
To date, the SignON project has successfully developed working versions of the SignON App and Framework according to user requirements and expectations, established a repository for the code-base which will be openly accessible after the end of the project and, which, acts as a platform where development can take place in parallel. The SignON Framework has been developed through a distributed and highly scalable approach which allows for components to operate in parallel (to accommodate multiple users without overhead) as well as to be redesigned, improved and replaced seamlessly. Software components for ASR, SLR, MT, NLP and synthesis have been developed and integrated in the framework. State of the art models to facilitate the translation pipeline have been built and released.
Through continuous communication and dissemination activities, we keep the main stakeholders and the general public informed about our progress.
To ensure that all activities within the consortium as well as towards the community follow correct ethical principles and respect personal privacy and legislations, the SignON Research Ethics Committee (REC) provides guidance for all of these activities and is on hand to offer support when needed.
To date, the SignON project has successfully developed working versions of the SignON App and Framework according to user requirements and expectations, established a repository for the code-base which will be openly accessible after the end of the project and, which, acts as a platform where development can take place in parallel. The SignON Framework has been developed through a distributed and highly scalable approach which allows for components to operate in parallel (to accommodate multiple users without overhead) as well as to be redesigned, improved and replaced seamlessly. Software components for ASR, SLR, MT, NLP and synthesis have been developed and integrated in the framework. State of the art models to facilitate the translation pipeline have been built and released.
Through continuous communication and dissemination activities, we keep the main stakeholders and the general public informed about our progress.
SignON aims to go beyond the state-of-the-art by (i) providing an unprecedented translation service and mobile application that are user-driven and co-created with users; (ii) researching and developing new methods, methodologies and models for sign and speech recognition, as well as machine translation; (iii) delivering a new, customisable 3D virtual signer; (iv) collecting and unifying existing (in terms of formatting and organisation) and generating new data (synthetic data as well as sign language specific lexicons to drive the 3D virtual signer); (v) integrating machine learning capabilities in a large-scale framework to allow new languages, user preferences and use-cases to be integrated; as well as (vi) bringing deaf, hard of hearing and hearing communities closer together through co-creation, collaboration and open communication. We would also hope to provide ethical guidelines and principles that would help ensure fair and respectful data collection and curation, as well as the research and development of open and unbiased communication and collaboration for future projects of the scale and scope of SignON.
The project will continue over the next 18 months. During this time, we will endeavour to complete all tasks that were specified in the Grant Agreement, resulting in the SignOn communication service that will be delivered through a mobile application, with the inclusion of the 5 sign languages and 4 spoken languages that were to be addressed in our project, supported by the final version of the 3D virtual character and synthesis component.
The expectation is that the project will continue to be publicised and spoken about via continuous communication and dissemination activities which will lead to a continuously growing community, even beyond the lifespan of SignON. Appropriate use-cases and usage domains will be identified and investigated and will feed into the SignON Sustainable Exploitation, Innovation and IPR plan. All of these expected results will be subject to continuous ethical and privacy monitoring.
In addition to this, and of more fundamental importance, we hope that what we will have achieved will be a step towards bridging the communication gap between deaf, hard of hearing and hearing individuals. We would hope that the co-creation process in our project would instill confidence in the project and its results, to allow the respective communities to communicate more easily and more freely. We would also like to see progress towards the establishment of a repository of sign language data that would allow further research to be performed in this domain.
The potential impacts for this project are huge. Besides the impact on scientific research in terms of technical advancements in multilingual speech processing and sign language recognition on mobile devices, the societal impact of having access to an easily accessible communication service in lieu of interpreters would create a huge benefit to society. The SignON application is not presented as a tool to replace interpreters and current means of communication but as an option available to the general public if it is required and appropriate. There has also been a noticeable impact on the various partners in the consortium, not least, an improved awareness of the interests, needs and requirements of DHH people as well as deaf culture and communities.
The project will continue over the next 18 months. During this time, we will endeavour to complete all tasks that were specified in the Grant Agreement, resulting in the SignOn communication service that will be delivered through a mobile application, with the inclusion of the 5 sign languages and 4 spoken languages that were to be addressed in our project, supported by the final version of the 3D virtual character and synthesis component.
The expectation is that the project will continue to be publicised and spoken about via continuous communication and dissemination activities which will lead to a continuously growing community, even beyond the lifespan of SignON. Appropriate use-cases and usage domains will be identified and investigated and will feed into the SignON Sustainable Exploitation, Innovation and IPR plan. All of these expected results will be subject to continuous ethical and privacy monitoring.
In addition to this, and of more fundamental importance, we hope that what we will have achieved will be a step towards bridging the communication gap between deaf, hard of hearing and hearing individuals. We would hope that the co-creation process in our project would instill confidence in the project and its results, to allow the respective communities to communicate more easily and more freely. We would also like to see progress towards the establishment of a repository of sign language data that would allow further research to be performed in this domain.
The potential impacts for this project are huge. Besides the impact on scientific research in terms of technical advancements in multilingual speech processing and sign language recognition on mobile devices, the societal impact of having access to an easily accessible communication service in lieu of interpreters would create a huge benefit to society. The SignON application is not presented as a tool to replace interpreters and current means of communication but as an option available to the general public if it is required and appropriate. There has also been a noticeable impact on the various partners in the consortium, not least, an improved awareness of the interests, needs and requirements of DHH people as well as deaf culture and communities.