Periodic Reporting for period 2 - SignON (SignON - Sign Language Translation Mobile Application and Open Communications Framework)
Berichtszeitraum: 2022-07-01 bis 2023-12-31
Crossing language barriers is essential for global information exchange and unobstructed, fair communication. According to the World Federation of the Deaf (WFD), over 70 million people are deaf and communicate primarily via sign language (SL). The amount of content generated daily, mainly in spoken language, and the increase in social interactions, e.g. via social media, poses a challenge when it comes to unobstructed and direct communication between deaf, hearing and hard of hearing people. Currently, human interpreters are the main medium for sign-to-spoken, spoken-to-sign and sign-to-sign language translation. The availability and cost of these professionals is often a limiting factor in communication between signers and non-signers.
SignON is a Horizon 2020 project which aims to develop a communication service that translates between signed and spoken (in both text and audio modalities) languages and caters for the communication needs between deaf, hard of hearing and hearing individuals.
The SignON communication service will translate between sign and spoken languages reducing this communication gap. The SignON communication service will be delivered through a mobile application which is co-designed with the user communities. In fact the SignON project follows a co-creation approach, where potential users are involved in the research, design and development activities from the start of the project. The tight connection with the deaf and hard of hearing communities makes SignON a unique endeavour with strong social and societal impact.
SignON is a Horizon 2020 project which aims to develop a communication service that translates between signed and spoken (in both text and audio modalities) languages and caters for the communication needs between deaf, hard of hearing and hearing individuals.
The SignON communication service will translate between sign and spoken languages reducing this communication gap. The SignON communication service will be delivered through a mobile application which is co-designed with the user communities. In fact the SignON project follows a co-creation approach, where potential users are involved in the research, design and development activities from the start of the project. The tight connection with the deaf and hard of hearing communities makes SignON a unique endeavour with strong social and societal impact.
The main focus of our research and development activities is driven by our co-creation process. This engagement with our user communities fuels the ideas that further the development of the project. This work aims to assess the gaps in communication between researchers, industry and the user communities, and in particular, deaf and hard of hearing communities. During these processes we have conducted interviews and surveys, have looked at case-studies and user-oriented design to ensure that users have been involved at every step of the design and development process and that the SignON services (including the App) meet users' expectations. Through open communication with potential users, we also manage their expectations, demonstrating the realistic capabilities of current AI. Our co-creation approach is iterative to ensure that both users and the SignON team advance steadily and together.
To ensure that all activities within the consortium as well as towards the community follow correct ethical principles and respect personal privacy and legislations, the SignON Research Ethics Committee (REC) provides guidance for all of these activities and is on hand to offer support when needed.
To date, the SignON project has successfully developed working versions of the SignON App and Framework according to user requirements and expectations, established a repository for the code-base which will be openly accessible after the end of the project and, which, acts as a platform where development can take place in parallel. The SignON Framework has been developed through a distributed and highly scalable approach which allows for components to operate in parallel (to accommodate multiple users without overhead) as well as to be redesigned, improved and replaced seamlessly. Software components for ASR, SLR, MT, NLP and synthesis have been developed and integrated in the framework. State of the art models to facilitate the translation pipeline have been built and released.
Through continuous communication and dissemination activities, we keep the main stakeholders and the general public informed about our progress.
To ensure that all activities within the consortium as well as towards the community follow correct ethical principles and respect personal privacy and legislations, the SignON Research Ethics Committee (REC) provides guidance for all of these activities and is on hand to offer support when needed.
To date, the SignON project has successfully developed working versions of the SignON App and Framework according to user requirements and expectations, established a repository for the code-base which will be openly accessible after the end of the project and, which, acts as a platform where development can take place in parallel. The SignON Framework has been developed through a distributed and highly scalable approach which allows for components to operate in parallel (to accommodate multiple users without overhead) as well as to be redesigned, improved and replaced seamlessly. Software components for ASR, SLR, MT, NLP and synthesis have been developed and integrated in the framework. State of the art models to facilitate the translation pipeline have been built and released.
Through continuous communication and dissemination activities, we keep the main stakeholders and the general public informed about our progress.
SignON aims to go beyond the state-of-the-art by (i) providing an unprecedented translation service and mobile application that are user-driven and co-created with users; (ii) researching and developing new methods, methodologies and models for sign and speech recognition, as well as machine translation; (iii) delivering a new, customisable 3D virtual signer; (iv) collecting and unifying existing (in terms of formatting and organisation) and generating new data (synthetic data as well as sign language specific lexicons to drive the 3D virtual signer); (v) integrating machine learning capabilities in a large-scale framework to allow new languages, user preferences and use-cases to be integrated; as well as (vi) bringing deaf, hard of hearing and hearing communities closer together through co-creation, collaboration and open communication. We would also hope to provide ethical guidelines and principles that would help ensure fair and respectful data collection and curation, as well as the research and development of open and unbiased communication and collaboration for future projects of the scale and scope of SignON.
The project ran over 36 months with all tasks that were specified in the Grant Agreement completed and all deliverables submitted on time. The SignON communication framework and the SignON SLMT and ML applications were developed following co-creation principles. The applications achieved Technology Readiness Level 6.
The expectation we set at the beginning of the project, is that the project will continue to be publicised and spoken about via continuous communication and dissemination activities which will lead to a continuously growing community, even beyond the lifespan of SignON. In the 3 months after its end we see its impact on academic and user communities; SignON resonates in new publications, presentations, events, satellite and spin-off projects, and has the potential to continue doing so in the future.
Appropriate use-cases and usage domains were identified and investigated within the SignON Sustainable Exploitation, Innovation and IPR plan. These have been ethical and privacy monitored.
In addition to this, and of more fundamental importance, is that SignON has made a large step towards bridging the communication gap between deaf, hard of hearing and hearing individuals. We believe that the co-creation process in our project instills confidence in the project and its results, and allows the respective communities to communicate more easily and more freely. Through SignON’s success we demonstrated the effectiveness of the adopted co-creation process and we advocate that this should be the default methodology in future projects involving DHH communities. We also see the need for a sign language data repository and draw plans towards the establishment of such a repository in the future.
The impacts for this project are huge. Besides the impact on scientific research in terms of technical advancements in multilingual speech processing and sign language recognition on mobile devices, the societal impact of having access to an easily accessible communication service in lieu of interpreters would create a huge benefit to society. The SignON application is not presented as a tool to replace interpreters and current means of communication, nor it is an off-the-shelf ready-to-use tool, but it sets a stepping stone towards a more open and easily accessible translation option available to the general public if it is required and appropriate. There has also been a noticeable impact on the various partners in the consortium, not least, an improved awareness of the interests, needs and requirements of DHH people as well as deaf culture and communities.
The project ran over 36 months with all tasks that were specified in the Grant Agreement completed and all deliverables submitted on time. The SignON communication framework and the SignON SLMT and ML applications were developed following co-creation principles. The applications achieved Technology Readiness Level 6.
The expectation we set at the beginning of the project, is that the project will continue to be publicised and spoken about via continuous communication and dissemination activities which will lead to a continuously growing community, even beyond the lifespan of SignON. In the 3 months after its end we see its impact on academic and user communities; SignON resonates in new publications, presentations, events, satellite and spin-off projects, and has the potential to continue doing so in the future.
Appropriate use-cases and usage domains were identified and investigated within the SignON Sustainable Exploitation, Innovation and IPR plan. These have been ethical and privacy monitored.
In addition to this, and of more fundamental importance, is that SignON has made a large step towards bridging the communication gap between deaf, hard of hearing and hearing individuals. We believe that the co-creation process in our project instills confidence in the project and its results, and allows the respective communities to communicate more easily and more freely. Through SignON’s success we demonstrated the effectiveness of the adopted co-creation process and we advocate that this should be the default methodology in future projects involving DHH communities. We also see the need for a sign language data repository and draw plans towards the establishment of such a repository in the future.
The impacts for this project are huge. Besides the impact on scientific research in terms of technical advancements in multilingual speech processing and sign language recognition on mobile devices, the societal impact of having access to an easily accessible communication service in lieu of interpreters would create a huge benefit to society. The SignON application is not presented as a tool to replace interpreters and current means of communication, nor it is an off-the-shelf ready-to-use tool, but it sets a stepping stone towards a more open and easily accessible translation option available to the general public if it is required and appropriate. There has also been a noticeable impact on the various partners in the consortium, not least, an improved awareness of the interests, needs and requirements of DHH people as well as deaf culture and communities.