Periodic Reporting for period 1 - ABILITY (HAPTIC TABLET FOR THE ACCESSIBILITY OF DIGITAL CONTENT TO THE VISUALLY IMPAIRED)
Reporting period: 2022-09-01 to 2024-02-29
Digital technologies have become pervasive in our daily lives and have fundamentally changed the ways we communicate and interact with information and with each other. We now have access to digital content anytime, anywhere and in a palette of formats, ranging from images, videos, text, speech to virtual and augmented reality experiences. Typically, this has made smartphones, and similar mobile devices, an invaluable source of information and a mean of connection to the world and to others. Yet, by relying almost exclusively on visual and auditory feedbacks, these devices pose evident accessibility issues. While most smartphones and tablets are equipped with accessibility features like text-to-speech, braille displays require extra peripherals that are often too expensive and too limited. As a result, the visually impaired have limited access to graphical content and complex notations.
ABILITY aims to revolutionise the situation by proposing the first multisensory tablet covering the wide range of visual disabilities and needs of the visually impaired population, including collaboration with the sighted. For that, it will provide a display equipped with multitouch surface haptics capabilities and a 2D multiline pin display with a novel cost-effective mechanism, relying on fewer and remote actuators. The ABILITY device will provide multisensory interactions and feedback, leveraging AI algorithms for device adaptability to the users’ needs and behaviour, notably for image analysis and predictive writing. In this process, ABILITY will adopt a user-centred design approach throughout the project to collect the users’ needs and involve them in the different design and evaluation stages in an iterative way.
ABILITY aims to revolutionise the situation by proposing the first multisensory tablet covering the wide range of visual disabilities and needs of the visually impaired population, including collaboration with the sighted. For that, it will provide a display equipped with multitouch surface haptics capabilities and a 2D multiline pin display with a novel cost-effective mechanism, relying on fewer and remote actuators. The ABILITY device will provide multisensory interactions and feedback, leveraging AI algorithms for device adaptability to the users’ needs and behaviour, notably for image analysis and predictive writing. In this process, ABILITY will adopt a user-centred design approach throughout the project to collect the users’ needs and involve them in the different design and evaluation stages in an iterative way.
At this stage, the project has progressed on three main activities: the iterative studies for the collection of needs and specifications, the hardware development, and the software development (incl. the AI algorithms).
For the iterative design of the ABILITY solution, we have finalised a number of user studies to gather initial input for the design process. We have collected valuable information and insights to both inspire and direct our design work (and described in deliverables from WP2). In total, 104 individual representatives of the future users, of whom 97 had a visual impairment (including 6 persons who were deafblind), participated in interviews, demonstrator workshops or lo-fi workshops to gather information about current practices for accessing digital information, challenges, needs and wishes for specifications, at four different locations (Sweden, France, Lithuania and Germany) from experts to end-users with various backgrounds and age groups.
On the hardware side, we tested various solutions for the integration of the localised multitouch feedback approach both with OLED screens and the engraved cover glasses of the insideONE tablet. We demonstrated the ability to vibrate those screens and have successfully integrated the localised multitouch technology with an OLED display for a functional prototype.
Regarding the actuators themselves, to facilitate the integration and increase the reliability in the future product, we developed an innovative patch technology able to provide piezoelectric patches that can be integrated collectively onto the OLED screen.
For the braille display, we tested a first design of the pins, and obtained satisfactory results in terms of required force and displacement and holding force. This was validated with a prototype actuating two braille cells. New pin designs are currently investigated to increase the holding force to satisfy users’ requirements while keeping in mind the future industrial process for assembly. Studies also revealed the need for touch detection on the display, for cursor routing or interaction gestures, thus solutions are also being investigated to satisfy these requirements.
In parallel, the software of the tablet has been upgraded for the integration of the future capabilities. A new version of the InsideONE tablet has been released early 2024 with boosted capabilities, such as using the latest Intel Core i7 processor, with up to 1TB storage with a high-speed SSD, improved battery life, using a USB-C power supply, a stereo audio system with four speakers, 2 cameras with buil-in OCR, and all operating on Windows 11.
On the AI algorithms side, initial investigations on the AI algorithms best suited for image and content analysis, and predictive typing were conducted both through a literature review and testing of algorithms on sample images and texts. This led to the development of an image analysis pipeline to demonstrate and test different AI-based methods for image analysis with automatic image captioning, free question and answering on images, images segmentation, object detection, face detection, saliency and edge detection; as well as methods and tools to extract text from images and converters to transform text to braille and braille to text as well as text to speech. Initial work has also been conducted on implementing the AI-based next word prediction to assist typing with multimodal interaction. Finally, through the interviews and workshops with experts and target users, potential use cases were also preliminary explored.
For the iterative design of the ABILITY solution, we have finalised a number of user studies to gather initial input for the design process. We have collected valuable information and insights to both inspire and direct our design work (and described in deliverables from WP2). In total, 104 individual representatives of the future users, of whom 97 had a visual impairment (including 6 persons who were deafblind), participated in interviews, demonstrator workshops or lo-fi workshops to gather information about current practices for accessing digital information, challenges, needs and wishes for specifications, at four different locations (Sweden, France, Lithuania and Germany) from experts to end-users with various backgrounds and age groups.
On the hardware side, we tested various solutions for the integration of the localised multitouch feedback approach both with OLED screens and the engraved cover glasses of the insideONE tablet. We demonstrated the ability to vibrate those screens and have successfully integrated the localised multitouch technology with an OLED display for a functional prototype.
Regarding the actuators themselves, to facilitate the integration and increase the reliability in the future product, we developed an innovative patch technology able to provide piezoelectric patches that can be integrated collectively onto the OLED screen.
For the braille display, we tested a first design of the pins, and obtained satisfactory results in terms of required force and displacement and holding force. This was validated with a prototype actuating two braille cells. New pin designs are currently investigated to increase the holding force to satisfy users’ requirements while keeping in mind the future industrial process for assembly. Studies also revealed the need for touch detection on the display, for cursor routing or interaction gestures, thus solutions are also being investigated to satisfy these requirements.
In parallel, the software of the tablet has been upgraded for the integration of the future capabilities. A new version of the InsideONE tablet has been released early 2024 with boosted capabilities, such as using the latest Intel Core i7 processor, with up to 1TB storage with a high-speed SSD, improved battery life, using a USB-C power supply, a stereo audio system with four speakers, 2 cameras with buil-in OCR, and all operating on Windows 11.
On the AI algorithms side, initial investigations on the AI algorithms best suited for image and content analysis, and predictive typing were conducted both through a literature review and testing of algorithms on sample images and texts. This led to the development of an image analysis pipeline to demonstrate and test different AI-based methods for image analysis with automatic image captioning, free question and answering on images, images segmentation, object detection, face detection, saliency and edge detection; as well as methods and tools to extract text from images and converters to transform text to braille and braille to text as well as text to speech. Initial work has also been conducted on implementing the AI-based next word prediction to assist typing with multimodal interaction. Finally, through the interviews and workshops with experts and target users, potential use cases were also preliminary explored.
The main result expected at the end of the project is a multisensory solution for the visually impaired with a full page braille display combined with a multisensory tablet with auditory, visual, tactile engravings and localised haptic feedbacks on the hardware side. On the software side, a solution interoperable with other ICT and accessibility devices, similar to a consumer tablet, and integrating AI-powered algorithms for image transcription and predictive writing.
This solution goes beyond the state of the art by providing a novel actuation technology for braille displays with the potential to significantly decrease overall costs, and the first tablet with localised multifinger haptic feedbacks, combined with multisensory interaction. The solution will also support various needs through the AI algorithms and back-end software.
The envisioned impacts include:
• Scientific impacts: improved interactions with advanced tactile graphics generation, improved predictive writing for the visually impaired, multisensory technologies for accessing digital content, guidelines on multisensory feedback design for the visually impaired.
• Societal: designing a solution for all level of impairments and needs; that will in the long term contribute to mitigating literacy issues, braille stigma and digital barriers in Europe and beyond; promoting braille literacy and inclusive design; positively impacting the inclusion, education, employment and quality of life of visually impaired; improving the collaboration between sighted and visually impaired peers.
• Economic/Technological: a more cost-effective and sustainable inclusive solution available in “all shapes and sizes”; with an uptake and new markets for assistive technologies and technologies for consumer mobile devices.
These impacts will be achieved through active dissemination & communication, and exploitation strategies.
This solution goes beyond the state of the art by providing a novel actuation technology for braille displays with the potential to significantly decrease overall costs, and the first tablet with localised multifinger haptic feedbacks, combined with multisensory interaction. The solution will also support various needs through the AI algorithms and back-end software.
The envisioned impacts include:
• Scientific impacts: improved interactions with advanced tactile graphics generation, improved predictive writing for the visually impaired, multisensory technologies for accessing digital content, guidelines on multisensory feedback design for the visually impaired.
• Societal: designing a solution for all level of impairments and needs; that will in the long term contribute to mitigating literacy issues, braille stigma and digital barriers in Europe and beyond; promoting braille literacy and inclusive design; positively impacting the inclusion, education, employment and quality of life of visually impaired; improving the collaboration between sighted and visually impaired peers.
• Economic/Technological: a more cost-effective and sustainable inclusive solution available in “all shapes and sizes”; with an uptake and new markets for assistive technologies and technologies for consumer mobile devices.
These impacts will be achieved through active dissemination & communication, and exploitation strategies.