Speeding up the virtual reality revolution with realistic & real-time animation of hand-to-object interaction

Project Information

VirtualGrasp

Grant agreement ID: 829467

DOI

10.3030/829467

Project closed

EC signature date 26 September 2018

Start date 1 November 2018

End date 30 April 2021

Funded under

• INDUSTRIAL LEADERSHIP - Innovation In SMEs

Total cost

€ 2 918 625,00

EU contribution

€ 2 043 037,50

Coordinated by

GLEECHI AB
Sweden

Periodic Reporting for period 3 - VirtualGrasp (Speeding up the virtual reality revolution with realistic & real-time animation of hand-to-object interaction)

Reporting period: 2020-11-01 to 2021-04-30

Summary of the context and overall objectives of the project

Animating hands in VR in a realistic way is a complex task. Hand presence isn’t just about accuracy of the position of the hand in space, but also about representing the myriad subtleties that our endlessly adaptable hand joints and finger digits are capable of. VirtualGrasp by Swedish SME Gleechi is the first commercially available software that creates realistic generic animation of hand movements and their interactions with any kind of objects while being fully hardware agnostic. VirtualGrasp replaces labor-intensive manual animations of the hands by using predictive and adaptive machine learning algorithms that analyze the physical properties of a virtual object, model the most appropriate and realistic formation for the hand and places the hand to that position. VirtualGrasp is based on 8+ years of research in robotics and is a core technology that will speed up the VR revolution. As VirtualGrasp provides natural hand-to-object interactions, it enables VR applications that were not possible before of which several application areas have significant impact on the society. Through the project, Gleechi has focused on three application areas of VirtualGrasp within VR:

A - VR training
While learning by doing has been proven as the most efficient way to learn practical tasks, staff in manufacturing industry often cannot practice on the machines due to the high costs and dangers. VirtualGrasp allows machine operators to practice hands-on training in a safe virtual environment and to train emergency responses. It also allows to stage situations that are expensive and difficult-to-simulate in real-life. By introducing a standardized technology to enable hand interactions for training purposes, Gleechi is significantly reducing the time it takes to create VR training applications as well as increasing the quality of the training experience.

B – Stroke rehabilitation
One of the key reasons for why people struggle to recover from stroke is because of lack of motivation to travel to rehabilitation centers and perform repetitive rehabilitation exercises. By allowing stroke survivors to perform rehabilitation exercises with the help of virtual reality, the rehabilitation is made accessible and motivating. In a VR-based rehabilitation application, users can do exercises in a gamified environment.

C – VR games
Hands are notoriously difficult and time-consuming to animate in 3D video games, and many of the developers settle with unrealistically looking hands, with fingers protruding objects, or pre-animate a limited number of grasps, restricting the user from naturally interacting with the environment. VirtualGrasp solves the problem of accurate hand visualization and hand-to-object interaction and makes it easy to apply for any interactive VR game.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the project, Gleechi has taken the product from an early proof of concept to a product ready to be sold. The work has included improvements of the core technology, packaging it as a scalable product and testing it in three different application areas.

VR training: The work has been focused on implementation of VirtualGrasp for industry training applications, including evaluation of the results, improvement of the technology, and preparation for commercial exploitation. The demand among industry companies is growing rapidly and the segment has been verified to be ready for accelerated growth. We have now started to sell the software solution to enable easy creation of interactive VR training towards several multinational industrial clients.

Stroke rehabilitation: we have tested and evaluated VirtualGrasp for VR-based stroke rehabilitation together with experts and potential users. The software has been proven to enable specifically hand and finger rehabilitation exercises that has the potential to improve the rehabilitation effect significantly. However, while the market has a strong potential, the COVID-19 pandemic has resulted in a temporary pause of the commercialization process due to several partners in the healthcare industry putting innovation efforts outside of the pandemic on hold.

VR games: We have been integrating VirtualGrasp towards the two major game development engines - Unity and Unreal - and tested them together with VR game developers. We have also improved VirtualGrasp based on the developers’ feedback. The game developers that we have been working with have identified that high-quality animations are a must-have for their applications to be successful.

The product, results of research and results from successful client collaborations has been shared through our website as well as several conferences and articles.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

For VR training, VirtualGrasp has been packaged as a software to allow for easy creation of interactive VR training scenarios. This way, industrial companies can create VR training for their staff or clients without extensive development work. The technology has been tested and evaluated together with clients to train staff with the end-result showing significant improvements of the training. We have proven that training with VR leads to fewer errors when performing work tasks, and with that, reduced injuries and fatalities. For end users, VirtualGrasp allows users to intuitively interact with objects, using their natural hand movements to grasp and hold the objects in VR, instead of interacting through artificial interfaces. By visualizing realistic grasps, VirtualGrasp removes the cognitive load that comes from seeing one’s own hands behaving in non-natural ways, which reduces the mental effort and does not demand visual focus. This effect has been proven to enhance the memorization ability for users that perform training tasks in VR significantly. Finally, as part of our focus towards VR training, we have identified that interactive VR training helps people with disabilities to easier learn complex tasks, making training more inclusive. We have verified that the VR training market has a huge potential the coming years and will put particular focus on commercialization towards this segment.

For stroke rehabilitation, we have implemented VirtualGrasp for VR-based rehabilitation applications and concluded based on our studies that rehabilitation with VR and with empowered movement capabilities has the potential to speed up the rehabilitation process. Although competitive VR rehabilitation approaches exist, VirtualGrasp is the only solution available for predicting and visualizing fine finger movement and accurate grasping. With VirtualGrasp, limited movement from patients’ hands are amplified to make them able to use their virtual hands perfectly. Studies have shown that visually amplified movement can significantly improve the rehabilitation effect. By enabling stroke rehabilitation for the hands, we will enable millions of people who suffer from stroke every year to perform more convenient stroke rehabilitation from their home and return to normal life.

For VR games, VirtualGrasp has been tested, improved and packaged to be used by developers. The software has been proven to both speed up implementation of natural interaction, and enhance the virtual experience for the end-user.

VirtualGrasp enables natural object grasping

Using VR equipment to train industrial tasks

Objects looks like they are glued to the hands (left), vs realistic grasps (right)

Gleechi CEO Jakob Way presenting VirtualGrasp at startup conference Slush

Example of VirtualGrasp implementation for an industrial training application

Correct animation of hand interaction is a key challenge in VR