Periodic Reporting for period 1 - SPIRIT (Scalable Platform for Innovations on Real-time Immersive Telepresence)
Reporting period: 2022-10-01 to 2024-09-30
Immersive telepresence technologies will have game-changing impacts on interactions amongst individuals or with non-human objects (e.g. machines), in cyberspace with blurred boundaries between the virtual and physical world. The impacts of this technology are expected to range in a variety of vertical sectors, including education and training, entertainment, healthcare, manufacturing industry, etc. The key challenges include limitations of both the application platform and the underlying network support to achieve seamless presentation, processing, and delivery of immersive telepresence content at a large scale. Innovative design, rigorous validation and testing exercises aim to fulfil the key technical requirements identified such as low-latency communication, high bandwidth demand, and complex content encoding/rendering tasks in real-time.
The industry-leading SPIRIT consortium will build on the existing TRL4 application platforms and network infrastructures developed by the project partners, aiming to address key technical challenges and further develop all major aspects of telepresence technologies to achieve targeted TRL7.
The SPIRIT Project will focus its innovations on network-layer, transport-layer, application/content-layer techniques, as well as security and privacy mechanisms to facilitate the large-scale operation of telepresence applications. The project team will develop a fully distributed, interconnected testing infrastructure across two geographical sites in Germany and UK, allowing large-scale testing of heterogeneous telepresence applications in real-life Internet environments. The network infrastructure will host two mainstream application environments based on WebRTC and low-latency DASH. In addition to the project-designated use case scenarios, the project team will test a variety of additional use cases covering heterogeneous vertical sectors through FSTP participation.
The industry-leading SPIRIT consortium will build on the existing TRL4 application platforms and network infrastructures developed by the project partners, aiming to address key technical challenges and further develop all major aspects of telepresence technologies to achieve targeted TRL7.
The SPIRIT Project will focus its innovations on network-layer, transport-layer, application/content-layer techniques, as well as security and privacy mechanisms to facilitate the large-scale operation of telepresence applications. The project team will develop a fully distributed, interconnected testing infrastructure across two geographical sites in Germany and UK, allowing large-scale testing of heterogeneous telepresence applications in real-life Internet environments. The network infrastructure will host two mainstream application environments based on WebRTC and low-latency DASH. In addition to the project-designated use case scenarios, the project team will test a variety of additional use cases covering heterogeneous vertical sectors through FSTP participation.
Activities and results of the project during this 1st reporting period can be grouped into two main categories. The first is developing a multi-site, open application /network system to enable immersive telepresence applications and secondly to open this platform to a wider community to offer testing and developing possibilities to 3rd parties.
The 1st objective and group of activities can be further split into:
1. development of the multi-site aspect of the platform
2. carry out and implement innovations at different layers to support telepresence
3. define and test a limited set of specific use-cases to evaluate the telepresence platform.
Towards each of these objectives, progress has been made during this 1st reporting period. The different (individual) testbeds, which were available at the start of the project, were further developed with new tools, new APIs, .. and were interconnected, now offering the envisaged multi-site environment, however still not in a fully operational mode. A third testbed has been included to deal with scalability requirements put forward by the Open Call 1 proposers and initial experimentation for holographic conferencing is already offered.
Various innovations have been developed covering key aspects of collaborative telepresence dealing with content formats, network resource management, scalability, split rendering, and security. Both hologram- and avatar-based immersive communication platforms are integrated into the project testbeds and ready for open call partners and end-user validation. Additional innovations are in the phase of integration onto the testbeds.
To test and evaluate the SPIRIT platform, a limited set of 4 specific use cases were defined:
Use Case 1 – "Multi" - Multi-source live telepresence
Use Case 2 – "Hologram" - Live holographic telepresence
Use Case 3 – "Avatar"- Real-time animation and streaming of realistic avatars
Use Case 4 – "Robot"- Distributed steering of Autonomous Mobile Robots
The 2nd major objective, i.e. opening the SPIRIT platform to external users was succesfully achieved through the 1st of the 2 planned Open Calls. This initial call attracted over 60 applications from more than 15 countries, with over half submitted by SMEs.
Of these applications, 44 were deemed feasible in view of the current status of the SPIRIT Platform. And 11 proposals were granted following an external review process. The proposals spanned more than ten industry sectors, with the chosen experiments focusing on applications in Manufacturing, Education, Entertainment, Healthcare, and Universal fields. These Open Call experiments are anticipated to provide valuable feedback on the SPIRIT platform’s limitations, performance, usability, and readiness, ultimately informing the addition of new features and innovations.
The 1st objective and group of activities can be further split into:
1. development of the multi-site aspect of the platform
2. carry out and implement innovations at different layers to support telepresence
3. define and test a limited set of specific use-cases to evaluate the telepresence platform.
Towards each of these objectives, progress has been made during this 1st reporting period. The different (individual) testbeds, which were available at the start of the project, were further developed with new tools, new APIs, .. and were interconnected, now offering the envisaged multi-site environment, however still not in a fully operational mode. A third testbed has been included to deal with scalability requirements put forward by the Open Call 1 proposers and initial experimentation for holographic conferencing is already offered.
Various innovations have been developed covering key aspects of collaborative telepresence dealing with content formats, network resource management, scalability, split rendering, and security. Both hologram- and avatar-based immersive communication platforms are integrated into the project testbeds and ready for open call partners and end-user validation. Additional innovations are in the phase of integration onto the testbeds.
To test and evaluate the SPIRIT platform, a limited set of 4 specific use cases were defined:
Use Case 1 – "Multi" - Multi-source live telepresence
Use Case 2 – "Hologram" - Live holographic telepresence
Use Case 3 – "Avatar"- Real-time animation and streaming of realistic avatars
Use Case 4 – "Robot"- Distributed steering of Autonomous Mobile Robots
The 2nd major objective, i.e. opening the SPIRIT platform to external users was succesfully achieved through the 1st of the 2 planned Open Calls. This initial call attracted over 60 applications from more than 15 countries, with over half submitted by SMEs.
Of these applications, 44 were deemed feasible in view of the current status of the SPIRIT Platform. And 11 proposals were granted following an external review process. The proposals spanned more than ten industry sectors, with the chosen experiments focusing on applications in Manufacturing, Education, Entertainment, Healthcare, and Universal fields. These Open Call experiments are anticipated to provide valuable feedback on the SPIRIT platform’s limitations, performance, usability, and readiness, ultimately informing the addition of new features and innovations.
Progress beyond the state-of-the-art goes along the activities carried out in this reporting period. The development of a multi-site platform is an open application /network system enabling immersive telepresence applications and opening this platform to a wider community to offer testing and developing possibilities to 3rd parties, is beyond what is currently offered in the community. The response from the 1st Open Call clearly shows the interest and the need covered by this offering.
Other innovations being developed and integrated into the SPIRIT platform in different layers (network layer, transport layer, application layer,..) as well as innovations regarding content (capture human participants, photorealistic avatars, animation of face to speech, neural rendering,..) are also bringing the work within the SPIRIT project beyond the state of the art.
This is also reflected in the scientific contributions to the research community on immersive solutions, but also to wider audiences on network and service management, security and 5G/6G technologies.
Other innovations being developed and integrated into the SPIRIT platform in different layers (network layer, transport layer, application layer,..) as well as innovations regarding content (capture human participants, photorealistic avatars, animation of face to speech, neural rendering,..) are also bringing the work within the SPIRIT project beyond the state of the art.
This is also reflected in the scientific contributions to the research community on immersive solutions, but also to wider audiences on network and service management, security and 5G/6G technologies.