Periodic Reporting for period 2 - iv4XR (Intelligent Verification/Validation for Extended Reality Based Systems)
Reporting period: 2021-07-01 to 2022-12-31
Extended Reality (XR) is an umbrella term for advanced interactive systems such as Virtual and Augmented Reality systems. XR systems are emerging in different areas such as entertainment (e.g. games), training (e.g. soldiers, healthcare professionals) and retail (e.g. Rolex, IKEA), amongst others. Testing these complex systems is critical to ensure that they function correctly and deliver a high-quality user experience (UX). As the complexity of these systems keeps increasing, the XR industry now finds itself confronting a soaring engineering challenge: XR's fine-grained and high level of interactivity and realism make such systems very hard and expensive to test.
The current XR systems authoring and development toolset poses no XR testing technology beyond rudimentary record and replay tools with very little automation support. They typically do not support comprehensive test scenarios, and more importantly, the test cases are not easily replicable and do not accommodate changes to the system, making them quickly obsolete. This lack of suitable testing tools means that testing complex XR systems involves hours of manual work done by human testers, increasing costs and difficulty associated with testing the systems. The primary issue associated with these testing difficulties relies mainly on time. Given the time it takes to test one version of one XR system effectively, this lack of an appropriate toolset actively hinders the industry's growth and ability to respond to market demands for sophisticated systems that provide a high-quality user experience.
This project aims to build novel verification and validation technology and a toolkit for XR systems based on techniques from AI to provide learning and reasoning over a virtual world. With this technology, XR developers can deploy powerful test agents to automatically explore and test the correct parameters of their virtual worlds as they iteratively develop and refine them.
The agents will be driven by testing goals that developers may define and configure. These goals include the exploration of parameters and concerns with testing coverage. The agents will have diverse capabilities which can be achieved with the integration of external tools (e.g. FAtiMA toolkit - https://fatima-toolkit.eu/ - and TESTAR - https://testar.org).
The agents will be able to test a wide range of functionalities and address UX issues, an essential part of XR systems. We aim to develop socio-emotional and functional test agents to improve the validation and verification of XR systems, making them more systematic and cost-effective. Our main output will be a framework that allows developers and testers to formulate test goals and deploy intelligent agents to interact with the system.
Intelligent agents can bring several advantages to automated testing, from "human-like" play-testing to the creation of personas, ensuring XR systems are adapted to all relevant demographics. Several agents can also be employed to test systems that can be used by multiple users simultaneously.
The iv4XR project aims to deploy the developed Framework to three use-cases that will provide continuous benchmarking for the test agents developed:
an advanced training simulation system from THALES,
an advanced 3D game called Space Engineers from GoodAI, and
an advanced structure/building monitoring system from Gameware Europe.
All three use cases use different interaction and visualisation technologies, each representing a unique virtual world, reflecting the diversity we have to deal with in the XR market.
The current XR systems authoring and development toolset poses no XR testing technology beyond rudimentary record and replay tools with very little automation support. They typically do not support comprehensive test scenarios, and more importantly, the test cases are not easily replicable and do not accommodate changes to the system, making them quickly obsolete. This lack of suitable testing tools means that testing complex XR systems involves hours of manual work done by human testers, increasing costs and difficulty associated with testing the systems. The primary issue associated with these testing difficulties relies mainly on time. Given the time it takes to test one version of one XR system effectively, this lack of an appropriate toolset actively hinders the industry's growth and ability to respond to market demands for sophisticated systems that provide a high-quality user experience.
This project aims to build novel verification and validation technology and a toolkit for XR systems based on techniques from AI to provide learning and reasoning over a virtual world. With this technology, XR developers can deploy powerful test agents to automatically explore and test the correct parameters of their virtual worlds as they iteratively develop and refine them.
The agents will be driven by testing goals that developers may define and configure. These goals include the exploration of parameters and concerns with testing coverage. The agents will have diverse capabilities which can be achieved with the integration of external tools (e.g. FAtiMA toolkit - https://fatima-toolkit.eu/ - and TESTAR - https://testar.org).
The agents will be able to test a wide range of functionalities and address UX issues, an essential part of XR systems. We aim to develop socio-emotional and functional test agents to improve the validation and verification of XR systems, making them more systematic and cost-effective. Our main output will be a framework that allows developers and testers to formulate test goals and deploy intelligent agents to interact with the system.
Intelligent agents can bring several advantages to automated testing, from "human-like" play-testing to the creation of personas, ensuring XR systems are adapted to all relevant demographics. Several agents can also be employed to test systems that can be used by multiple users simultaneously.
The iv4XR project aims to deploy the developed Framework to three use-cases that will provide continuous benchmarking for the test agents developed:
an advanced training simulation system from THALES,
an advanced 3D game called Space Engineers from GoodAI, and
an advanced structure/building monitoring system from Gameware Europe.
All three use cases use different interaction and visualisation technologies, each representing a unique virtual world, reflecting the diversity we have to deal with in the XR market.
The project has now reached its first milestone: Basic iv4XR Framework. We have an operational framework capable of deploying functional test agents (FTA's) in a System Under Test (SUT). The World Object Model (WOM) and coupling interface are almost finished, and the interface between iv4XR and the pilots allows test agents to have basic controls on the pilots.
The current prototype of the iv4XR framework already supports deploying multiple test agents that can share information and synchronize their progress.
We develop two types of Functional Test Agents (FTAs): FTAs that can determine strategies that will allow them to navigate and interact with different entities to solve their specified goals; and FTAs that explore and validate the robustness of the SUT at the same time.
The Socio-Emotional Training Agents (SETAs) development is also advancing steadily and according to the objectives. Two different versions of SETAs have been integrated with the framework, one being currently trained and validated through user testing. Several modules that will integrate the final SETA (cognitive load and difficulty estimation, player profiling, human-like behaviour, multi-agent and collaboration analysis) are also under development.
We have been working on the dissemination of our results. We have three editions of our project's newsletter, and we have nine publications so far and several communications in different venues. On May 27, 2021, we co-organised a workshop with the ARETE project to discuss the future of XR and create networking opportunities between H2020 project participants.
The current prototype of the iv4XR framework already supports deploying multiple test agents that can share information and synchronize their progress.
We develop two types of Functional Test Agents (FTAs): FTAs that can determine strategies that will allow them to navigate and interact with different entities to solve their specified goals; and FTAs that explore and validate the robustness of the SUT at the same time.
The Socio-Emotional Training Agents (SETAs) development is also advancing steadily and according to the objectives. Two different versions of SETAs have been integrated with the framework, one being currently trained and validated through user testing. Several modules that will integrate the final SETA (cognitive load and difficulty estimation, player profiling, human-like behaviour, multi-agent and collaboration analysis) are also under development.
We have been working on the dissemination of our results. We have three editions of our project's newsletter, and we have nine publications so far and several communications in different venues. On May 27, 2021, we co-organised a workshop with the ARETE project to discuss the future of XR and create networking opportunities between H2020 project participants.
iv4XR aims to build a novel verification and validation technology for XR systems based on AI techniques to provide learning and reasoning over a virtual world.
Current XR testing procedures have little automation support. This lack of automation means that testing complex XR systems is a long and expensive procedure. Aside from the costs of testing XR systems, this lack of automation poses a more pressing challenge. Current testing procedures are not easily replicable, and they do not generalise to updated versions of the system, which means that each new version should undergo a new cycle of manual testing.
Automation will decrease manual labour, decreasing both the costs and the time required to test XR systems. Reducing the expenses reduces entry cost barriers, thus reducing the time it takes for a product to enter the market. More reliable testing and a shorter time to market will improve competitiveness and empower SMEs. Agent-based testing for automated testing of XR systems is a novel approach that aims to test more than mere functionality. With technology so embedded in our daily lives, user experience (UX) has become more critical than ever, and good quality UX can very well make a difference between a product that thrives and a product that fails. Agent-based testing that retrieves information about functionality and UX will dramatically improve the XR industry by delivering reliable and trustworthy XR.
The project uses agent-based testing to develop a technology that allows for the deployment of powerful test agents to automatically explore and test the correct parameters of their virtual worlds as they are iteratively developed and refined. The project will deploy two types of agents: Functional test agents and socioemotional test agents. The functional test agents focus on the functionality qualities of the systems, whereas the socioemotional agents focus on user experience qualities.
Providing a better QA technology to verify XR systems will strengthen industrial capacity to deliver trustworthy future augmented interactive devices and aid European R&D capacity in future large interactive structures. Expediting testing procedures coupled with lower costs will free resources that companies can apply to innovation and social inclusion.
Current XR testing procedures have little automation support. This lack of automation means that testing complex XR systems is a long and expensive procedure. Aside from the costs of testing XR systems, this lack of automation poses a more pressing challenge. Current testing procedures are not easily replicable, and they do not generalise to updated versions of the system, which means that each new version should undergo a new cycle of manual testing.
Automation will decrease manual labour, decreasing both the costs and the time required to test XR systems. Reducing the expenses reduces entry cost barriers, thus reducing the time it takes for a product to enter the market. More reliable testing and a shorter time to market will improve competitiveness and empower SMEs. Agent-based testing for automated testing of XR systems is a novel approach that aims to test more than mere functionality. With technology so embedded in our daily lives, user experience (UX) has become more critical than ever, and good quality UX can very well make a difference between a product that thrives and a product that fails. Agent-based testing that retrieves information about functionality and UX will dramatically improve the XR industry by delivering reliable and trustworthy XR.
The project uses agent-based testing to develop a technology that allows for the deployment of powerful test agents to automatically explore and test the correct parameters of their virtual worlds as they are iteratively developed and refined. The project will deploy two types of agents: Functional test agents and socioemotional test agents. The functional test agents focus on the functionality qualities of the systems, whereas the socioemotional agents focus on user experience qualities.
Providing a better QA technology to verify XR systems will strengthen industrial capacity to deliver trustworthy future augmented interactive devices and aid European R&D capacity in future large interactive structures. Expediting testing procedures coupled with lower costs will free resources that companies can apply to innovation and social inclusion.