Periodic Reporting for period 1 - SERMAS (Socially-acceptable Extended Reality Models and Systems)
Reporting period: 2022-10-01 to 2024-03-31
SERMAS will improve human-machine interaction by providing new models and systems of eXtended Reality with higher level of interaction and greater awareness of the context. SERMAS will pursue the following 9 objectives (specified with the corresponding success criteria):
OBJ 1: Define the SERMAS Methodology
The SERMAS Methodology is oriented at XR engineers who, assisted by security analysts and social scientists, intend to develop next-generation XR systems that can be accepted by their human users.
Success criteria: A precise description of the (well defined, understandable, concretely reproducible) steps to establish social acceptance by adopting methods, tools and activities that the Methodology suggests.
OBJ 2: Define the SERMAS XR Agent
Define the SERMAS XR Agent as a prototypical, general-purpose system realizing an XR model through a combination of HW, SW and algorithmic modules, allowing frictionless interaction with non-specialized users, adapting to the context.
Success criteria: Definition of the spectrum of functional/non-functional requirements (including security and privacy) and technical specifications of the Agent’s modules.
OBJ 3: Improve open natural language generation
The XR Agent will access visual and language information, and communicate with users using both verbal and non-verbal signals.
Success criteria: Performance on automatic metrics and evaluation from human users in controlled and real-world settings.
OBJ 4: Context awareness and integration of structured knowledge
The SERMAS XR Agent uses its sensing suite to be constantly aware of its physical and social context.
Success criteria: Ability to localize the agent in the environment and determine the locations of relevant entities. Ability to track and react appropriately to the movements of nearby users.
OBJ 5: Frictionless interaction Building upon knowledge on the characteristics of communication among humans and of the mechanisms underlying effective interactions, the SERMAS XR Agent will be able to interact with users in a similar fashion to human-human communication.
Success criteria: Ability to recognize the interlocutor within the environment. Ability to identify their mental states and adapt the interaction modalities accordingly. Ability to understand the intention of the user and ability to provide relevant information.
OBJ 6: Augmented gesture-based communication skills The SERMAS XR Agent will recognize and generate pointing gestures to refer to spatially-located entities (e.g. a counter at a post office).
Success criteria: Ability to: recognize when the user points at an object or generic direction in the environment; correctly determine which entity is pointed at when user interacts using pointing gestures; refer to spatially-located entities using the nonverbal expressive capabilities of the agent; effectively interpret and use multiple communication channels (e.g. voice and gestures) in a coordinated way.
OBJ 7: Security assessment
Social acceptance of an XR system will ultimately rely on the interaction’s security (which we mean to include privacy and trustworthiness).
Success criteria: Formal and automated methods and tools for the analysis of socio-technical security, which are still in their infancy in that they can model only basic communication systems, are successfully extended to next-generation XR systems.
OBJ 8: Release the SERMAS Toolkit
The Toolkit will integrate the SERMAS XR Agent and will leverage the methods and tools developed in the previous objectives. It will also integrate the technologies and case studies provided by the Third Parties that will join SERMAS following our open calls.
Success criteria: Toolkit deployed as open source and applied to the case studies as proof-of-concept, and Third Party contributions successfully integrated.
OBJ 9: Demonstrate the SERMAS Proof-of-Concept and establish social acceptance of the case studies.
Success criteria: 3 sets of case study requirements and scenarios, and 3 proof-of-concepts based validations; the Toolkit successfully handles the complexities of our three and the selected 4–5 Third Party case studies.
OBJ 1: Define the SERMAS Methodology
The SERMAS Methodology is oriented at XR engineers who, assisted by security analysts and social scientists, intend to develop next-generation XR systems that can be accepted by their human users.
Success criteria: A precise description of the (well defined, understandable, concretely reproducible) steps to establish social acceptance by adopting methods, tools and activities that the Methodology suggests.
OBJ 2: Define the SERMAS XR Agent
Define the SERMAS XR Agent as a prototypical, general-purpose system realizing an XR model through a combination of HW, SW and algorithmic modules, allowing frictionless interaction with non-specialized users, adapting to the context.
Success criteria: Definition of the spectrum of functional/non-functional requirements (including security and privacy) and technical specifications of the Agent’s modules.
OBJ 3: Improve open natural language generation
The XR Agent will access visual and language information, and communicate with users using both verbal and non-verbal signals.
Success criteria: Performance on automatic metrics and evaluation from human users in controlled and real-world settings.
OBJ 4: Context awareness and integration of structured knowledge
The SERMAS XR Agent uses its sensing suite to be constantly aware of its physical and social context.
Success criteria: Ability to localize the agent in the environment and determine the locations of relevant entities. Ability to track and react appropriately to the movements of nearby users.
OBJ 5: Frictionless interaction Building upon knowledge on the characteristics of communication among humans and of the mechanisms underlying effective interactions, the SERMAS XR Agent will be able to interact with users in a similar fashion to human-human communication.
Success criteria: Ability to recognize the interlocutor within the environment. Ability to identify their mental states and adapt the interaction modalities accordingly. Ability to understand the intention of the user and ability to provide relevant information.
OBJ 6: Augmented gesture-based communication skills The SERMAS XR Agent will recognize and generate pointing gestures to refer to spatially-located entities (e.g. a counter at a post office).
Success criteria: Ability to: recognize when the user points at an object or generic direction in the environment; correctly determine which entity is pointed at when user interacts using pointing gestures; refer to spatially-located entities using the nonverbal expressive capabilities of the agent; effectively interpret and use multiple communication channels (e.g. voice and gestures) in a coordinated way.
OBJ 7: Security assessment
Social acceptance of an XR system will ultimately rely on the interaction’s security (which we mean to include privacy and trustworthiness).
Success criteria: Formal and automated methods and tools for the analysis of socio-technical security, which are still in their infancy in that they can model only basic communication systems, are successfully extended to next-generation XR systems.
OBJ 8: Release the SERMAS Toolkit
The Toolkit will integrate the SERMAS XR Agent and will leverage the methods and tools developed in the previous objectives. It will also integrate the technologies and case studies provided by the Third Parties that will join SERMAS following our open calls.
Success criteria: Toolkit deployed as open source and applied to the case studies as proof-of-concept, and Third Party contributions successfully integrated.
OBJ 9: Demonstrate the SERMAS Proof-of-Concept and establish social acceptance of the case studies.
Success criteria: 3 sets of case study requirements and scenarios, and 3 proof-of-concepts based validations; the Toolkit successfully handles the complexities of our three and the selected 4–5 Third Party case studies.
SERMAS will develop a Methodology and a Toolkit that will greatly simplify the design, development, deployment, and management of XR systems. The Methodology and the Toolkit will be applied to case studies drawn from industrial application scenarios. The work will start by employing the case studies and their scenarios to elicit the guiding requirements for project activities. These requirements form the basis of the design of the SERMAS XR Agent, an innovative modular hardware and software architecture to ensure general applicability and openness for integration of external input. Novel formal and automated methods and tools will be devised to carry out a security assessment to check that the designed modules and their composition into the architecture satisfy the security (and privacy and trustworthiness) requirements on human-agent communication and data protection.
The SERMAS Toolkit will be developed using agile software development principles and continuous integration and testing services. The Toolkit will be an extension of the implemented SERMAS XR Agent modular architecture with cloud infrastructure resources supported with edge computing frameworks.
An iterative in-lab validation of both the SERMAS XR Agent and the Toolkit is foreseen before the real-world validation.
The SERMAS Toolkit will be developed using agile software development principles and continuous integration and testing services. The Toolkit will be an extension of the implemented SERMAS XR Agent modular architecture with cloud infrastructure resources supported with edge computing frameworks.
An iterative in-lab validation of both the SERMAS XR Agent and the Toolkit is foreseen before the real-world validation.
SERMAS will achieve a powerful impact on the expected outcomes set out for the work programme and, more generally, on the ongoing digital revolution by enabling the social acceptance of XR systems. This is measurable through the breadth and strength of the expected impacts and the corresponding measures to maximize these impacts.
SERMAS will improve human-machine interaction by providing new models and systems of eXtended Reality with higher level of interaction and greater awareness of the context.
The services offered will be easily usable through automated technologies that will interact with the user in human — like way (conversation), making them attractive even for target users not accustomed or unwilling to interact with XR Agents, such as the elderly, and at the same time stimulating curiosity of the younger audience, inclined to use innovative technologies. Through the functions of auto training, speech recognition and understand and derive meaning from human languages, it will be able to adapt to different forms of expression, interaction, languages, domains, styles and intent, making the services more accessible and giving the impression of having a personal assistant. Greater inclusiveness of digital services both in terms of increased use of technologies by a public reluctant to interact with machines, and as a territorial presence, offering such services in areas where they were absent. By combining technology with human operators, greater use of services will be obtained in terms of time, optimizing the requests management (e.g. support to counter operators). The target groups that will benefit are: users unable to access services at standard times, low-digitized users who have difficulty interacting with technology, young users who are looking for innovative ways of using services.
SERMAS will improve human-machine interaction by providing new models and systems of eXtended Reality with higher level of interaction and greater awareness of the context.
The services offered will be easily usable through automated technologies that will interact with the user in human — like way (conversation), making them attractive even for target users not accustomed or unwilling to interact with XR Agents, such as the elderly, and at the same time stimulating curiosity of the younger audience, inclined to use innovative technologies. Through the functions of auto training, speech recognition and understand and derive meaning from human languages, it will be able to adapt to different forms of expression, interaction, languages, domains, styles and intent, making the services more accessible and giving the impression of having a personal assistant. Greater inclusiveness of digital services both in terms of increased use of technologies by a public reluctant to interact with machines, and as a territorial presence, offering such services in areas where they were absent. By combining technology with human operators, greater use of services will be obtained in terms of time, optimizing the requests management (e.g. support to counter operators). The target groups that will benefit are: users unable to access services at standard times, low-digitized users who have difficulty interacting with technology, young users who are looking for innovative ways of using services.