Skip to main content

Photoreal REaltime Sentient ENTity

Periodic Reporting for period 1 - PRESENT (Photoreal REaltime Sentient ENTity)

Reporting period: 2019-09-01 to 2021-02-28

PRESENT sets out to create virtual digital humans––sentient agents––that look entirely naturalistic, demonstrate emotional sensitivity, establish an engaging dialogue, add sense to the experience, and act as trustworthy guardians and guides in the interfaces for AR, VR, and more traditional forms of media.
There is no richer and more complex interaction than the human experience, when we use all our senses together with language and our cognitive abilities to understand our surroundings and interact with other people. Current Intelligent Personal Assistants or IPAs (such as Apple Siri, Microsoft Cortana, Google Assistant, Amazon Alexa) are limited by comparison. IPAs do not as a rule have anthropomorphic representations: put more simply, they lack most of the features we rely on to communicate. We interact with them via smart phones and smart speakers; communication is voice driven and episodic, based on a request-response model. The assistants do not take advantage of visual clues or build an affective relationship that might evolve. While AR and VR involve visual and spatial data, the systems create physical barriers that restrict interaction by getting in the way of sensory input and communication. However, advances in the real-time creation of photorealistic computer generated characters, coupled with emotion and behaviour recognition, and natural language technologies, allow us to envisage a virtual agent that is realistic in both looks and behaviour; that can interact with users as they navigate rich and complex environments; converses in a natural manner; responds to human moods and emotional states; and evolves in response to the user’s behaviour.
The Objectives are therefore to:
• Research and develop new methods of creating sentient agents with highly realistic facial and bodily appearance;
• Research and develop ways of endowing the agent with a high degree of emotional sensitivity as the basis for interactions with users, with lifelike behaviours that evolve over time;
• Research and test paradigms for interaction with a highly realistic digital character in augmented and virtual environments, with security and trust in the identity of the virtual agent;
• Specify and develop tools and a pipeline for the efficient creation of sentient agents, with APIs to integrate different functional components and use the agent in interfaces on a range of devices;
• Integrate the system components and deploy them on both high-end professional and consumer level devices;
• Create a range of use cases, test and evaluate experiences and interfaces incorporating the prototype sentient agent on different consumer-level and professional devices;
• Demonstrate and promote concepts and prototypes to the media, entertainment and creative industries, and communicate results to the public.
Despite the difficulties posed by the Covid emergency and the changes to the workplan due to the departure of IK and transfer of some tasks to CM, the project has made satisfactory progress towards the achievement of its objectives.

A range of use cases and scenarios to drive the development, demonstration and testing has been created and the specification and APIs for an end-to-end modular system linking all components and features of the agent behaviour, has been produced and revised, with the agreement of all partners. It is appropriately based on the Unreal game engine, which should provide a solid platform for future work.

R&D is well under way on the creation of the photorealistic real-time agent, with improvements to the look development, body rigging, facial modelling and hand detail.
Research into the component areas of speech and gesture recognition, emotional arousal and response, behavioural response, interaction and reaction with and between agents is well under way. The security and trust architecture has been well defined.

WP6 and WP7 have yet to produce their first Deliverable, but satisfactory progress is reported in those tasks that have started.
Early demonstration of concepts to the public have been hampered by Covid: UPF’s demo of the virtual clerk concept had to be suspended. CREW has however reported demonstrations to users in Belgium with strong emotional responses, and different responses from men and women.

In practice, due to the way in which work is split across work packages and tasks and addresses different objectives, the partners have at each quarterly meeting discussed the issues associated with providing capabilities for a yet-to-be created Agent. Coupled with delays to capture material due to Covid restrictions the partners have pivoted and worked on feasible capabilities in silos - subsets of the grander design of an integrated agent using the modular system specification. This has been more productive given the restrictions imposed and allows for the capabilities to be added to a fully integrated Agent POC at future stages.

For example, in terms of the M12 PoC targets, the team has demonstrated real-time rendering of an agent with good facial expressiveness and skin texture/detail. Work on the hair groom was in progress but was not completed in time for the M12 deliverable D3.1 Real-time Agent Creation Demonstration. Speech input and arousal response has been well demonstrated.
The work in PRESENT remains highly relevant to the Market. The project’s initial Exploitation Plan in D9.2 sets out the relevance well and we think there will be many and significant opportunities in light of the market conditions set out on 4.1 above.
We have not yet seen any significant evidence of work done to deliver innovation (nor would we expect to in the first year of an RIA, even without Covid) and it is too soon to talk about innovation activities prototype testing or demonstration, new products, processes or methods to be launched to the market.