Photoreal REaltime Sentient ENTity

PRESENT sets out to create virtual digital humans––sentient agents––that look entirely naturalistic, demonstrate emotional sensitivity, establish an engaging dialogue, add sense to the experience, and act as trustworthy guardians and guides in the interfaces for AR, VR, and more traditional forms of media.

As a reminder, the project’s Objectives have been to:

1- Research and develop new methods of creating sentient agents with highly realistic facial and bodily appearance;

2- Research and develop ways of endowing the agent with a high degree of emotional sensitivity as the basis for interactions with users, with lifelike behaviours that evolve over time;

3- Research and test paradigms for interaction with a highly realistic digital character in augmented and virtual environments, with security and trust in the identity of the virtual agent;

4- Specify and develop tools and a pipeline for the efficient creation of sentient agents, with APIs to integrate different functional components and use the agent in interfaces on a range of devices;

5- Integrate the system components and deploy them on both high-end professional and consumer level devices;

6- Create a range of use cases, test and evaluate experiences and interfaces incorporating the prototype sentient agent on different consumer-level and professional devices;

7- Demonstrate and promote concepts and prototypes to the media, entertainment and creative industries, and communicate results to the public.

Objective 1 has been fully achieved, with results that go well beyond initial expectations. Realism of face and body appearance has been enhanced by unexpected realism in hair and clothes (with the application of AI) and a translation pipeline has made it possible to convert complex offline rendered shading graphs to simplified real-time compatible shading models giving a very high degree of realism.

In relation to Objective 2, the project has successfully researched and developed ways of endowing the agent with a high degree of emotional sensitivity as the basis for interactions with users. Original research has been carried out and demonstrated in the final months on evolving behaviour and speech response.

Objective 3: interaction with a highly realistic digital character with security and trust in the identity of the virtual agent has been successfully demonstrated, in a limited flat-screen environment.

Objective 4: tools and a pipeline for the creation of sentient agents, with APIs to integrate different functional components in interfaces on a range of devices, have been developed and demonstrated in a range of scenarios. The introduction of photorealistic agent creation technologies into professional pipelines and use is ahead of expectations

Objective 5: system components have been integrated and deployed them on both high-end professional and consumer level devices, typically using UE as a common platform

Objective 6, to create a range of use cases, test and evaluate experiences and interfaces incorporating the prototype sentient agent on different consumer-level and professional devices has been properly achieved.

Objective 7, to demonstrate and promote concepts and prototypes to the media, entertainment and creative industries, and communicate results to the public has been achieved. Professional demonstrations at FMX were well received and to quote D8.4 “The results of the Experimental Productions and their evaluation exceed the expectations. The appreciation of the artistic and the technical concepts and results of the Delirious Departure performances was high, the same is true for the therapeutic value of Soulhacker and Adam.” The integration of some tools and processes into professional media creation pipelines has been well ahead of expectation.

The aims and objectives, and the potential for market impact, are as relevant now as when the project was proposed; the effects of the Covid pandemic on the market, the highly publicised launch of Epic’s ‘MetaHuman Creator’ as part of UE, the massive promotion of the ‘Metaverse’ concept, and the advance of AI within the media and entertainment industry all increase rather than diminish the potential for exploitation and impact.

In professional demonstration use cases (character creation and animation, broadcast, registration authority officer, virtual clerk) evaluations found that the PRESENT technology is ready for its use in the proposed environments. The achievement of photo-realistic visual appearance has been outstandingly successful and results are already being integrated into FS’s professional CGI pipeline. PRESENT research has succeeded in advancing the state of the art in behaviour recognition, agent response automation and interaction.

In the artistic and psychiatric demonstration use cases, which deployed behavioural elements and complex interactions between human subjects and non-photo-realistic virtual agents, results showed dramatic advances on the state of the art and delivered rich new insights into immersive behaviour and perception.

Overall, we conclude that the project has achieved remarkable results in the exceptionally difficult circumstances of the Covid pandemic, with significant opportunities for impact both through industrial deployment, further development and future research into intelligent agent technologies.

We remain confident that the project will result in a wide range of exploitable products as well as opportunities for further research and innovation. The immediate results will appear in producing better / quicker / less expensive Visual Effects, for use in feature films, TV, Games, and many other forms of entertainment, possibly revolving around XR experiences. The use of personal avatars for carrying out one’s business is still a little way away, but we believe it will happen.

However, we believe that photorealistic virtual agents will increase the need for widespread, robust identity control and verification. As underlined by the example quoted below, social media is overrun with dangerous misinformation: bots and clickware agents present a real threat to democratic politics and international relations. In the Metaverse (or any other future on-line media space), photorealistic human agents with realistic behaviour are potentially powerful vectors for fake news. It will be essential to incorporate strong identity control and verification mechanisms if the public is to have a hope of distinguishing between truth and falsehood.

Periodic Reporting for period 2 - PRESENT (Photoreal REaltime Sentient ENTity)

Related documents

Share this page

Download