CORDIS Archive : ISTweb | KA4 | Interface & SimVis | Mixed Realities & New Imaging Frontiers cluster

IST Mixed Realities & New Imaging Frontiers cluster

Objectives and scope

The objective of this Action Line is to bridge the gap between real and virtual worlds for innovative applications. The focus is on the Reality-Virtuality continuum:

Augmenting virtuality and bringing virtual worlds to life by enhancing realism and level of detail, introducing intelligence, making them persistent and reactive environments;
Augmenting reality and fusing real and virtual universes by enhancing real environments for a range of applications going from wearable computing for navigation and industrial processes to programme production and interactive entertainment; and
Discovering new sensory frontiers by addressing high definition, 3D, full space imaging, multisensory cues and very advanced display systems to create fully immersive environments distributed over heterogeneous networks and platforms in which users will be able to enjoy rich, multisensory experiences for virtual- or tele-presence.

Costs, real-time, human factors, control, protection and ethical issues should be considered.

Mixed Reality (MR) makes the best out of two worlds, effectively marrying the flexibility of computer graphics with the realism of real-life pictures. For this, computer vision, computer graphics and advanced audio-visual representation and coding techniques need to be integrated. There is little doubt that within the not too distant future, MR will lead to new visual interfaces which are needed to move beyond the desktop paradigm. MR is not limited to visualisation and should also be seen in a wider context, opening the way to the integration of mechanics, robotics, toys and appliances with visualisation and IT equipment. Future MR applications will seamlessly integrate real world objects one can touch and feel with software and audio-visual representations.

Running projects

Areas of work

The projects in this cluster are positioned at the cross-road of content and interfaces. Some provide generic frameworks for leveraging computer vision, AV representation, computer graphics, agents and mixed reality technology. The diversification of the projects' target implementation platforms ranging from embedded "smart" cameras to second generation set-top-boxes is testimony that we are moving into the "post-PC" era. These projects also strive to keep a good balance between theoretical research and dealing with practical constraints. Collectively, the projects contribute to the development of novel Audio-Visual interactive services as illustrated below:

The following picture gives a mapping of current projects in the Mixed Realities & New Imaging Frontiers cluster on the Real-Virtual continuum. The vertical axis is the interactivity continuum indicating the type of application from consumption of content (with various degrees of interactivity) to full real-time telepresence.

Concertation activities

Projects exchange information in the following areas:

Avatars and human emotions modelisation (INTERFACE, CROSSES, SAFIRA, MEGA, SONG)
Image analysis, computer vision and 3D reconstruction (VIRTUE, ENREVI, PISTE, INTERFACE, Art.Live, VISIRE, RENAISSANCE, VISIONS )
Second Generation set-top-boxes (PISTE, SAMBITS,...)
Metadata for production (METAVISION, ENREVI)

More details about the concertation process are available here.

Contribution to standards

The following standards are of interest to the group:

Web3D (VRML)
IETF
MPEG (4-7-21),
DVB/MHP, TV-Anytime etc.

Specific contributions so far are:

2D profile for Augmented Reality within MPEG-4 (ENREVI)
FIPA "Real Time Systems" group (Art.Live)
MPEG-7 Version 2 , semantic descriptors for faces (INTERFACE)

Dissemination activities

An estimated 20 publications were produced by this group of project in 2000 (CROSSES, PING, INTERFACE, VISIRE, VIRTUE)

Public demonstrations were held at:

Art.Live: IMAGINA2000, Monaco (01-2000), ISMR 2001, Japan (3-2001), Parc de Bercy, Paris (4-2001), IST2000, Nice (11-2000)
Interface: IBC2000, Amsterdam (9-2000)

A conference on 3D imaging and augmented/virtual reality (EuroImage2001) is organised by INTERFACE in May 2001 (Greece). A special session on Mixed Realities is arranged by Art.Live.

Strategic impact

The exploitation of mixed reality and visualisation technologies will arguably be the next step of development of the information infrastructure as we know it today (WWW, dTV and mobile). These technologies will give us new interfaces for the "nintendo&playstation generation", the generation used to interact with highly visual and interactive systems, such as the games they grew up with. They will offer new images and new forms of content, new ways of learning, new ways of caring, new ways of working, new ways of making business, new opportunities and new markets.

Beyond the hype and the stunning images MR is able to produce, there are already many very practical and high potential applications foreseeable in fields such as maintenance, medical visualisation, guidance and information, entertainment, e-learning, traffic, architecture or office communication. All these applications are currently being explored in collaborative projects.

But the EU agenda goes beyond technology and markets to include societal and socio-economical issues important for the development of the "information society". Since it enables more natural and intuitive interfaces, MR could have a large impact on usability and access to services. By lowering the access threshold, it could consequently contribute to bridging the "digital divide" and favour a wider access to information, every-time and every-where. Through its contribution to the visualisation of large amounts of information, edutainment and learning by doing, it could also revolutionise education and training.

There are still a number of key technical issues to resolve before reliable and acceptable MR systems can be deployed. The integration of mobility, real-virtual visualisation aspects and communications is one of the keys. The development of light and non-intrusive sensors and displays is another. Analysis, understanding and tracking are paramount. More work is needed on human factors aspects.

Once these problems are overcome, a technology such as MR is likely to lead to the creation of completely new markets. In some manufacturing industries, it has the potential to be the next wave of innovation following CAD and robotics. In this context, it may lead to the "revaluation" of manual work. In the media area, MR will be very appealing to service providers in need of differentiation and to operators in need of bandwidth-hungry applications. MR is also likely to be an integral part of the next generations of interactive TV, a mixture of broadcasting and Internet services.

By making it easier to find, use and share information, and by offering new, attractive service options, these technologies will provide fuel for the digital economy and help it achieve a sustained growth. The development potential is enormous.

Visions of the future - Illustration scenarios

Telepresence (VIRTUE)

You are at a meeting table with people spaced around in front of you. You are able to communicate with them effectively as if they are sitting next to you in the same room, in fact you are led to believe they are present in the same room - but they are actually located at several remote locations. For many types of meeting this high-realism telepresence conferencing system will replace the need to travel. Technologies in computer vision and graphics have developed to a position where this vision is achievable.

VR-assisted eCommerce (Jorge Santos)

Your old car is about to give up on you. You switch on your web terminal and log in to check the new VR-enhanced site at www.mycar.com. You quickly locate the type of car you are looking for and enter the virtual show room. You look at the car from all sides, simply by manipulating your 3D mouse. You don't like the colour, you want a "Ferrari Red". You click on the colour palette and here it is, much much better now. You take a look inside, close the virtual door. Your surround sound system goes "swoosh". You like the sound and think "quality". You start the engine. It is quiet, very quiet indeed. You take the car for a mixed test drive in the city and in an open snowy road to see how it handles. You are convinced. This is the car for you. You enquire about the price and instruct your web agent to look for the best offers available and negotiate the very best price with the best dealers it finds. A few hours later, you have got 10 goods offers on your mobile with delivery dates and payment conditions. You start a secure transaction over your mobile phone and place your order. The following Friday you have got new wheels and take the family out.

Documentation and online training of a service worker (STAR)

Scenario 1:

A service worker walks through the plant and finds a machine that needs to be repaired. While he is working he watches a small monitor. He is using a camera system to observe the real environment and his work steps. For help during the repair process he uses his small STAR-system. As an interface for communication with STAR the worker uses hand and gesture recognition as well as face detection. This allows him a communication in a loud or dirty environment. After hand and gesture analyses the modules send parameters to object recognition and object manipulation that are integrated in the rendering platform.

For more help in some unexpected situations he can connect to the control centre where a STAR-system is embedded into other engineering systems. The video stream produced by a camera hand-over module is used as an input to a STAR system in the control centre (another place). The hand-over module automatically chooses the best view of the working area. To remote control the work steps hand-gesture and face recognition are used. This is needed to send him the right information in the moment when he will need this.

Scenario 2:

A service worker has to maintain a machine in a very noisy part of a plant. The machine is due to it's intense use dirty. The job is to follow the maintenance instructions to successfully fulfil his task. The first step is to find the off-switch which is hidden under a layer of dirt. The STAR modules are connected with the steps of the maintenance procedure.

The object recognition module recognise the machine and highlight the off-switch to help the worker find it. He changes position and presses the switch. This is recognised by the gesture recognition module which now indicates that the next step is reached which is to open up one side of the machine to look into the interior. The screws necessary to open and remove the side are being recognised and highlighted. With this help he easily goes on to the next step which is to work inside the possibly broken machine. He starts maintaining the machine with help of animations coming from STAR until he reaches a point where he finds a broken part.

Because this is not a regular step of the maintenance procedure he has to indicate that he needs additional help. Due to the noise it is not possible to use speech recognition. The worker now uses predefined gestures of his hand and face to tell STAR what kind of instructions he needs (i.e. what part is broken, how many parts have to be replaced ...). The instruction volume is provided by existing engineering tools via connection interface module. In the control centre the watchers receive a video stream chosen by the camera hand over module. Now having a good view of the situation they are able to send the necessary additional information to the worker.

CORDIS Archive