Skip to main content

MultimodAl analysis/synthesis systEm for human INTERaction to virtual and augmented environments

Objective

The project will define new models and will implement advanced tools for audio-video analysis, synthesis and representation in order to provide essential technologies for the implementation of large-scale virtual and augmented environments. The metaphor, which will inspire the project approach, is oriented to make man-machine interaction as natural as possible, based on everyday human communication means like speech, facial expressions and body gestures from both sides. A case study application on Internet will be developed and demonstrated.

Objectives:
The objective of the project is to define new models and implement advanced tools for audio-video analysis, synthesis and representation in order to provide essential technologies for the implementation of large-scale virtual and augmented environments. The work is oriented to make man-machine interaction as natural as possible, based on everyday human communication by speech, facial expressions and body gestures. Man to machine action will be based on coherent analysis of audio-video channels to perform low-level tasks, or high level interpretation and data fusion, speech emotion understanding or facial expression classification. Machine-to-man action, on the other hand, will be based on human-like audio-video feedback simulating a "person in the machine". A common sw platform will be developed by the project for the creation of Internet-based applications. A case study application will be developed, demonstrated and evaluated.

Work description:
The project activity will last three years, the first part devoted to specifications definition and tools development, the last part dedicated to the implementation of demo applications. Project coordination, within WP1, will be carried out in synergy and concentration with other projects working in similar research areas within and outside the IST program. Most of the WPs will be activated at the beginning of the project. In particular, WP2 will take responsibility of defining systems specification with particular reference to the common software platform, which will be developed later on in WP5, that will integrate all the tools into an environment with authoring capability for the creation of applications of shared virtual and augmented reality. Technical work packages WP3 and WP4 will develop a library of tools for managing man-machine interaction with reference to man-to-machine and machine-to-man communication, respectively. WP3 will be oriented to the implementation of audio-video analysis tools, WP4 o the implementation of audio-video synthesis tools. Both WP3 and WP4 will guarantee full compliance to specifications defined in WP2. Activities in WP5 will be focused on the integration of the developed tools into a common software platform and on the implementation of an example of application based on shared virtual and augmented environments.

The integrated sw platform, will be developed progressively through upgraded releases, the first of which at the end of the first year of project activity. Compliance with MPEG-4 and MPEG-7 will be guaranteed by deep project commitment in the standardisation process (WP2). At project conclusion, the Interface consortium will organise an International Workshop for public demonstration and dissemination of the achieved results.

Milestones:
During the 3 year project, major deliverables will be produced that will represent the actual project milestones. In particular, after the first year, the common sw platform with integrated tools, after the second year, the updated sw platform with added authoring functionality and, at the third year, the implementation of a case study application, based on the common sw platform. This last milestone will also include the organisation of a public event with live demonstration of results.

Funding Scheme

CSC - Cost-sharing contracts

Coordinator

UNIVERSITA DEGLI STUDI DI GENOVA
Address
Via Balbi 5
16126 Genova
Italy

Participants (13)

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE
France
Address
3, Rue Michel-ange
75794 Paris Cedex 16
CURTIN UNIVERSITY OF TECHNOLOGY
Australia
Address
Kent Street
6845 Perth
ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE
Switzerland
Address
Ecublens
1015 Lausanne
ELAN SPEECH
France
Address
4 Rue Jean Rodier
31400 Toulouse
INFORMATICS AND TELEMATICS INSTITUTE
Greece
Address
Kyvernidou Str 1
54639 Thessaloniki
LINKOEPINGS UNIVERSITET
Sweden
Address
Campus Valla
581 83 Linkoeping
SCANSOFT BELGIUM
Belgium
Address
Guldensporenpark 32
9820 Merelbeke
TAU - TECNOLOGIA AUTOMAZIONE UOMO - SOCIETA COOPERATIVA A RESPONSABILITA LIMITATA
Italy
Address
Via Venti Settembre 637603/6
16121 Genova
UMEA UNIVERSITET
Sweden
Address
C/o Ekonomi Adm Enheten
901 87 Umea
UNIVERSITAT POLITECNICA DE CATALUNYA
Spain
Address
Jordi Girona 31
Barcelona
UNIVERSITE DE GENEVE
Switzerland
Address
Rue Du General Dufour 24
1211 Geneve 4
UNIVERZA V MARIBORU
Slovenia
Address
Slomskov Trg 15
2000 Maribor
W INTERACTIVE SARL
France
Address
Route De Geneve 118
74240 Gaillard