Skip to main content

Event-Driven Compressive Vision for Multimodal Interaction with Mobile Devices

Objective

The visually impaired and the elderly, often suffering from mild speech and/or motor disabilities, are experiencing a significant and increasing barrier in accessing ICT technology and services. Yet, in order to be able to participate in a modern, interconnected society that relies on ICT technologies for handling everyday issues, there is clear need also for these user groups to have access to ICT, in particular to mobile platforms such as tablet computers or smart-phones.
The proposed project aims at developing and exploiting the recently matured and quickly advancing biologically-inspired technology of event-driven, compressive sensing (EDC) of audio-visual information, to realize a new generation of low-power multi-modal human-computer interface for mobile devices.
The project is based on two main technology pillars: (A) an air gesture control set, and (B) a vision-assisted speech recognition set. (A) exploits EDC vision for low and high level hand and finger gesture recognition and subsequent command execution; (B) combines temporal dynamics from lip and chin motion acquired using EDC vision sensors with the auditory sensor input to gain robustness and background noise immunity of spoken command recognition and speech-to-text input.
In contrast to state-of-the-art technologies, both proposed human-computer communication channels will be designed to work reliably under uncontrolled conditions. Particularly, mobile devices equipped with the proposed interface technology will facilitate unrestricted outdoor use under uncontrolled lighting and background noise conditions. Furthermore, due to the sparse nature of information encoding, EDC excels conventional approaches in energy efficiency, yielding an ideal solution for mobile, battery-powered devices.
ECOMODE is committed to pave the way for industrialization of commercial products by demonstrating the availability of the required hardware and software components and their integrability into a mobile platform.

Call for proposal

H2020-ICT-2014

See other projects for this call

Sub call

H2020-ICT-2014-1

Coordinator

SORBONNE UNIVERSITE
Net EU contribution
€ 758 125,00
Address
21 Rue De L'ecole De Medecine
75006 Paris
France

See on map

Region
Ile-de-France Ile-de-France Paris
Activity type
Higher or Secondary Education Establishments
Other funding
€ 0,00

Participants (8)

FONDAZIONE ISTITUTO ITALIANO DI TECNOLOGIA
Italy
Net EU contribution
€ 592 708,75
Address
Via Morego 30
16163 Genova

See on map

Region
Nord-Ovest Liguria Genova
Activity type
Research Organisations
Other funding
€ 0,00
AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS
Spain
Net EU contribution
€ 556 278,75
Address
Calle Serrano 117
28006 Madrid

See on map

Region
Comunidad de Madrid Comunidad de Madrid Madrid
Activity type
Research Organisations
Other funding
€ 0,00
Streetlab
France
Net EU contribution
€ 449 665,00
Address
17 Rue Moreau
75012 Paris

See on map

SME

The organization defined itself as SME (small and medium-sized enterprise) at the time the Grant Agreement was signed.

Yes
Region
Ile-de-France Ile-de-France Val-de-Marne
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00
INNOVATI NETWORKS SL

Participation ended

Spain
Net EU contribution
€ 232 506,30
Address
Calle Isabel Colbrand 10 Oficina 113-114
28050 Madrid
SME

The organization defined itself as SME (small and medium-sized enterprise) at the time the Grant Agreement was signed.

Yes
Region
Comunidad de Madrid Comunidad de Madrid Madrid
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00
FONDAZIONE BRUNO KESSLER
Italy
Net EU contribution
€ 488 429,00
Address
Via Santa Croce 77
38122 Trento

See on map

Region
Nord-Est Provincia Autonoma di Trento Trento
Activity type
Research Organisations
Other funding
€ 0,00
PROPHESEE
France
Net EU contribution
€ 286 250,00
Address
74 Rue Du Faubourg Saint Antoine
75012 Paris

See on map

Region
Ile-de-France Ile-de-France Val-de-Marne
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00
FONDATION DE COOPERATION SCIENTIFIQUE VOIR ET ENTENDRE
France
Net EU contribution
€ 257 500,00
Address
17 Rue Moreau
75012 Paris

See on map

Region
Ile-de-France Ile-de-France Val-de-Marne
Activity type
Research Organisations
Other funding
€ 0,00
EXPERIS MANPOWERGROUP SL
Spain
Net EU contribution
€ 176 743,70
Address
Avenida De Burgos 18
28036 Madrid

See on map

Region
Comunidad de Madrid Comunidad de Madrid Madrid
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
Other funding
€ 0,00