Face and body Analysis Natural Computer Interaction

Project Information

FANCI

Grant agreement ID: 645267

DOI 10.3030/645267

Closed project

EC signature date 16 December 2014

Start date 1 January 2015

End date 30 April 2017

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost € 3 524 688,75

EU contribution € 1 918 000,00

Coordinated by

CEVA D.S.P. LTD
Israel

Periodic Reporting for period 2 - FANCI (Face and body Analysis Natural Computer Interaction)

Reporting period: 2016-01-01 to 2017-04-30

Summary of the context and overall objectives of the project

The FANCI near-market Innovation Action, funded under the EC H2020 Grant Agreement No. 645267, has researched, innovated, architected, designed, developed and validated innovative multimodal face and body analytics and natural computer interfaces (NCI), developed a reference platform fitting a variety of applications, and provided validation through user-experience evaluation and benchmarking in the rapidly expanding automotive field.

The project focused on intelligent user sensing methods, including face analysis, eye tracking, emotion, intention & authentication; hand gesture recognition; voice tone analysis; and head gesture & pose analysis. It complemented those with smart user intention or distraction estimation and system reaction methods, supplying visual (HUD, etc.), voice and haptic feedback as well as autonomous system actions.

FANCI delivered these capabilities in a market ready reference architecture, demonstrated by building a creative Automotive application-suite demonstrator. An Application Programming Interface for software application developers of pervasive natural computer interaction devices, was supplied. The platform empowers embedded products that capture and act upon user intentions, and imaginative utility applications fusing real and virtual content.

The project’s overall objectives were:
- Designing a multi-modal, modular, low power, real-time, “always-on’, user sensing speech and vision system, cutting power consumption of existing systems by factor of 10-50x
- Developing the FANCI API suitable for application developers who are non-domain experts
- Providing a real-time Cloud-based API
- Building a reference platform for ultra low-cost devices applicable to a range of mass market applications and products
- Validating FANCI capabilities in automotive infotainment user experiences

To achieve these goals FANCI combined multi-modal sensor data in a hardware platform to enable robust operation in natural environments, sensing explicit, implicit, and emotional actions of the user in a simple, yet powerful extensible software architecture; provided a low-cost, low-power optimised reference design for embedded products; and validated the approach by building an automotive application demonstrator. The project’s developed methods were integrated into the FANCI creative automotive applications on the partners’ universal reference platform, and validated for its user experience value as well as developer experience value.

The FANCI partners include an effective mix of innovative pioneering SMEs, industry leaders and academic researchers, guided by a large industrial Automotive systems market leader. The project, partially funded by the European Commission (EC), responded to the EC’s Horizon-2020 ICT-22-2014 call’s C focus action. It had a 28-month duration, and had a total budget of over 3.5 M€, of which the EC contributed a total funding of 2.5 M€, though in fact consumed 4.6 M€ with the difference covered by the project partners.
The consortium led various dissemination and communication activities, spearheaded by the FANCI public website, and aided by the professional participation of its industrial partners’ in-house Marketing Communications specialists, leveraging social media (Twitter, LinkedIn, other) interaction and local TV and press exposure. The FANCI partners have also taken active leadership and participation roles in dissemination events, automotive and high-standard conferences and tradeshows such as CES-2017, a concertation event (ICT-2015), industry exhibitions, webinars and seminars.

The consortium’s publication activity yielded results in par with its Integration Action near-market orientation, with one Paper nominated as Best Industrial Paper at IEEE Sensors 2016. The standardisation and certification strategies activities related to formally establish acceptance and compliance of FANCI.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project researched, innovated, architected, designed, developed and validated innovative multimodal face and body analytics and natural computer interfaces (NCI), developed a reference platform fitting a variety of applications, and provided validation through user-experience evaluation and benchmarking in the rapidly expanding automotive field.
FANCI delivered a market-ready reference architecture, demonstrated by building a creative Automotive application-suite demonstrator. An Application Programming Interface for software application developers of pervasive natural computer interaction devices, was supplied. The platform empowers embedded products that capture and act upon user intentions, and imaginative utility applications fusing real and virtual content.
Throughout the project, 20 deliverables were submitted as planned.
The consortium led various dissemination and communication activities, spearheaded by the FANCI public website, and aided by the professional participation of its industrial partners’ in-house Marketing Communications specialists, leveraging social media (Twitter, LinkedIn, other) interaction and local TV and press exposure. The FANCI partners have also taken active leadership and participation roles in dissemination events, automotive and high-standard conferences and tradeshows such as CES-2017, a concertation event (ICT-2015), industry exhibitions, webinars and seminars.
The consortium’s publication activity yielded results in par with its Integration Action near-market orientation, with one Paper nominated as Best Industrial Paper at IEEE Sensors 2016. The standardisation and certification strategies activities related to formally establish acceptance and compliance of FANCI.
The project partners held face-to-face consortium meetings and technical integration workshops, as well as periodic conference calls, to ensure optimal synergy and collaboration. An online collaboration framework and document repository was set-up to further this purpose.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Progress beyond the state of the art:
a) Multi-modal, real-time, low-cost, interoperable, low power NCI platform
b) Driver predictive, multimodal and natural interaction assistive system
c) Face and body Human Machine Interfaces (HMI)
d) Emotion and intention-triggered interaction (understanding human behaviour)
e) Creative applied augmented reality

Expected potential impact:
The multi-modality capabilities implemented at the core of the FANCI project will increased robustness and accuracy of biometrics and user state estimation. Combined with an always-on, low cost, and real-time platform, the platform will enable creation of new use cases not possible before. This will further strengthen the European business of the projects partners who each have either an OEM or licensing business model, supplying technology and services to other European companies. FANCI will have impact in several important domains, some of which are directly related to impacts expected by the call, including:
● Expected Impact A: Enable better uses of ICT technologies within the creative industries by providing directly usable solutions addressing their specific needs
● Expected Impact B: Provide a large spill over of the knowledge acquired to a maximum of European industries
● Expected Impact C: Improve the competitive position of the European industries through the provision of cost effective, innovative and high-value products and services

fanci-platform-concept-2017.jpg