Final Report Summary - IRIS (IRIS: Towards Natural Interaction and Communication)
The Marie Curie FP7-PEOPLE-2013-IAPP IRIS Towards Natural Interaction and Communication (http://www.iris-interaction.eu/(odnośnik otworzy się w nowym oknie)) Grant PIAPP-GA-2013-610986, started in 1st January 2014 and, during its 4 year duration, with the aim of providing a natural interaction communication platform accessible and adapted for all users, particularly for people with speech impairments and elderly in indoor scenarios, has contributed to the advance the state-of-the-art in these domains, while delivering most of its promises. The project started with a consortium coordinated by Microsoft Portugal and involving a SME (FIM – Face In Motion) and 3 Universities: University of Aveiro (UAVR) in Portugal, Middle East Technical University (METU) in Turkey, and University of Zaragoza (UNIZAR) in Spain.
IRIS progress towards achieving the objectives stated in the Annex I of the Grant, was carried by means of the following networking activities, that span throughout the 4 years of project duration: (a) Secondments and recruitments: a total of 111 secondments/months and recruitments/months were executed (corresponding to 72% of the planned figure), where 46 where Inter-sectorial-international secondments (71% of the planned figure), 22.8 where intra-national secondments (79% of the planned figure) and 42.25 referred to recruitments/months (70% of the planned figure); (b) Meetings: Partners have meet every month for quick online sync-up consortium meetings and once per year in scheduled physical meetings. Physical consortium meetings occurred 6 times during the project period: @ kickoff meeting, M1 – Microsoft, Portugal, @M09 - UNIZAR, Spain, @M15 – UAVR, Portugal, @M22 – MSFT, Portugal, @M35 – ISCTE-IUL, Portugal and @M41 – ISCTE-IUL, Portugal. A Mid-Term project review successfully occurred @M22 - Microsoft, Portugal; (c) Courses and workshops both restricted to Consortium members and open to any interested researcher. The project organized 3 short courses restricted to Consortium members and 11 large workshops open to Consortium members and the general public; (d) Specific seminars and keynotes, given by the recruited experienced researchers and/or invited researches, to all interested staff members of each partner, particularly, inter-sectorial seminars to share knowledge and experiences. The project organized 15 seminars and talks; (e) International conferences: active presence in prestigious international conferences on speech, signal processing, human-computer interaction, cognitive sciences, etc. The collaborative activities of the project partners resulted in the publication of 101 scientific papers in top conferences and journals and the partners delivered also 15 academic theses; (f) Media coverage promoted by the industry partners and presentations of the project. The partners had the opportunity to present explicitly the project and its results in 16 occasions, in international events in Turkey, Spain, Portugal and Germany.
In the first 2 years of the project, the core objective of the collaborative activities was centered in “Advances in Human-Computer interaction methodologies”. To that end, the consortium produced the following core results, that can be seen as major scientific contributions of the project:
1. Definition of personas and use cases of the IRIS platform, referred to as the IRINA platform, (D1.1 D1.2).
2. Definition and early prototyping of the core set of input and output human-computer interaction modalities, to be addressed by the project (D2.1 D3.1): Input: speech recognition, silent speech recognition, gaze, pictograms; Output: speech synthesis, audiovisual speech synthesis, pictograms; Definition of data collection methodology to gather multimodal data.
3. Definition of voice and face biometrics (speaker and face recognition systems) for elder people and people with speech and physical disabilities (D5.1).
4. Definition of a service-based multimodal framework platform to support the fusion of the mentioned HCI modalities (D4.1 D4.2).
5. Definition of the general concepts and architecture of the IRINA platform, an augmentative and alternative communication system based in services hosted in the cloud (D6.1).
6. Definition of a set of recommendations (guidelines) concerning usability in multimodal interfaces design (D8.1).
The IRIS project was subject to a Mid-Term Project Review (MTR of the project, held at MSFT on M22), that was carried successfully. During the first semester of 2016, Microsoft decided to withdrawal from the project, due to the closing of its R&D activities in Portugal. The consortium took the decision of replacing Microsoft with ISCTE - University Institute of Lisbon (ISCTE-IUL), as the new project coordinator. INOVA (NOVAMAIS - Serviços de Consultadoria em Inovação Tecnológica, S.A.) a portuguese SME, entered the project as a new beneficiary, replacing Microsoft in leading the Dissemination and Exploitation activities. With this change, the project consortium consisted of 6 partners (2 industrial and 4 academic), that proceed with its networking activities, including a relevant number of outreach activities directed at the general public and events, paving the way for strong and durable repercussions in the scientific, technological, social and economic domains of our society. In the second 2 year part of the IRIS project, the core objective of the collaborative activities was centered in the theme, “moving to real world scenarios”, were we applied and piloted the proposed technologies into real world scenarios, achieving the following results:
1. Development of prototypes of Silent Speech Interface (SSI) systems (D2.2)
2. Development of an Audiovisual Speech Synthesis service based in an Anthropomorphic (Articulatory) approach that provides audiovisual synthesis of simple sentences in European Portuguese, illustrating its viability (D3.3). Development of an additional Audiovisual Speech Synthesis service with a rig-based advanced facial animation approach, in English and European Portuguese (D3.3).
3. Development of augmentative and alternative communication services and client app for the IRINA platform based in pictograms (seen as both input and output HCI modalities), targeting specially the communication needs of citizens with speech and autism spectrum disorder (D3.2).
4. Development of a specification and cloud-based service API for HCI modalities fusion, capability of configuring and performing combination of events from several modalities acting as the Interface (API) for the Programmer/Application developer, aligned with the W3C standard (D4.2 D4.2).
5. Development of state-of-the-art speaker and face recognition biometrics systems, deployed as SaaS – Software as a Service, with a simple protocol. The service allows access to biometric authentication, to any client app or third party platform. Voice and face biometrics provides a transparent method to authenticate people without experience with new technologies or with some impairments (D5.2).
6. Development of open and cloud-based IRINA platform and client, supporting the IRIS requirements and targeting specially the elderly and formal and informal caregivers. The platform supports collaborative care with multimodal (speech, gaze) communication services, shared calendar and tasks, safety advisor, well-being and everyday living support, including also health data collection. (D6.1 D6.2 D6.3).
Although we´ve carried a substantial range of usability evaluation user studies of our platform and service developments, taken in real world scenarios, where we achieve quite interesting and positive results describe in our project deliverables, some of the enumerated services, particularly, A (Silent Speech Interface), B (Audiovisual Speech Synthesis), C (Communication based in Pictograms) and E (Speaker and Face Biometrics), were evaluated with user studies in an autonomous way. In fact, the IRINA client developed in WP 6, was evaluated in real settings, while serving usage scenarios involving elderly and caregivers and not children with ASD, for example, which was evaluated with the project developed app Tolki. The fact that we could not count with the 12 months recruitment of INOVA in the last year of the project prevented us from achieving the optimal level of integration of all services available in the IRINA platform, although its service-oriented and cloud driven architecture is fully ready for such integration.
The IRIS project has been presented in different social media, newspapers and via the webpage http://www.iris-interaction.eu/(odnośnik otworzy się w nowym oknie)
IRIS progress towards achieving the objectives stated in the Annex I of the Grant, was carried by means of the following networking activities, that span throughout the 4 years of project duration: (a) Secondments and recruitments: a total of 111 secondments/months and recruitments/months were executed (corresponding to 72% of the planned figure), where 46 where Inter-sectorial-international secondments (71% of the planned figure), 22.8 where intra-national secondments (79% of the planned figure) and 42.25 referred to recruitments/months (70% of the planned figure); (b) Meetings: Partners have meet every month for quick online sync-up consortium meetings and once per year in scheduled physical meetings. Physical consortium meetings occurred 6 times during the project period: @ kickoff meeting, M1 – Microsoft, Portugal, @M09 - UNIZAR, Spain, @M15 – UAVR, Portugal, @M22 – MSFT, Portugal, @M35 – ISCTE-IUL, Portugal and @M41 – ISCTE-IUL, Portugal. A Mid-Term project review successfully occurred @M22 - Microsoft, Portugal; (c) Courses and workshops both restricted to Consortium members and open to any interested researcher. The project organized 3 short courses restricted to Consortium members and 11 large workshops open to Consortium members and the general public; (d) Specific seminars and keynotes, given by the recruited experienced researchers and/or invited researches, to all interested staff members of each partner, particularly, inter-sectorial seminars to share knowledge and experiences. The project organized 15 seminars and talks; (e) International conferences: active presence in prestigious international conferences on speech, signal processing, human-computer interaction, cognitive sciences, etc. The collaborative activities of the project partners resulted in the publication of 101 scientific papers in top conferences and journals and the partners delivered also 15 academic theses; (f) Media coverage promoted by the industry partners and presentations of the project. The partners had the opportunity to present explicitly the project and its results in 16 occasions, in international events in Turkey, Spain, Portugal and Germany.
In the first 2 years of the project, the core objective of the collaborative activities was centered in “Advances in Human-Computer interaction methodologies”. To that end, the consortium produced the following core results, that can be seen as major scientific contributions of the project:
1. Definition of personas and use cases of the IRIS platform, referred to as the IRINA platform, (D1.1 D1.2).
2. Definition and early prototyping of the core set of input and output human-computer interaction modalities, to be addressed by the project (D2.1 D3.1): Input: speech recognition, silent speech recognition, gaze, pictograms; Output: speech synthesis, audiovisual speech synthesis, pictograms; Definition of data collection methodology to gather multimodal data.
3. Definition of voice and face biometrics (speaker and face recognition systems) for elder people and people with speech and physical disabilities (D5.1).
4. Definition of a service-based multimodal framework platform to support the fusion of the mentioned HCI modalities (D4.1 D4.2).
5. Definition of the general concepts and architecture of the IRINA platform, an augmentative and alternative communication system based in services hosted in the cloud (D6.1).
6. Definition of a set of recommendations (guidelines) concerning usability in multimodal interfaces design (D8.1).
The IRIS project was subject to a Mid-Term Project Review (MTR of the project, held at MSFT on M22), that was carried successfully. During the first semester of 2016, Microsoft decided to withdrawal from the project, due to the closing of its R&D activities in Portugal. The consortium took the decision of replacing Microsoft with ISCTE - University Institute of Lisbon (ISCTE-IUL), as the new project coordinator. INOVA (NOVAMAIS - Serviços de Consultadoria em Inovação Tecnológica, S.A.) a portuguese SME, entered the project as a new beneficiary, replacing Microsoft in leading the Dissemination and Exploitation activities. With this change, the project consortium consisted of 6 partners (2 industrial and 4 academic), that proceed with its networking activities, including a relevant number of outreach activities directed at the general public and events, paving the way for strong and durable repercussions in the scientific, technological, social and economic domains of our society. In the second 2 year part of the IRIS project, the core objective of the collaborative activities was centered in the theme, “moving to real world scenarios”, were we applied and piloted the proposed technologies into real world scenarios, achieving the following results:
1. Development of prototypes of Silent Speech Interface (SSI) systems (D2.2)
2. Development of an Audiovisual Speech Synthesis service based in an Anthropomorphic (Articulatory) approach that provides audiovisual synthesis of simple sentences in European Portuguese, illustrating its viability (D3.3). Development of an additional Audiovisual Speech Synthesis service with a rig-based advanced facial animation approach, in English and European Portuguese (D3.3).
3. Development of augmentative and alternative communication services and client app for the IRINA platform based in pictograms (seen as both input and output HCI modalities), targeting specially the communication needs of citizens with speech and autism spectrum disorder (D3.2).
4. Development of a specification and cloud-based service API for HCI modalities fusion, capability of configuring and performing combination of events from several modalities acting as the Interface (API) for the Programmer/Application developer, aligned with the W3C standard (D4.2 D4.2).
5. Development of state-of-the-art speaker and face recognition biometrics systems, deployed as SaaS – Software as a Service, with a simple protocol. The service allows access to biometric authentication, to any client app or third party platform. Voice and face biometrics provides a transparent method to authenticate people without experience with new technologies or with some impairments (D5.2).
6. Development of open and cloud-based IRINA platform and client, supporting the IRIS requirements and targeting specially the elderly and formal and informal caregivers. The platform supports collaborative care with multimodal (speech, gaze) communication services, shared calendar and tasks, safety advisor, well-being and everyday living support, including also health data collection. (D6.1 D6.2 D6.3).
Although we´ve carried a substantial range of usability evaluation user studies of our platform and service developments, taken in real world scenarios, where we achieve quite interesting and positive results describe in our project deliverables, some of the enumerated services, particularly, A (Silent Speech Interface), B (Audiovisual Speech Synthesis), C (Communication based in Pictograms) and E (Speaker and Face Biometrics), were evaluated with user studies in an autonomous way. In fact, the IRINA client developed in WP 6, was evaluated in real settings, while serving usage scenarios involving elderly and caregivers and not children with ASD, for example, which was evaluated with the project developed app Tolki. The fact that we could not count with the 12 months recruitment of INOVA in the last year of the project prevented us from achieving the optimal level of integration of all services available in the IRINA platform, although its service-oriented and cloud driven architecture is fully ready for such integration.
The IRIS project has been presented in different social media, newspapers and via the webpage http://www.iris-interaction.eu/(odnośnik otworzy się w nowym oknie)