Periodic Reporting for period 3 - AIDE (Adaptive Multimodal Interfaces to Assist Disabled People in Daily Activities) Reporting period: 2017-02-01 to 2018-05-31 Summary of the context and overall objectives of the project Around 80 million people in the EU, a sixth of its population, have a disability. They are often hindered from full social and economic participation by various barriers physical, psychological and social factors. Nowadays, the recent trends in assistive technology for supporting activities of daily living (ADL), mobility, communication and so on are based on the integration of the capabilities of the user and the assistive technologies.AIDE project has achieved its challenging objective of developing and testing a revolutionary modular and adaptive multimodal interface customizable to the individual needs of people with disabilities. A totally new shared-control paradigm for assistive devices that integrates information from identification of residual abilities, behaviors, emotional state and intentions of the user on one hand and analysis of the environment and context factors on the other hand has been developed and successfully tested with 29 subjects affected by different pathologies (Figure 1).A series of applications for the AIDE system have been developed across several domains in which disabled people could greatly benefit: Communication, Environmental control, Wearable robots and Entertainment. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far During the first period, the ethical guidelines to be adopted within AIDE were defined. After that, the target scenarios were identified by end-users’ focus groups. Specifically, four scenarios emerged gathering the user requirements of the different focus groups: 1) communication; 2) Environmental control; 3) Hygiene task; and 4) Preparing and eating a meal. Based on end-users’ requirements in each scenario, AIDE system were characterized and specified. One of the main results of the first period was the prototype of a wearable and wireless system for biosignal recording and processing to be used as a key element in the AIDE multimodal sensory processing system. The main outcomes of the second year are:- Cedar Foundation obtained the ethical approval for testing AIDE system. In addition, three experimental sessions were conducted to monitor the design and developments of AIDE hardware/software components.- A modular standard architecture for AIDE multi-modal interface based on the messaging system Yet Another Robotic Platform (YARP) has been developed, completed and tested. - A novel shoulder-elbow robotic exoskeleton attached to a wheel chair fixed system with the possibility of using it for either the left or the right arm, according to the user's residual motion capabilities. To complete AIDE exoskeleton, a new prono-supination and hand assistance exoskeleton has been developed, integrated and tested.- A universal system to control the movements of electric wheelchairs linked with AIDE interfaces has been developed and tested. - Two intention detection methods based on: 1) hybrid EEG/EOG system; and 2) EMG activity. - Algorithms to detect and track the 3D position and orientation of texture-less objects and Gaze estimation algorithms.- Algorithms to recognize the user activity in real ADL tasks: Support Vector Machine (SVM), Artificial Neural Network (ANN) and Decision Tree (DT) algorithms. - Development and optimization of the Low Level Controller of AIDE arm exoskeleton. - A motion planning system grounded on a Learning by Demonstration approach and Dynamic Movement Primitives (DMP) was developed. -The Finite-State Machine (FSM) was enriched with additional modules, to give the user full control of the task execution. - A first version of the HLC and LLC for communication, control and entertainment was developed.During the third period, the main outcomes are:-An operative prototype of the AIDE exoskeleton was delivered and obtained all the certifications needed for be tested with human subjects.-The implementation of user’s intention detection realized by fusing the physiological signals of EEG and EOG was successfully accomplished and documented in a published paper in Nature Scientific Reports (Figure 2).-EMG analysis focused on the onset detection algorithm was developed, implemented in real time and tested with healthy subjects.-To estimate and compute the current position of the users in their home setting, two solutions have developed and tested: 1) based on WiFi Fingerprinting techniques; and 2) based on laser range-finders.-The performance of the developed algorithms to estimate the 3D pose of texture-less objects and gaze point can be ranked as optimum regarding KPIs of the project.-Algorithms to recognize the user activity in real ADL tasks were tested and achieved an accuracy about 87%.-Hierarchical Learning control architecture was completed and tested. The achieved results showed a 100% success rate in the task fulfillment, with a high level of generalization with respect to the environment variability.-The finite-state machine (FSM) for the AIDE shared control system was enriched with new features (Figure 2).-A successful integration of the final prototype.-The AIDE validation testing was completed onsite at Cedar Foundation premises in Belfast Northern Ireland with a total of 17 participants Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) AIDE project has produced a high impact in the scientific community with a total of 30 papers published in scientific journals. Moreover, the media coverage of the project only during the third reporting period sums up for more than 2.300.000 people, being the major contribution to this cipher is due to the appearances on TV. AIDE project has produced 6 technologies identified as candidates to be protected, one of them already filled a patent. Moreover, AIDE project has produced a set of six products as the most interesting to be commercially exploited. These products can be classified into interfaces and algorithms, exoskeletons and robotized wheelchair. Companies from inside and outside the consortium have already expressed their interest to establish an agreement for the exploitation of these results.The fully autonomous AIDE final prototype is a first step to bring this technology to the market in the near future, which will produce a clear benefit for the full social and economic participation of people with disabilities (Figure 3). Final AIDE fully autonomous prototype Validation of AIDE prototype Finite-state machine (FSM) user’s intention detection fusing EEG and EOG
