Predictive Neural Information for Proactive Actions: From Monkey Brain to Smart House Control

Project Information

Plan4Act

Grant agreement ID: 732266

DOI

10.3030/732266

Project closed

EC signature date 12 October 2016

Start date 1 January 2017

End date 30 April 2021

Funded under

• EXCELLENT SCIENCE - Future and Emerging Technologies (FET)

Total cost

€ 4 236 000,00

EU contribution

€ 4 236 000,00

Coordinated by

GEORG-AUGUST-UNIVERSITAT GOTTINGEN STIFTUNG OFFENTLICHEN RECHTS
Germany

Periodic Reporting for period 3 - Plan4Act (Predictive Neural Information for Proactive Actions: From Monkey Brain to Smart House Control)

Reporting period: 2019-07-01 to 2021-04-30

Summary of the context and overall objectives of the project

Humans are very often proactively plan ahead when performing a sequence of actions and we usually know already at the beginning of an action sequence, which action has to happen at the end. This helps us to perform action chains smoothly and seamlessly. Robots cannot do this and their actions often appear chunked and clumsy in our eyes. Similarly, also in other technical systems smooth action chaining is difficult, while – on the other hand – such a property is especially desired for all systems that directly have to interact with humans. The main goal of Plan4Act is to exploit the proactive planning behavior of a human to obtain a faster reaction of – in this case – a Smart House. For this, are extracting sequence-predicting knowledge from brain activity to provide the smart devices inside the house with more time to react and, if necessary, to support the planned action. Our future vision is that such proactive devices may endow human patients with the ability to plan a daily-life goal – like making coffee – and let the smart environment achieve it without having to trigger onerously, one by one every single individual action to reach this goal. As a first step towards such a system, Plan4Act develops a prototype of an automated brain-machine interface (BMI) being able to extract and process the signatures of an action plan with which to control different features in a smart house proactively.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In order to extract sequence-predicting knowledge from the brain activity, we have set up the experimental environment and defined protocols for enforcing that the observed agent needs to employ mental planning when performing sequences of actions. The environment is equipped with a multitude of sensors such as touch sensors and cameras and a wireless recording device to monitor the ongoing brain activity during planning behavior. Different action-sequence protocols with and without behavioral distractors have been created to investigate and visualize planning activity in the brain. In addition, we developed a simulation environment consisting of all relevant components from the behaving agent to the smart house and its devices. This simulation environment allows us to perform faster and more general development of the intended BMI-system and provides direct opportunities to test the system before running tests in real situations. Methods have been developed to analyze the resulting (simulated or real) neural activity to extract information about proactive planning. Analysis methods address the problem of day-by-day signal variability by using a novel method for adapting and correcting changed signals. A signal-classifier based on an artificial neural network is used to interpret these signals and to create control commands for the smart house environment. To speed up computations the signal-interpreting ANN-software has been translated into FPGA-based hardware, which words as edge device in the smart house environment. In addition to this, a novel tool for fast reprogramming and adapting the FPGA has been developed by which changes to the analysis can be implemented in minutes as compared to hours/days before. The experimental protocol used to generate proactive planning activity in the brain have been match in the smart house by developing a novel device (smart cabinet) with four shelf spaces that can be accessed by sequential rotate-and-door-open actions. This implements a 4-branch, 2-stage decision tree, which is identical to the protocols used in the experiment. For this we have by now implemented the complete analysis-to-control workflow in software as well as in hardware to use neural signals to control smart house devices.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

In general, the project will lead to a set of tools consisting of methods, algorithms, and hardware to extract and use planning knowledge from brain activity to control smart devices. This will yield a radically novel control paradigm potentially useful for healthcare problems as controlling a wheel chair or prostheses in a more “human” way. During the development of this control paradigm, we are obtaining crucial insights in the way the brain forms and encodes predictive, planning knowledge, which could become important to understand the effects of diverse neural diseases impeding patients’ ability of forming and executing plans. Thereby, in contrast to state-of-the-art setups, our newly developed experimental environment does not restrict the agent in its movements and actions and, in parallel, allows the measurement of all relevant neural and behavioral data. This environment provides a new fundamental platform to investigate natural behavior and related brain activities. Furthermore, we develop new types of hardware-accelerated interfaces between a smart house, its smart devices, and the user enabling a smoother, more natural interaction between them. Overall, this project seeks to improve the interplay between humans and surrounding devices by equipping them with the ability to act proactively.

plan4actimage.png