Final Report Summary - QFATIGUE (Quantification of mental fatigue by means of visual and physiological measures)
In neuromuscular rehabilitation, mental fatigue of a patient is closely related to effectiveness, usability and attractiveness of the rehabilitation process itself. It reflects possible patient non-compliance which is commonly believed to be one of the main reasons for low-efficiency of current neurorehabilitation techniques. The phenomenon of mental fatigue has been addressed by many studies investigating the association between brain electrical activity using electroencephalography (EEG) and the onset of fatigue symptoms. It has been demonstrated that EEG activity reflects differences in processing of attended and unattended information and that deterioration of selective attention leads to decreased ability of patients to focus their attention to rehabilitation.
However, EEG-based systems are still relatively new, they suffer from low signal-to-noise ratio and often provide unreliable results. On the other hand, several visual signs of fatigue can be conveniently detected by video monitoring of patient's face, for example eye blinking rate, pupil diameter changes, etc.
The main objectives of the qFATIGUE project were to design and develop a multimodal bio-feedback interface for simultaneous extraction of mental fatigue from video-based and EEG-based monitoring of a person. To our knowledge, in the field of neurorehabilitation, the visual and physiological measures of mental fatigue have never been systematically compared in different training environments and combined into a joint measure of mental fatigue. Our project represents one of the first attempts to integrate EEG-based and vision-based mental fatigue detection in order to monitor patient’s compliance during rehabilitation process. Its main impact is in merging of EEG-based and video-based information with the purpose of detecting mental fatigue, and in augmenting the unreliable EEG data with additional modality. Tools developed during the project can help with time consuming rehabilitation procedures, by increasing their efficiency and by making the rehabilitation more adaptable to patient’s abilities, as well as his/her acceptance of the rehabilitation process.
Conducting the qFATIGUE project we organized experimental measurements in cooperation with Institute of Knowledge Discovery from Technical University in Graz, Austria. Volunteers performed several walking tasks in a robotic gait trainer, during which they got physically and mentally fatigued. We recorded their EEG and signals as well as frontal video of their face. Videos were processed to detect frontal faces and facial features based on Active Appearance Model (AAM). For EEG signals we performed noise and artefact removal using routines developed during the course of the project. The analysis of power spectral values of different cortical regions in standard brainwave bands showed significant differences between non-fatigued and fatigued brain activity.
We selected the most informative fatigue metrics from both modalities: for video modality we used eye blinking statistics and relative distances between eye components and eyebrows; for EEG modality we used increases in spectrum power in selected frequency bands located in different regions of the brain. By joining those metrics we developed a multimodal fatigue model that can be used to detect when a person is tired.
The qFATIGUE project was executed in accordance with Description of Work and there were no major deviations from our initial plans. During the whole period of the project we took great care to adhere to all best practices regarding the ethical/privacy issues and obtained consent for our research from Slovenian National Medical Ethics Committee.
During the project Dr. Divjak worked closely with a small group of researchers, as required by each specific project task. Throughout the project he had a high level of professional independence and good support from the host institution, including all the hardware and facilities needed for successful completion of the project.
Dr. Divjak was involved in preparation of four EU FP7 projects (TREMOR-EEU, BETTER-EEU, NeuroTREMOR, EXPRESS) of which three were fully funded and successfully implemented. He also participated in preparation of project entitled “SAMinZDRAV: Home Devices in Support of Independent and Healthy Living” for the EU regional Competency Center for Biomedical Engineering in Slovenia.
During the project Dr. Divjak prepared or was involved with 12 publications related to the project and four papers are currently in preparation. Other transfer of knowledge activities included mainly presentations at conferences in Slovenia and abroad and mentoring students of Faculty of Electrical Engineering and Computer Science in Maribor, particularly on topics of computer vision, facial feature detection, 3D reconstruction and EEG signal processing.
The qFATIGUE project website is available at http://lspo.feri.um.si/qFATIGUE/.
However, EEG-based systems are still relatively new, they suffer from low signal-to-noise ratio and often provide unreliable results. On the other hand, several visual signs of fatigue can be conveniently detected by video monitoring of patient's face, for example eye blinking rate, pupil diameter changes, etc.
The main objectives of the qFATIGUE project were to design and develop a multimodal bio-feedback interface for simultaneous extraction of mental fatigue from video-based and EEG-based monitoring of a person. To our knowledge, in the field of neurorehabilitation, the visual and physiological measures of mental fatigue have never been systematically compared in different training environments and combined into a joint measure of mental fatigue. Our project represents one of the first attempts to integrate EEG-based and vision-based mental fatigue detection in order to monitor patient’s compliance during rehabilitation process. Its main impact is in merging of EEG-based and video-based information with the purpose of detecting mental fatigue, and in augmenting the unreliable EEG data with additional modality. Tools developed during the project can help with time consuming rehabilitation procedures, by increasing their efficiency and by making the rehabilitation more adaptable to patient’s abilities, as well as his/her acceptance of the rehabilitation process.
Conducting the qFATIGUE project we organized experimental measurements in cooperation with Institute of Knowledge Discovery from Technical University in Graz, Austria. Volunteers performed several walking tasks in a robotic gait trainer, during which they got physically and mentally fatigued. We recorded their EEG and signals as well as frontal video of their face. Videos were processed to detect frontal faces and facial features based on Active Appearance Model (AAM). For EEG signals we performed noise and artefact removal using routines developed during the course of the project. The analysis of power spectral values of different cortical regions in standard brainwave bands showed significant differences between non-fatigued and fatigued brain activity.
We selected the most informative fatigue metrics from both modalities: for video modality we used eye blinking statistics and relative distances between eye components and eyebrows; for EEG modality we used increases in spectrum power in selected frequency bands located in different regions of the brain. By joining those metrics we developed a multimodal fatigue model that can be used to detect when a person is tired.
The qFATIGUE project was executed in accordance with Description of Work and there were no major deviations from our initial plans. During the whole period of the project we took great care to adhere to all best practices regarding the ethical/privacy issues and obtained consent for our research from Slovenian National Medical Ethics Committee.
During the project Dr. Divjak worked closely with a small group of researchers, as required by each specific project task. Throughout the project he had a high level of professional independence and good support from the host institution, including all the hardware and facilities needed for successful completion of the project.
Dr. Divjak was involved in preparation of four EU FP7 projects (TREMOR-EEU, BETTER-EEU, NeuroTREMOR, EXPRESS) of which three were fully funded and successfully implemented. He also participated in preparation of project entitled “SAMinZDRAV: Home Devices in Support of Independent and Healthy Living” for the EU regional Competency Center for Biomedical Engineering in Slovenia.
During the project Dr. Divjak prepared or was involved with 12 publications related to the project and four papers are currently in preparation. Other transfer of knowledge activities included mainly presentations at conferences in Slovenia and abroad and mentoring students of Faculty of Electrical Engineering and Computer Science in Maribor, particularly on topics of computer vision, facial feature detection, 3D reconstruction and EEG signal processing.
The qFATIGUE project website is available at http://lspo.feri.um.si/qFATIGUE/.