Final Report Summary - AISENSE (Human-Computer Interaction and Computer Vision for Improving Healthy Living of Elderly through Exer-gaming)
Usually, exergames are games full of exercises which would have a positive effect on balance ability of the players. However, most games are not developed for the elderly. Sometimes, games could be boring, or too fast and difficult to operate for them. Game scenarios composed of very simple set of instructions asking elderly to stand upright or sit down in a rehabilitation game might lead elderly people get bored very quickly and quit exercising. The exergaming system in this project aims to engage elderly people in mild exercising that wouldn’t extremely physically challenge them and entertain them while assessing their physical performance. Hence, Dr. Yalcin collaborated with professionals in the areas of Neurology and Physical Training in order to develop an exer-gaming scenario (AISENSE Exergaming System) that will be engaging, entertaining, increasing physical activity levels and has the capability of sustaining the attention of the subject.
The AISENSE project is developed around the idea of motivaving the player to move around the room searching for some objects and performing certain tasks with those objects. The exergaming scenario involves recognizing certain objects in a room equipped with depth sensors and cameras. The player is rewarded/penalized depending on whether he successfully accomplishes the subtasks of the game. The physical performance of the player is assessed both by evaluating his body pose throughout the exergaming duration as well as evaluating the measurements taken from a smartphone fastened securely on the body of the subject acting as a biosensor. Since the overall objective of AISENSE is to build an exergaming system that allows elderly people to naturally interact with a virtual environment through their body movements and assess their physical performance, the scenario is devised to engage elderly people in mild exercising that wouldn’t extremely physically challenge them and entertain them while assessing their physical performance.
Two critical components of AISENSE Exergaming system are human activity recognition and 3-D object recognition. A human-computer interaction needed for the AISENSE Exergaming system and it is achieved through recognition of the body movements of the subject using multiple numbers of vision sensor platforms which have conventional cameras and depth sensors on them. Human activity recognition has many applications in computer vision, including personal assistive robotics and smart homes/environments. Due to the large temporal and spatial variations in actions performed by humans, human action recognition has been a long-standing challenge. Throughout AISENSE project, Dr. Yalcin developed a method that recognizes certain human activities based on a motion descriptor that uses 3D human skeleton data. Motion descriptors (SHOJD) are defined using the 3D distance between the most frequent key poses that occur throughout the action that is intended to be recognized. Motion features are then fed into an artificial neural network for classification. Experimental results indicate that the SHOJD based human action recognition system is robust with high recognition rate.
The objects of interest in the exergaming environment are detected by the state-of-art 3-D object recognition algorithms. The research is focused on the developing of algorithms that allow different types of sensors to detect various aspects of the activity in the environment collectively. The AISENSE exergaming scenario is devised such that its objectives would establish a basis for developing state of art computer vision algorithms. One of the major challenges of the AISENSE project has been ensuring that spatial and temporal features obtained from multiple set of sensors to be processed simultaneously and developing fusion algorithms to combine the results obtained from different modalities and to improve the recognition rate by strengthening performance of each sensor by the results of the other sensors.
Many undergradute and graduate students have completed their thesis with themes that are subtopics of AISENSE project, learning to use point cloud library (PCL), OpenCV and OpenNI libraries for 3-d scene understanding, 3-d object recognition with depth sensors. Dr. Yalcin has been guiding them to develop algorithms for 3D object recognition, human body recognition and gesture recognition using RGB-D data acquired from depth sensors. Research carried out through AISENSE project complements the high quality research carried out in the department, which includes the computer vision, pattern recognition, robotics areas as well as areas like signal processing, partial diffential equations, time series analysis, and numerical methods due to her Electrical Engineering and Applied Mathematics background. Dr. Yalcin has been collaborating with other young faculty both in Electrical Engineering and Computer Engineering departments to apply for project proposals to be submitted to several national research funds to develop user-friendly multimodal human-computer interaction techniques such as body, face and facial gesture recognition for an assisted living smart space of the elderly.
Outreach activities through AISENSE project encompassed broad interactions with the general public in order to promote the communication between the scientific community and the general public and increase awareness of science. Part of the outreach activities was implemented through TUBITAK (The Scientific and Technological Research Council of Turkey). AISENSE project increased public awareness in nearby local hospitals and the municipality about improving the life quality of elderly people with exergaming.
Dr. Yalcin continues to work on the research initiated by AISENSE project to have an exergaming environment where different components of the exergaming system are working seamlessly together.