How computers can read body language
An EU-funded group of scientists has invented a host of innovative solutions ranging from escalator safety to online marketing that advances people's communication with computers. The MIAUCE ('Multimodal interactions analysis and exploration of users within a Controlled Environment') project received just over EUR 2 million under the 'Information society technologies' (IST) Thematic area of the Sixth Framework Programme (FP6) to evaluate and develop techniques to analyse the multi-modal behaviour of users within the context of real applications. MIAUCE project coordinator Chaabane Djeraba from the Centre national de la recherche scientifique (CNRS) in France explained that he and his team wanted to imagine a world where hidden computers try to anticipate human needs and then develop various applications that could operate in such a universe. 'The motivation of the project is to put humans in the loop of interaction between the computer and their environment,' he said. 'We would like to have a form of ambient intelligence where computers are completely hidden.' Professor Djeraba said that this would mean the existence of a 'multimodal interface so people can interact with their environment. The computer sees their behaviour and then extracts information useful for the user'. MIAUCE has developed concrete prototypes of three kinds of such applications. The first monitors the safety of crowds at busy places like airports and shopping centres with surveillance cameras being used to detect various situations including accidents on escalators. 'The background technology of this research is based on computer vision,' Professor Djeraba said. 'We extract information from videos. This is the basic technology and technical method we use.' The video stream is then analysed in real time to extract a hierarchy of three levels of features: starting at a mathematical description of shapes, movements and flows, before moving to descriptions of crowd density, speed and direction, and finally the computer is able to decide when the activity becomes 'abnormal' because, for example, someone has fallen on an escalator and caused a pile-up that needs urgent intervention. The team is now working with a manufacturer of escalators to increase existing video monitoring systems at international airports. If a collapse can be detected automatically then the seconds saved in responding could save lives as well, according to the researchers. A second application for this technology could be in marketing, specifically to monitor how customers behave in shops. Researchers are now developing two products to this end: one will be a 'people counter' to monitor pedestrian flows in the street outside a shop, which they expect to be particularly attractive to fashion stores who wish to attract passers-by. Another is a 'heat map generator' to watch the movements of people inside the store, so that the manager can see which parts of the displays are attracting the most attention. The third application addressed by the scientists is interactive web television, allowing viewers to select what they want to watch. As part of the project, viewers' webcams are used to monitor their faces to see which part of the screen they are looking at. The researchers said the results could be used to feed users further information based on their interests. Project partner Tilde, a software company based in Latvia, is already commercialising this application. Such inventions clearly give rise to a plethora of ethical and legal questions. But Professor Djeraba said his team has addressed such concerns primarily by 'anonymising' people. 'Generally speaking, anonymity is the critical point,' he explained. 'If we anonymise it's OK, if we don't anonymise it's not OK.'