Feature Stories - ICT to fostering inter-family relationships, naturally
'There is a lot of technology out there that helps individuals communicate with each other, whether it's a simple phone call or a video conferencing application. What's lacking are applications that enable groups of people in different places to communicate and interact in a natural way,' explains Dr. Doug Williams, broadband research project director at BT in the United Kingdom (UK). That gap is being filled by applications developed by the 'Together anywhere, together anytime' (TA2) project. Their solution combines state-of-the-art artificial intelligence (AI) with ambient intelligence, multimedia tools and audio and video capturing, encoding, processing and transmission to enable near-natural interaction and communication between dispersed groups of people. Enhanced audio, automated multi-camera editing, online games facilities and 'availability' signals have been integrated into a new type of system that makes communication easy to initiate and intuitive to use - to try to achieve an experience similar to family members all being in the same room. One of the key differences between the TA2 approach and video conferencing used in business environments is that the researchers have strived to make communication and interaction as natural as possible. Supported by EUR 12.8 million in funding from the European Commission, the researchers and developers from 14 industrial and academic partners in eight countries directly addressed the question of how technology can help nurture group-to-group relationships. Using multiple internet-connected cameras, microphones and speakers in the lounge of a family home, the system goes several steps beyond standard video conferencing by enabling different groups of people in two, three or more locations to interact and enjoy using shared applications. Artificial intelligence analyses activity in each location, and an AI engine makes decisions about how best to represent what's going on, selecting the optimum camera and camera angle, providing wide-angle shots during lulls in the conversation or focusing on specific events taking place. In essence, this AI 'virtual director' fills the role of a real life cameraman or director in the TV or cinema industry. A 'visual composition engine', jointly developed by BT and Dutch project partner CWI, handles the video decoding, composition and rendering, enabling the system to seamlessly switch between up to five simultaneous HD video streams. 'Using the system is a bit like watching a movie or a TV talk show except the people you are watching are interacting with you and others in real time,' Dr. Williams, the technical director of the TA2 project, says. 'It can be used simply for two families to talk and catch up with each other, or they can use multimedia tools to share photos and watch videos together, or they can play games together.' 'The impact of audio must never be underestimated,' he adds. 'In TA2 we use an "audio communication engine" developed by Fraunhofer IIS: it delivers Full-HD Voice and captures all audible frequencies - compared to a normal telephone which captures only about one quarter of the frequencies we can hear. And because we capture sound with more than one microphone, we are better able to reproduce the sound field at the remote ends. All this helps make the interactions easier, more natural and, as an added bonus, due to something called 'multimodal effects', the better audio makes people believe the image quality is better too.' Fun, games and eLearning Digital versions of traditional family games such as Memory, Ludo or Pictionary can also be played between dispersed families using the TA2 system, and parents or grandparents can use it to read bedtime stories to their children or grandchildren via tablet computers when they are away. 'In the real world, people usually get together around some sort of activity, whether it's dinner, a movie, showing holiday photos or playing a game, we are trying to reflect that sort of natural interaction in our system,' Dr. Williams says. 'When you get involved with communicating and interacting on the TA2 system, it's fun, it all makes sense. But one thing we have struggled with is how such interactions would start. How would someone know when friends or family wanted to interact?' To address that question, the TA2 researchers incorporated ambient intelligence using sensors and a system of coloured lights to subtly let friends and family members elsewhere know when people are available, while at the same time protecting users' privacy. Another application developed by the TA2 researchers focuses on using the technology for learning to play a musical instrument, though it could equally be used for learning other skills that require both visual and audio communication. 'A lot of the components of the system are already available commercially, although we adapted and improved many of them to suit our purposes,' Dr. Williams emphasises. The TA2 researchers conducted long-term trials in user homes in Sweden, at a music school in the UK and conducted numerous lab tests in special purpose 'sitting rooms' built at project partner facilities in Belgium, the Netherlands and the UK. They also conducted a survey, asking more than 2,000 people about the ways in which they think they would use the new technology (and in particular tablet computers) as part of a home-based video communication system. 'Users really warmed to the idea of reading a story to a child, or to occasionally going through photos together with a friend, all while being able to see and hear each other with a good audio video link,' Dr. Williams says. 'However, as with most new technology it's not always clear how it will be used until it is on the market.' In that vein, Dr. Williams sees the potential for a TV production company to employ the system or elements of it for a new style of TV game show in which people would be able to participate from within their own homes or from communal areas such as pubs and cafes. 'I think it's safe to say that this sort of system or something like it will start to be used widely within the next five years,' he says. TA2 received research funding under the European Union's Seventh Framework Programme (FP7). Useful links: - 'Together Anywhere, Together Anytime' website - The TA2 project factsheet on CORDIS Related articles: - How technology can bring people together