Periodic Report Summary 1 - DUALGAZE (Social cognition in gaze-based interaction)
The overarching objective of the DUALGAZE project and the primary research focus of the Social INteraction and Consciousness lab (www.sinclab.org) is to develop a behavioural account of how human understanding of others is both shaped by, and expressed in our most crucial capacity, on-going social interaction. Social interaction ostensibly adds a different quality to how we think about others —in fact, we possibly "think about others" very little during ongoing interaction— but what is added? There are two candidates that spring to mind: (1) interaction allows us to observe the effect of our actions on and in others, indeed, the actions of the other are this effect; (2) interaction allows for the development of complex dynamics, whereby individuals become a system, any state of which becomes irreducible to a linear combination of its constituents: you can’t explain the system just by looking at its components. We implement interaction methodologically by focusing on eye-gaze as one of the most salient and rich components of human communication. We look at how people experience and respond to others’ reaction to their own gaze, as well as dynamic gaze-based interactions, measured via “interactive eye-tracking”.
Gaze & reward.
In a first set of studies we have looked specifically at what it does to me to have my gaze followed. We show that we find this rewarding, and in fact this reward can make us learn behaviours. That is, my brain interprets someone else following my gaze the same way as being given money. In one experiment, people see a face on the screen. On the left or right of that face, with equal probability, a small ball can appear, and when it does, people should shift their gaze towards it as quickly as possible. Now, when they do this fast enough, they get rewarded. Some people by money, and others... by the face following their gaze! Crucially however, the sides have a different pay-off. On one side, however fast they are, they get almost no reward, while on the other they get a lot. If the quantity of reward makes a difference, people should make increasingly faster shifts towards the highly-rewarded ball. And if our hypothesis about self-initiated joint attention being rewarding is correct, this should happen not just for money, but for gaze as well — which is exactly what we found.
Why is this important? Theoretically, because if motor learning can occur simply by having someone else respond to my actions (such as following my gaze), via reward, it provides a possible mechanism through which we learn social skills: our brain likes social interaction, and because it comes back for more, it gets better at it. If you love playing the piano, you will practice more, and you will get better at it. Here, the point is to prove we love, so to speak, the equivalent of playing the piano, social interaction. Practically it is important, because it may provide a way to diagnose psychiatric disorders, which are usually characterised by disturbances of social interaction, and probably, for instance in autism, an inability to experience this reward. This paradigm could be an efficient way of finding out whether someone does not feel motivated to interact.
Finally, it is important because we can use this rewarding nature of interaction to reduce bias and stereotypes. On such effect is the Cross Race Effect, the phenomenon whereby we have better memory for faces of our own race than that of others, which in court has led many a white person to mistake one black person for another. In an experiment we are currently running, we look at whether having people follow your gaze can reduce this: if someone from a different racial background follows your gaze, does your memory for their face improve?
Dynamic interaction.
Another essential aspect of interaction is that when two people engage in interaction, they become entrained: one person’s behaviour starts to be dependent on the other’s, which starts to be dependent on that of the first, etc. Usually, these dynamics cannot be reduced to the parts, the individuals in an interaction. In order to look at how these interactions develop, we developed a new method called Dual interactive eye tracking with Virtual anthropomorphic Avatars, DiVA in short. It’s a long name for people interacting with each other but though artificial faces on a screen. The face I’m looking at displays in real time the gaze of the other person, and vice versa.
What we want to look at is how the dynamics between people’s gaze in a certain task determine how they will decide, whether they will follow the other’s judgment more, etc. We compare a human face with a cursor displaying the other person’s gaze, because that’s what almost all previous studies on gaze dynamics have done. We find that whereas seeing a person’s gaze as a cursor may be more informative, when you look at the dynamics, they are more elaborate when there’s a face.
But what kind of face? Do we look at avatars in the same way as at real faces? Someone looking you in the eye will be perceived as more attractive and more likeable... but again, many studies have been done with avatars, and certainly all of them with pictures. But an avatar, or a picture, doesn’t look back! So, in another set of experiments, we looked at whether, when people look at videos of other people, this positive effect of direct gaze remains, and then one more step: what happens when people are given the illusion that they’re interacting live via a video link with someone? Turns out the “longer direct gaze = more likeable”-effect disappears, and remains only when people watch films of others and are aware of what the study is about. However, in a real interaction context, while there is no effect of increased direct gaze, people generally find the other more likeable... pointing again in the direction of interaction as rewarding!
A night in the pub above texting, and above Skype?
What can this bring us? For one, it can show the mechanisms behind something that has been known for a long time in social psychology, namely that being online via PC/smartphone instead of in live interaction may lead to deindividuation, less empathy, more extreme opinions. Ever felt the unease because on Skype you can’t look the other in the eye? Our studies suggest that in the end a crucial aspect to social communication is that for it to be effective, effective in eradicating hate or violence, it has to be real, live, and dynamics have to develop, because our brain loves it. And perhaps the devastating effect of disturbances in this, is shown in the suffering that comes with mental disorders. If we can understand more fully what interaction brings us, we could perhaps use it for the good.
Gaze & reward.
In a first set of studies we have looked specifically at what it does to me to have my gaze followed. We show that we find this rewarding, and in fact this reward can make us learn behaviours. That is, my brain interprets someone else following my gaze the same way as being given money. In one experiment, people see a face on the screen. On the left or right of that face, with equal probability, a small ball can appear, and when it does, people should shift their gaze towards it as quickly as possible. Now, when they do this fast enough, they get rewarded. Some people by money, and others... by the face following their gaze! Crucially however, the sides have a different pay-off. On one side, however fast they are, they get almost no reward, while on the other they get a lot. If the quantity of reward makes a difference, people should make increasingly faster shifts towards the highly-rewarded ball. And if our hypothesis about self-initiated joint attention being rewarding is correct, this should happen not just for money, but for gaze as well — which is exactly what we found.
Why is this important? Theoretically, because if motor learning can occur simply by having someone else respond to my actions (such as following my gaze), via reward, it provides a possible mechanism through which we learn social skills: our brain likes social interaction, and because it comes back for more, it gets better at it. If you love playing the piano, you will practice more, and you will get better at it. Here, the point is to prove we love, so to speak, the equivalent of playing the piano, social interaction. Practically it is important, because it may provide a way to diagnose psychiatric disorders, which are usually characterised by disturbances of social interaction, and probably, for instance in autism, an inability to experience this reward. This paradigm could be an efficient way of finding out whether someone does not feel motivated to interact.
Finally, it is important because we can use this rewarding nature of interaction to reduce bias and stereotypes. On such effect is the Cross Race Effect, the phenomenon whereby we have better memory for faces of our own race than that of others, which in court has led many a white person to mistake one black person for another. In an experiment we are currently running, we look at whether having people follow your gaze can reduce this: if someone from a different racial background follows your gaze, does your memory for their face improve?
Dynamic interaction.
Another essential aspect of interaction is that when two people engage in interaction, they become entrained: one person’s behaviour starts to be dependent on the other’s, which starts to be dependent on that of the first, etc. Usually, these dynamics cannot be reduced to the parts, the individuals in an interaction. In order to look at how these interactions develop, we developed a new method called Dual interactive eye tracking with Virtual anthropomorphic Avatars, DiVA in short. It’s a long name for people interacting with each other but though artificial faces on a screen. The face I’m looking at displays in real time the gaze of the other person, and vice versa.
What we want to look at is how the dynamics between people’s gaze in a certain task determine how they will decide, whether they will follow the other’s judgment more, etc. We compare a human face with a cursor displaying the other person’s gaze, because that’s what almost all previous studies on gaze dynamics have done. We find that whereas seeing a person’s gaze as a cursor may be more informative, when you look at the dynamics, they are more elaborate when there’s a face.
But what kind of face? Do we look at avatars in the same way as at real faces? Someone looking you in the eye will be perceived as more attractive and more likeable... but again, many studies have been done with avatars, and certainly all of them with pictures. But an avatar, or a picture, doesn’t look back! So, in another set of experiments, we looked at whether, when people look at videos of other people, this positive effect of direct gaze remains, and then one more step: what happens when people are given the illusion that they’re interacting live via a video link with someone? Turns out the “longer direct gaze = more likeable”-effect disappears, and remains only when people watch films of others and are aware of what the study is about. However, in a real interaction context, while there is no effect of increased direct gaze, people generally find the other more likeable... pointing again in the direction of interaction as rewarding!
A night in the pub above texting, and above Skype?
What can this bring us? For one, it can show the mechanisms behind something that has been known for a long time in social psychology, namely that being online via PC/smartphone instead of in live interaction may lead to deindividuation, less empathy, more extreme opinions. Ever felt the unease because on Skype you can’t look the other in the eye? Our studies suggest that in the end a crucial aspect to social communication is that for it to be effective, effective in eradicating hate or violence, it has to be real, live, and dynamics have to develop, because our brain loves it. And perhaps the devastating effect of disturbances in this, is shown in the suffering that comes with mental disorders. If we can understand more fully what interaction brings us, we could perhaps use it for the good.