GeoViSense: Towards a transdisciplinary human sensor science of human visuo-spatial decision making with geographic information displays

Project Information

GeoViSense

Grant agreement ID: 740426

DOI

10.3030/740426

Project closed

EC signature date 19 July 2017

Start date 1 November 2017

End date 31 October 2023

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 2 500 000,00

EU contribution

€ 2 500 000,00

Coordinated by

UNIVERSITAT ZURICH
Switzerland

Periodic Reporting for period 4 - GeoViSense (GeoViSense: Towards a transdisciplinary human sensor science of human visuo-spatial decision making with geographic information displays)

Reporting period: 2022-05-01 to 2023-10-31

Summary of the context and overall objectives of the project

Geographic information displays (GIDs) such as mobile maps with navigation guidance are an integral part of everyday life. When seeking directions for a new restaurant or the quickest connection to the office, people will retrieve their smartphones to check their location and navigate towards the destination almost effortlessly. Does the ease with which we can now find our way through assistance come with a price? The GeoViSense research team has been interested to assess the relationship between technology and human abilities; the adaptation to and of technology to fit specific needs, and the personalization of technology for specific types of users and use contexts.
For this we proposed studying people’s map-assisted navigation behavior in real and virtual urban environments, in various situations and use contexts. While behavior in virtual environments is easier to control and predict, studies outdoors are rare, because they are much harder to control. However, these allow us to determine the extent to which our predictions generalize to everyday situations. Mobile physiological devices, e.g. galvanic skin response (GSR) sensors, eye trackers (ET), electroencephalograms (EEG), etc. can be validated in controlled virtual reality, but only recently even conceivable, also to investigate in-situ real world behavior outdoors, as we are able to demonstrate.
GeoViSense aimed to:
. integrate human-visualization-environment research across the sciences (i.e. natural, social/behavioral, engineering, etc.)
. develop missing, empirically evaluated design guidelines for human-computer interfaces to support pedestrian mobility in urban environments, including affective, effective, and efficient spatio-temporal decision-making and spatial learning
. develop unconventional evaluation methods to assess perceptual, cognitive, psycho-physiological, and display design factors across broad ranges of users and mobile urban use contexts, and
. scale up empirical methods, from to-date controlled behavioral lab paradigms, towards a new in-situ mobile human sensor science.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

With backgrounds in geography, GIScience, cognitive-neuroscience, and engineering we first built novel research infrastructure for mobile outdoors and indoor virtual reality (VR) navigation experiments. We developed code for human sensing (i.e. ET, EEG, GSR) to be deployed in VR and outdoors. VR and mobile testbed set ups and code development is made openly accessible, after publication.
We completed one indoor navigation study in an urban VR and two outdoor navigation studies with difficult to recruit expert navigators (Swiss Army personnel). Recruitment was complemented by non-expert navigators from the general population, during COVID-19. We lead and participated in international and interdisciplinary workshops and conferences to share agenda-setting research, and to bring together researchers across cognate disciplines with potential stakeholders. Our results have been made available almost exclusively through open science.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

We designed 2D and 3D mobile maps, deployed on open Android-technology, using GPS to track movement and map interaction behavior in VR and outdoors, including ET, EEG, and GSR sensing to study spatial learning, cognitive load, visual attention, and affect in-situ during navigation.
We implemented a feedback loop to adapt a VR display based on real-time EEG response data. A gamified version was shown at a public science fair (https://www.geo.uzh.ch/en/units/giva/services/virtual-reality-HMD.html and https://www.geo.uzh.ch/en/units/giva/news0/Scientifica.html). This is available in our open science repository (https://gitlab.uzh.ch/giva/geovisense) and on the group’s website https://www.geo.uzh.ch/en/units/giva/services.html.

We find:

1. Behavioural responses to mobile map designs with different landmark density may be similar while brain activity can show different response patterns. While cognitive load might steadily increase with mobile map displays that show increasingly dense landmarks along a route, behavioral responses may already plateau with a medium dense landmark display design. Aan increased parieto-occipital P3 amplitude indicates higher cognitive load in a 7-landmark condition, compared to showing only 3 or 5 landmarks. Navigators learn more in the 5- and 7-landmark conditions, compared to the 3-landmark condition. Showing a medium density of (5) landmarks, compared to 3 or 7 landmarks on a mobile map improves spatial learning without overtaxing cognitive load during navigation in different virutal urban environments.

Possibly a cognitive load spillover effect during map-assisted wayfinding occurs whereby cognitive load during map viewing might have affected cognitive load during goal-directed locomotion or vice versa. We suggest that users’ cognitive load and spatial learning should be considered together when designing the display of future navigation aids and that navigators’ eye blinks can serve as useful event makers to parse continuous human brain dynamics reflecting cognitive load in naturalistic settings.

2. Primacy and recency features of serial memory that are a hallmark of typical memory functions can also be observed in uncontrolled, real-world navigation behaviour. This suggests that general memory mechanisms are involved in spatial learning, and that landmark sequence knowledge is a feature of spatial knowledge which is affected by navigation aids.

3. While navigating with realistic-looking landmarks on a mobile map outdoors, low-spatial-ability wayfinders focused more on the landmarks in the environment and show improved directional knowledge between landmarks. Fixation event-related (EEG) potentials reveal that the amplitude of the parietal P200 component was enhanced when participants fixated landmarks in the real world that were visualized on the mobile map in a realistic style, and that frontal P200 latencies were prolonged for landmarks depicted in either a realistic or abstract style compared with features of the environment that were not presented on the map, but only for the male participants. The cognitive matching process between landmarks seen in the environment and those previously seen on a map is facilitated by the more realistic map display, while low-level perceptual processing of landmarks and recall of associated information are unaffected by map visualization style.

We find that the visualization style of landmarks on mobile map aids partially modulates wayfinders’ gaze behavior, which, in turn, can predict some aspects of spatial learning. Our studies highlight that self-reported spatial abilities are a consistent predictor of several spatial learning tasks and that individuals with lower spatial abilities are more likely to benefit from landmark visualizations with greater fidelity.
We suggest that future map-based navigation aids should focus on a visually salient depiction of landmarks to direct the attention of wayfinders with varying spatial abilities to these features for sustained spatial learning.

Taken together mobile map design decisions, including, for example, landmark visualization style or any other relevant information visualized on the navigation aid, should be adapted to wayfinders’ individual spatial abilities, their background and training, preferences, needs, and the changing environmental context (i.e. environmental familiarity, etc.).

A participant wearing a mobile EEG and mobile eye tracker during an outdoor navigation study.

GeoViSense: Indoor VR and outdoor mobile human sensing during assisted navigation and wayfinding

The world a test participant is experiencing wearing the HMD VR over an EEG cap is projected onto a