3D heat maps of multimodal travel planning:
Correlating prepositional and adverbial phrases with locating and routing gestures

Bela Brenger/Daniel Schüller (Accepted for ISGS 2016, July 18th-22th. Gesture — Creativity — Multimodality. The International Society for Gesture Studies)
How do interlocutors utilize their gesture spaces for spatial-geographical orientation during collaborative travel planning? Indicating potential destinations and routes typically involves the use of highly context-dependent indexical expressions such as certain closed-class items (Talmy 2000) or shifters (Jakobson 1971). Our assumption is that, in spoken German discourse, the use of place names and deictic expressions – such as prepositions (e.g., ‘nach’ (to), ‘von’ (from), ‘bei’ (at)) and locative or directional adverbials (e.g., ‘da’ (there), ‘hier (here), ‘rüber’ (over)) – correlates with distinct kinds of gestural shifts along the main axes (vertical, transversal, sagittal), which consist of locating and routing gestures.
Specifically, this study’s target structures are prepositional phrases such as PREP + ADV (e.g., ‘nach hier’, ‘nach da’ (to here/there)) and PREP + N (‘von Norden’ (from north), ‘nach Paris’ (to Paris)) and adverbial phrases comprising ADVlocative + ADVdirectional (e.g., ‘da rüber’ (over there), ‘hier hin’ (to here)).
Drawing on the Multimodal Speech & Kinetic Action Corpus (MuSKA), our approach combines qualitative and quantitative methods to analyze three time-aligned data streams: audio, video and motion-capture data.
Integrating the different data streams and annotated transcripts into a multimodal database allowed us to correlate the gestures’ spatial characteristics with co-occurring linguistic structures. In this mapping process, special attention was paid to the gestures’ spatial attributes, such as the primarily activated axis, movement extension, and location in the emergent and adaptive 3D gesture spaces. We suggest that the observed patterns of correlated verbal shifters and gestural shifts may possibly be considered instances of multimodal constructions (Goldberg 2006).