Prof. Dr. Irene Mittelberg
Winter term 2009/10
Research colloquium

Cognitive semiotics and natural media

This research colloquium is anchored in the HumTec project “Natural Media & Engineering” and provides a forum for discussing theoretical concepts and their application to coverbal gestures and other forms of bodily semiotics such as eye gaze and facial expressions. It takes a cognitive-semiotic approach to multimodal communication by combining insights and methods from contemporary cognitivist theories (e.g., Clark, Johnson, Lakoff, Talmy) with the older but still relevant work of Wilhlem Wundt, Charles Sanders Peirce, and Roman Jakobson.

These traditional semiotic frameworks continue to provide rich perspectives for multimodal research and lend themselves to being applied to gesture and other natural media, for they are broader in scope than linguistic theories and have been used to analyze a wide variety of dynamic, discursive phenomena such as theatre, cinema, myths, rituals, and music (in all of which gestures are used to convey ideas, emotions, and symbolic meaning). In addition, connections will be drawn between Jakobson’s view on aphasia and recent studies on gestural communication under clinically impaired conditions (e.g., Duncan, McNeill).

Different modes of gestural sign constitution and interpretation will be discussed. Basic semiotic modes (such as similarity/contiguity, iconicity/indexicality, and metaphor/metonymy) will provide the basis for fine-grained categorical distinctions of multimodal sign-processes, and build the conceptual thread throughout the discussions. We will also examine how these modes may feed into embodied patterns such as image schemas, geometric forms and conceptual metaphor and metonymy which have been argued to motivate some of the systematicity observed in fleeting gestural figures (e.g., Bouvet, Cienki, Fricke, Mittelberg, Müller, Wilcox, Taub).

The overall goal of this research colloquium is twofold:
a) to gain a better understanding of the cognitive-semiotic foundations that may underlie less-monitored aspects of cognition during communication and what we may want to call, following Johnson (2005), “the felt qualities of meaning”;
b) to look at how such basic patterns of communication via natural media under healthy and naturalistic conditions compare with clinically impaired and technologically modified conditions.

Participants are invited to explore specific aspects in detail, present alternative approaches, do sample analyses of multimodal data of their choosing, and present work in progress.


  • Gibbs, R. (2005). Embodiment and Cognitive Science. Cambridge University Press.
  • McNeill, David (2005). Gesture and Thought. Chicago University Press.
  • Cienki, Alan & Cornelia Müller (eds.) (2008). Metaphor and Gesture. Amsterdam/Philadelphia: John Benjamins.
  • Talmy, L. (2000). Toward a Cognitive Semantics. Vol I, II. Cambridge, MA: MIT Press.
  • Taub, S. (2001). Language from the Body: Iconicity and Metaphor in American Sign Language. Cambridge University Press.

Further information: CAMPUS.