Director:

Prof. Irene Mittelberg, Ph.D.
Building 6070 | Room 405
Theaterplatz 14
52062 Aachen

Phone: +49 241 8025486

Lab-Manager:

Bela Brenger
Building 6070 | Room 404
Theaterplatz 14
52062 Aachen

Phone: +49 241 80 25485
Fax: +49 241 80 22493

Natural Media Lab

In 2010, the Natural Media group has established one of the few motion capture labs specifically configured for gesture and multimodal communication research. It houses a state-of-the-art optical Vicon Motion Capture (MoCap) system aligned with high-speed and video cameras. MoCap overcomes the limits of traditional video recording by visualizing movement trajectories and measuring positions of articulators in 3D space. Group members have developed hand model prototypes suitable for gesture research, and are developing automated annotation tools (ELAN Plugin for Automated Annotation, EPAA) for specific hand configurations, motion patterns, velocities, and distances between gesturing hands or between the gesturer’s torso and a given freely moving articulator. Development of the first 3D models of gesture space to be shared with the gesture and sign language research communities is also underway.

The Natural Media Lab houses a Vicon MX series marker-based infrared motion capture system, consisting of

  • 10 Vicon MX T10 (1 Mp, 100 Hz)
  • 4 Vicon MX 13+ (1.3 Mp, 100 Hz)
  • 2 Basler high-speed cameras
  • DV cameras
  • HD cameras
  • Sound system

 

Recent Studies

Mapping and visualizing 3D gesture space

Based on data stemming from semi-structured interviews conducted in the Natural Media Lab, a prototype of a new method to map and visualize speakers' three-dimensional gesture spaces was developed. Taking a pragmatic approach, the locations in which the gestures were produced were correlated with their respective functions (referential, modal, interactive, etc.)

The method developed in this thesis subdivides the 3D space available for gestural communication into minimal segments, and measures the gesture density for each segment over the course of a conversation. The resulting data are visualized in two or three dimensions (see illustrations), which allows us to study the spatial and temporal structure of gesture spaces with unprecedented precision. The results offer new insights into natural gesture production in terms of both the influence of communicative function on gesture location as well as individual communication styles.

  • Priesters, Matthias A. (2012): Functional patterns in gesture space. Correlating location and function of gestures using 3D motion capture technology. Unpublished M.A. thesis, RWTH Aachen.
  • Priesters, Matthias A. & Irene Mittelberg (2013): Individual differences in speakers' gesture spaces: Multi-angle views from a motion-capture study. Proceedings of the Tilburg Gesture Research Meeting (TiGeR), June 19-21, 2013. Link

Description and appreciation of abstract art

Three abstract paintings by Paul Klee were presented to participants, who were subsequently asked to describe the paintings from memory and to talk about the meaning and message of the images. The illustration shows Klee’s “Tightrope Walker” and MoCap traces of a participant drawing the picture’s structure into the gesture space in front of her during her description of the image.

  • Mittelberg, Irene (2012): Balancing acts. Felt qualitites of meaning in descriptive gestures and artworks by Paul Klee. CSDL 9, UBC, Vancouver, May 17-20, 2012.
  • Mittelberg, Irene (2013): Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. Language and the Creative Mind, eds. B. Dancygier, M. Bokrent, J. Hinnell. Stanford: Center for the Study of Language and Information.

Gesture and Architecture

In winter term 2011/12, the Natural Media Lab hosted an Architecture seminar by Lehrstuhl für Bildnerische Gestaltung (Prof. Thomas Schmitz, Hannah Groninger) about visualizing motion. The students pursued their own artistic projects, using the MoCap system to e.g. create three-dimensional sketches through gestures. The illustration shows a sketch of Aachen Cathedral, created by Paula Basilio using a pencil with a MoCap marker attached.

  • Groninger, Hannah, Mittelberg, Irene & Thomas Schmitz (2012): Gestural drawings and models. Motion capture technologies in architectural design processes. Gestures@Work Workshop, Aachen, May 4, 2012.
  • Mittelberg, I. (2012). Ars memorativa, Architektur und Grammatik. Denkfiguren und Raumstrukturen in Merkbildern und spontanen Gesten. In: Schmitz, T., Groninger, H. (Hrsg.): Werkzeug / Denkzeug. Manuelle Intelligenz und Transmedialität kreativer Prozesse. Bielefeld: Transcript Verlag.
  • Mittelberg, I., Schmitz, T.H., Groninger, H. (to appear). Operative Manufakte. Gesten als real räumliche Skizzen in frühen Stadien des Entwurfsprozesses. In Hinterwaldner, I., Ammon, S. (Eds.) Bildlichkeit im Zeitalter der Modellierung. Operative Artefakte in Entwurfsprozessen der Architektur und des Ingenieurwesens, Basel: Eikones Reihe, Fink Verlag.

Influence of noise on gesture production

This study followed up on ethnographic work done by Simon Harrison in factory surroundings that indicated an influence of environmental conditions (e.g. obstacles, noise) on gesture production. In the lab, a production line situation was simulated on a computer screen, which participants had to monitor. Upon observing discrepancies from the norm, these had to be communicated by using predefined gestures. The experimental conditions included the presence vs. absence of factory noise. Currently the MoCap data are being analyzed to determine the influence of noise on gesture size.

  • Harrison, Grandhi & Mittelberg (2011): Effects of noise on gesture production. Insights from a Motion Capture study. Poster presented at Sensory Motor Concepts in Language and Cognition, Düsseldorf University. December 2nd, 2011

Acknowledgements

The Natural Media Team is grateful to HumTec and to Prof. Jarke (Informatik 5) for the generous support of the lab, and to Prof. Binkofski (University Hospital Aachen) for lending us 4 infra-red cameras.

www.vicon.com
www.prophysics.ch