Die Zwitschermaschine, Paul Klee, 1922
The project “Natural Media & Engineering: Emotion-Cognition Interactions and their Modification” takes a truly interdisciplinary approach to multimodal manifestations of human face-to-face and human-machine communication. Of central interest is the integration of speech, gestures and facial expressions. Semioticians, linguists, cognitive psychologists, clinical neuroscientists and computer scientists jointly investigate how cognitive and emotional processes are multimodally expressed and interpreted.
The aim is to establish the semiotic foundations of natural media by investigating how they collaborate under naturalistic, clinically impaired, and technologically modified conditions. Specifically, the goal is to find out in what ways the mapping of semiotic principles, cognitive processes, and neural substrates can provide new insights into the complex dynamics of human communication.
To this end, several methods and technologies will be employed: multimodal discourse analysis, experimental design, clinical tests, and neuro-imaging. A key element of this research is a comparison of the verbal and gestural behavior of healthy speakers as well as neurological and psychiatric patients suffering from, for instance, autism, aphasia, or schizophrenia. Another focus of attention is the role natural media play in the design and use of cooperative computer games in serious gaming contexts. The insights resulting from this work is meant to enhance the development of both clinical therapies and future interactive technologies.