Description:
Human communication is inherently multimodal, involving facial expressions, gestures, postures, speech, and more, all dynamically coordinated. While traditional cognitive science relies on expert annotators for labeling communicative units, machine recognition and classification of dynamic body movements remain relatively unexplored. This project, DYCLASSIFIED 1.0, will investigate the application of existing multimodality-oriented pretrained models (such as BERT, GPT-3, CLIP, and GATO) in recognizing and classifying structure and temporality of dynamic communicative body movements. Collaborating with cognitive scientists and computer scientists from the Max Planck Institute of Psycholinguistics, the Donders Institute, and the Hasso-Plattner Institute, the research will adopt a data-driven perspective to operationalize unsupervised body movement classification. By fine-tuning existing models with sample labels, the project aims to extract implicit patterns in specific gestures. The outcomes may revolutionize manual gesture classification, opening new avenues in areas like surveillance, autonomous vehicles, medical data, and commercial applications.
Literature:
Contacts: