Prof. Dr.-Ing. Bert Arnrich

Human Emotion and Activity Classification Using Brain
Activity Sensors


Ever wondered if the development of healthcare apps and smart sensors like smartwatches or similar wearable devices can have an actual impact on the lives of people that need help and not just on your running performance? Like people that suffer from epilepsy, people with depressive mental disorders, or the grandmother that you know that slowly seems to lose her ability to tell how she feels? Well,... we all here did, and that is why we decided to work with interesting data like time-series data from wearable devices in the chair of ‘Digital Health and Connected Healthcare’!

Time-series data constitutes significantly to assess health. Multiple choices exist when classifying time-series data, depending on the underlying model for health, the aspects of well-being analyzed, and how classification is made. Due to different recommendations in the literature, difficulties in recording sufficient amounts of data, and other reasons, no overall evaluation can tell which Time Series Classification (TSC) models to use for which TSC problem. Nowadays, recording a sufficient amount of data is ensured by the availability of cool wearable devices, like the Muse S headband. The wearables we have at our chair can be providing various physiological signals, like the Heart-Rate Variability (HRV), Photoplethysmogram (PPG), Electroencephalography (EEG), and acceleration data, amongst others. While medical professionals have access to high-quality measurement devices, the use of affordable, wearable devices will drastically increase the quality of life of the grandmother we just mentioned. These devices enable data collection in the wild, freeing participants from the burden of repeated hospital stays for data acquisition.

Nevertheless, the relatively new concept of introducing multimodality, particularly the introduction of headband sensors providing both EEG and IMU sensor data, is aiding the previously existing TSC models, especially in classifying the physiological signals driven by our mood and emotions in daily life. The underlying experience of feeling, emotion, or mood is called affects in terms of psychology. Affective state classification accuracy can be improved when other contexts, i.e., a person's activity is known.

To increase the signal-to-noise ratio (SNR) of data collected in the wild, specialized algorithms for time-series analysis have been developed and can be applied as data preprocessing. These range from statistical features over filtering of the recorded data in the spectral, spatial, or temporal domain to non-linear signal analysis utilizing chaos theory.

Hence, within the scope of this master project, we would investigate three-fold TSC classification problems where emotion classification and daily activity classification build the foundation for joint emotion and activity classification. Therefore, this research-driven Master project will assist in finding or developing approaches on how to deal with specific sub-problems that arise when dealing with difficult-to-address topics---like the prediction of epileptic seizures, depressive disorders, or any mental health problems driven by emotions---using wearable devices and thereby empowering patients.

The topic at hand

In this Master project, we want to work in the field of activity and emotion classification of different types of physiological data. There are many fascinating aspects to working with wearable devices. The most fascinating aspect of this project is the evaluation of the commercial-grade wearable EEG headbands over clinical-grade EEG devices. These headbands come with an onboard IMU sensor, which can be used to further classify the daily activities of a person. We will consider the inclusion of other sensors, such that we can compare between what our heart says and what our brain says. Therefore, for better classification, we would investigate the impact of data preprocessing pipelines on activity and emotion classification and whether the same pipeline used individually can be used for both. Additionally, we will analyze how well the developed subject-dependent models generalize.

Graphical abstract of the tasks and impact of this master project

What the students will do

Within the scope of this project, students will conduct their experiments to classify emotions and activity. They will be responsible for data recording, potentially utilizing already developed applications used within the chair. As a next step, they will build pre-processing pipeline designs based on literature reviews and evaluate multiple Machine learning and Deep learning models for TSC. The students will learn to use an already existing toolbox in the chair for synchronizing the multimodal data sampled at different recording frequencies. The emotions that will be subject to classification depend on the experimental setup of the students and will be subject to proper planning to compare findings to the literature but will at least make up a binary classification task. Working in a real-world scenario with an impact later down the line will pose a sufficiently tricky challenge. Eventually, the students could extend their setups to evaluate findings with medical professionals, building the foundation for exciting questions for their later master thesis.

Requirements and Expectations

This is an interdisciplinary project spanning areas of psychology, signal processing, computer science, and medicine. Hence, a broad and diverse field of experience and interests is required.

Regarding computer science, this project will cover:

  • Processing of medical data
  • Time-series data
  • Machine learning and deep learning

Regarding medicine, this project will cover:

  • Human psychology e.g., affects
  • Human physiology e.g., brain waves

However, students are not expected to have knowledge in all of these fields. We expect however the interest and dedication to fill in knowledge gaps apart from the time necessary for the conduct of the work described in this Master project description.


Sidratul Moontaha

Room: G-2.1.21

Phone: +49 331 5509-3481

E-Mail: sidratul.moontaha(at)hpi.de


Christoph Anders

Room: G-2.1.21

Phone: +49 331 5509-4501

E-Mail: christoph.anders@hpi.de

Bert Arnrich

Room: G-2.1.14

Phone: +49 331 5509-4850

E-Mail: bert.arnrich(at)hpi.de