Human Emotion and Activity Classification Using Brain Activity Sensors

Master Project (Winter Term 2021/2022)

Supervisors: Christoph Anders, Sidratul Moontaha

Graphical abstract of tasks performed during this master project

This was an interdisciplinary project spanning areas of psychology, signal processing, computer science, and medicine. Hence, a broad and diverse field of experience and interests was required.

Regarding computer science, this project covered:

Processing of medical data
Time-series data
Machine learning and deep learning

Regarding medicine, this project covered:

Human psychology e.g., affects
Human physiology e.g., brain waves

However, students were not expected to have knowledge in all of these fields. We expected however the interest and dedication to fill in knowledge gaps apart from the time necessary for the conduct of the work described in this Master project description.

Framework

Within the scope of this project, students conducted their experiments to classify emotions and activity. They were responsible for data recording and utilized already developed applications within the chair. As a next step, they built pre-processing pipeline designs based on literature reviews and evaluated multiple Machine learning and Deep learning models for TSC. The students learned to use an already existing toolbox in the chair for synchronizing the multimodal data sampled at different recording frequencies. The emotions that were subject to classification depended on the experimental setup of the students and were subject to proper planning to compare findings to the literature. Working in a real-world scenario with an impact later down the line posed a sufficiently tricky challenge. Eventually, future students could extend this setup to evaluate findings with medical professionals, building the foundation for exciting questions for their later master thesis.

Outcome

The final project was called 'Distinction Between Work and Relaxation: A Use Case of Multimodal Wearable Sensors'. The following abstract summarizes the students work:

Mental fatigue is a problem everybody is encountering at work, at school, while driving, and during other mentally demanding tasks. Several changes in physiological signals, e.g., rising heart rate or enhanced brain activity, can indicate whether a person is mentally exhausted or relaxed. In this project, Electroencephalogram (EEG), Photoplethysmogram (PPG), skin temperature, and Galvanic skin Response (GSR) are measured by Empatica and Muse devices during studies that induce a relaxed state and mental workload/stress. The collected data is synchronized and filtered using techniques like Savitzky-Golay Filter, Butterworth bandpass filter, Adaptive Noise Cancellation filter, and finally used to compute features, using, e.g., Common Spatial Pattern. A multi-modal platform for visualizing and managing participants’ study data is built. The prediction of mental fatigue is implemented using several machine learning models. Implemented models are Support Vector Machine, Random Forest, and Deep Convolutional Neural Networks directly provided on the platform and with configurable hyper- parameters, train-test-splits, and leave-one-out cross-validation, with the best performing model being a Random Forest at an average balanced accuracy of 82.21% and the second best model being a Support Vector Machine at an average balanced accuracy of 81.23%.

Contact

Sidratul Moontaha

Room: G-2.1.21

Phone: +49 331 5509-3481

E-Mail: sidratul.moontaha(at)hpi.de

Christoph Anders

Room: G-2.1.21

Phone: +49 331 5509-4501

E-Mail: christoph.anders@hpi.de

Bert Arnrich

Room: G-2.1.14

Phone: +49 331 5509-4850

E-Mail: bert.arnrich(at)hpi.de

Resulting Publications

Towards Multi-Modal Recordings in Daily Life: A Baseline Assessment of an Experimental Framework Anders, Christoph; Moontaha, Sidratul; Arnrich, Bert in IS (2022). (Vol. H) 27–30. Information Society.

[ Abstract ] [ BibTeX ] [ URL ]

Background: Wearable devices can record physiological signals from humans to enable an objective assessment of their Mental State. In the future, such devices will enable researchers to work on paradigms outside, rather than only inside, of controlled laboratory environments. This transition requires a paradigm shift on how experiments are conducted, and introduces new challenges. Method: Here, an experimental framework for multi-modal baseline assessments is presented. The developed test battery covers stimuli and questionnaire presenters, and multi-modal data can be recorded in parallel, such as Photoplethysmography, Electroencephalography, Acceleration, and Electrodermal Activity data. The multi-modal data is extracted using a single platform, and synchronized using a shake detection tool. A baseline was recorded from eight participants in a controlled environment. Using Leave-One-Out Cross-Validation, the resampling of data, the ideal window size, and the applicability of Deep Learning for Mental Workload Classification were evaluated. In addition, participants were polled on the acceptance of using the wearable devices. Results: The binary classification performance declined by an average of 7.81% when using eye-blink removal, underlining the importance of data synchronization, correct artefact identification, evaluating and developing artefact removal techniques, and investigating on the robustness of the multi-modal setup. Experiments showed that the optimal window size for the acquired data is 30 seconds for Mental Workload classification, with which a Random Forest classifier and an optimized Deep Convolutional Neural Network achieved the best-balanced classification accuracy of 70.27% and 74.16%, respectively. Conclusions: This baseline assessment gives valuable insights on how to prototype stimulus presentation with different wearable devices and suggests future work packages, paving the way for researchers to investigate new paradigm outside of controlled environments.

@proceedings{is-2022,
  abstract = {Background: Wearable devices can record physiological signals from humans to enable an objective assessment of their Mental State. In the future, such devices will enable researchers to work on paradigms outside, rather than only inside, of controlled laboratory environments. This transition requires a paradigm shift on how experiments are conducted, and introduces new challenges. Method: Here, an experimental framework for multi-modal baseline assessments is presented. The developed test battery covers stimuli and questionnaire presenters, and multi-modal data can be recorded in parallel, such as Photoplethysmography, Electroencephalography, Acceleration, and Electrodermal Activity data. The multi-modal data is extracted using a single platform, and synchronized using a shake detection tool. A baseline was recorded from eight participants in a controlled environment. Using Leave-One-Out Cross-Validation, the resampling of data, the ideal window size, and the applicability of Deep Learning for Mental Workload Classification were evaluated. In addition, participants were polled on the acceptance of using the wearable devices. Results: The binary classification performance declined by an average of 7.81% when using eye-blink removal, underlining the importance of data synchronization, correct artefact identification, evaluating and developing artefact removal techniques, and investigating on the robustness of the multi-modal setup. Experiments showed that the optimal window size for the acquired data is 30 seconds for Mental Workload classification, with which a Random Forest classifier and an optimized Deep Convolutional Neural Network achieved the best-balanced classification accuracy of 70.27% and 74.16%, respectively. Conclusions: This baseline assessment gives valuable insights on how to prototype stimulus presentation with different wearable devices and suggests future work packages, paving the way for researchers to investigate new paradigm outside of controlled environments.},
  author = {Anders, Christoph and Moontaha, Sidratul and Arnrich, Bert},
  booktitle = {Proc.\ 25th International Multiconference INFORMATION SOCIETY Pervasive Health and Smart Sensing (IS 2022)},
  keywords = {christophanders bertarnrich sidratulmoontaha},
  pages = {27-30},
  publisher = {Information Society},
  series = {IS},
  title = {Towards Multi-Modal Recordings in Daily Life: A Baseline Assessment of an Experimental Framework},
  volume = {H},
  year = 2022
}