Hasso-Plattner-InstitutSDG am HPI
Hasso-Plattner-InstitutDSG am HPI
Login
 

The Classification of Mental States

My profile with the Chair for Digital Health - Connected Healthcare


PhD-student since 01.04.2021; Email: christoph.anders(at)hpi.de 


Supervised by Prof. Dr. Bert Arnrich


DE (Muttersprache) | EN (fluent) | ES (poquito)


Room: G-2.1.21 | Phone: +49-(0)1520 299 4501


 

Background

Objective measurement methods to quantify the health status of individuals date back a long time. As one such example, electroencephalography (EEG) has been used for over a century to analyze electric currents in and around the brain and is used widely in clinical healthcare systems. Unfortunately, gold-standard clinical EEG systems usually require a lot of time to set up, are limited in mobility, and are not designed to look inconspicuous. However, as EEG directly measures electrical correlates of the brain with a high temporal resolution, it is an important measurement method and of high interest for mental state classification. Therefore, wearable EEG devices are being developed for various use cases.

One area of application of wearable EEG devices that holds great promise is the classification of mental workload and stress. “Mental workload may be viewed as the difference between the capacities of the information processing system that are required for task performance to satisfy performance expectations and the capacity available at any given time.” (from: 'Workload: An examination of the concept', by Gopher and Donchin). Additionally, it has been shown that high levels of mental workload over long periods of time can lead to an increased risk of coronary heart disease and hypertension, amongst others. Mental workload and stress are closely related topics, and a high mental workload is regarded as a preliminary factor for stress. When mental workload increases, stress levels rise and mental fatigue can be developed, which might result in deteriorating mental health. While not every mental workload directly results in work-related stress, the long-term consequences of high levels of mental workload and stress might be very costly, which is why accurate time-series classification (TSC) is so important.

We want to move this technology directly where it is needed: outside of residing only in clinical or laboratory environments, bringing it right into your hands, ready to use and to assist you in living a longer and healthier life! For this purpose, we conducted a systematic literature review on mental state classification using wearable EEG. Alongside many findings on data pre-processing, the applicability of artificial intelligence and more, one of our findings was that multi-modal TSC applications outperform their best unimodal counterparts. Consequently, in our ongoing studies, we are combining the utilization of wearable EEG with other wearable devices capable of measuring additional physiological signals, such as the heart rate. Alongside the desire to bring the technology closer to individuals, as of Jan. 2024, we are working towards including low-cost wearables in our studies (e.g. the EmotiBit sensor suite).

Ongoing Work (as of Jan. 2024)

Our current and future work is focusing on differentiating levels of mental workload and mental workload task types in controlled, semi-controlled, and uncontrolled environments. These studies aim at assessing the influence of noise on recordings of psychophysiological correlates of mental workload, investigate the reliability of wearable sensor systems on mental workload classification, and on creating publicly available data sets to enable national and international collaboration on these important topics (As of Jan. 2024, we submitted an extensive data descriptor currently under review, and expect to have the respective publication and data publicly available in mid 2024). Furthermore, we want to work on automated techniques for the identification and removal of artefacts, are planning on investigating using Bio-/Neurofeedback as an intervention and as a means for noise-reduction as well as enhancing learning outcomes in mobile learning scenarios.

For our current ongoing studies revolving around the field of 'Measuring and Quantifying Mental Workload and Stress in Everyday Situations', 'Analysis of Synchronized Physiological Signals to Assess Mental Workload, Stress, Engagement, and Intervention-Effects on Small Groups', and 'Measuring and Quantifying Mental Workload and Stress in Everyday Situations', we are constantly searching for interested study participants. Please reach out to me if you are interested in participating in one of our studies and learning more about the study procedures, how you could assist, remuneration for your participation, and to find additional information. Furthermore, if you are a master's student interested in writing your master's thesis on, or with aspects of, the analysis of multi-modal time-series data in healthcare, please reach out to me and let us discuss possible topics. You can find information on the projects and thesis I am supervising / have supervised on our chairs websites. I am interested in supervising your project if it evolves around classification, prediction, or forecasting tasks on time-series data obtained from wearable on-body sensor systems, especially investigating the effects of data pre-processing, with a solid contribution of EEG.

Line-Plots: Example EEG signals (channels AF7, AF8, TP9, and TP10) of myself in resting-state with closed eyes, recorded using a Muse S device. No pre-processing was performed. Some aspects of the difficult nature of EEG data can be seen in this plot. Topographic Map: Schematic representation of the recording-sites.

The many applications of multi-modal time-series

To establish a solid foundation for my future work on current pressing challenges, timely developments, and future trends in the field of mental state classification using wearable EEG, I conducted a systematic literature review titled Wearable Electroencephalography for Mental State Classification: A Systematic Literature Review together with Prof. Dr. Bert Arnrich, which is published as open-access. The search initially led to 602 publications, which were reduced to 76 items after duplicate removal, quality control and topic matching, subsequently analyzed in the categories data, recording device, questionnaire, stimulus, data pre-processing steps, task-handling, and training set-up used. Comparing the results and further notes, we found that most studies use a rather small frequency band (Delta to Gamma), use spectrum- or amplitude-based features, and utilize support vector machines, artificial neural networks, and threshold-based TSC models for classification of the time-series data. Additionally, inter-subject training paradigms are utilized seldomly with wearable EEG, whereas studies using clinical-grade EEG oftentimes tackle inter-subject variability directly. Across publications, a window length of one second was mainly chosen for classification and spectral features were utilized the most. The achieved performance per time-series classification model was analyzed, finding linear discriminant analysis, decision trees, and k-nearest neighbors models outperform support-vector machines by a factor of up to 1.5. While TSC performance was found to vary strongly across architectures, training paradigms, problems, and recording devices uses, it was found that reproducibility across studies is one of the biggest challenges.

Comparison of wearable EEG devices. Left: EEG electrodes placed within an EEG cap, as utilized also in clinical-grade EEG setups, and available as wearable devices e.g. from OpenBCI. Right: somewhat inconspicuous wearable EEG device, offering less channels at the expense of being less versatile than clinical-grade EEG. The development of applications for wearable EEG devices is an open field with many challenges and opportunities, such as in the design of hidden wearable EEG devices.

Publications

2023

  • Optimization of data pre-processing methods for time-series classification of electroencephalography data. Anders, Christoph; Curio, Gabriel; Arnrich, Bert; Waterstraat, Gunnar in Taylor and Francis, Network: Computation in Neural Systems (2023). 34(4) 374–391.
     

2022

  • Towards Multi-Modal Recordings in Daily Life: A Baseline Assessment of an Experimental Framework Anders, Christoph; Moontaha, Sidratul; Arnrich, Bert in IS (2022). (Vol. H) 27–30. Information Society.
     
  • Wearable electroencephalography and multi-modal mental state classification: A systematic literature review. Anders, Christoph; Arnrich, Bert in Computers in Biology and Medicine (2022). 150 106088.
     

Ongoing (as of Jan. 2024)

Under Review:

  • UNIVERSE: UNobtrusIVE measuRement of mental workload and Stress in uncontrolled Environments

In Preparation:

  • COSMOS: Classification Of Stress and workload using multiMOdal wearable Sensors
  • Optimizing Mobile Learning Experiences: The Wearable Sensors Opportunity

Data collections to be published as Open-Access Datasets:

  • Measuring and Quantifying Mental Workload and Stress in Everyday Situations (Study data for UNIVERSE and COSMOS)
  • Analysis of Synchronized Physiological Signals to Assess Mental Workload, Stress, Engagement, and Intervention-Effects on Small Groups
  • Measuring and Quantifying Mental Workload and Stress in Everyday Situations (Extension focusing on typical office activities)