Wearable EEG for the Classification of Mental States

Christoph Anders, PhD-student since 01.04.2021

+49-1520 299 4501 | christoph.anders(at)hpi.de

Campus III Building G2 | Room G-2.1.21

Chair for Digital Health - Connected Healthcare

DE (mother tongue) | EN (fluent) | ES (un pocito)

Objective measurement methods to quantify the health status of individuals date back a long time. As one such example, electroencephalography (EEG) has been used for over a century to analyze electric currents in and around the brain and is used widely in clinical healthcare systems. Unfortunately, gold-standard clinical EEG systems usually require a lot of time to set up, are limited in mobility, and are not designed to look inconspicuous. However, as EEG directly measures electrical correlates of the brain with a high temporal resolution, it is an important measurement method and of high interest for mental state classification. Therefore, wearable EEG devices are being developed for various use cases.

One area of application of wearable EEG devices that holds great promise is the classification of mental workload and stress. “Mental workload may be viewed as the difference between the capacities of the information processing system that are required for task performance to satisfy performance expectations and the capacity available at any given time.” (from: [0]). Additionally, it has been shown that high levels of mental workload over long periods of time can lead to an increased risk of coronary heart disease and hypertension, amongst others. Mental workload and stress are closely related topics, and a high mental workload is regarded as a preliminary factor for stress. When mental workload increases, stress levels rise and mental fatigue can be developed, which might result in deteriorating mental health. While not every mental workload directly results in work-related stress, the long-term consequences of high levels of mental workload and stress might be very costly, which is why accurate time-series classification (TSC) is so important.

We want to move this technology directly where it is needed: outside of residing only in clinical or laboratory environments, bringing it right into the hands of everyone, ready to use and to assist in living a longer and healthier life! For this purpose, we conducted a systematic literature review on mental state classification using wearable EEG. Alongside many findings on data pre-processing, the applicability of artificial intelligence and more, one of our findings was that multi-modal TSC applications outperform their best unimodal counterparts. Consequently, in our ongoing studies, we are combining the utilization of wearable EEG with other wearable devices capable of measuring additional physiological signals, such as the heart rate.

Our current and future work is focusing on differentiating levels of mental workload and mental workload task types in controlled, semi-controlled, and uncontrolled environments. These studies aim at assessing the influence of noise on recordings of psychophysiological correlates of mental workload, investigating the reliability of wearable sensor systems on mental workload classification, and on creating publicly available data sets to enable national and international collaboration on these important topics. Furthermore, we want to work on automated techniques for the identification and removal of artefacts, are planning on investigating using Bio-/Neurofeedback as an intervention and as a means for noise-reduction, and are working towards conducting basic research on the potential possibility of self-modulation of response-strength to a given stimulus.

For our current studies revolving around the field of 'Measuring and Quantifying Mental Workload and Stress in Everyday Situations', we are constantly searching for interested study participants. Interested individuals wanting to participate in one of our studies and learn more about the study procedures, how to assist, remuneration for participation, and to find additional information are highly encouraged to reach out to me. Furthermore, master's students interested in writing a master's thesis on, or with aspects of, the analysis of multi-modal time-series data in healthcare, please reach out to me and let us discuss possible topics. I am interested in supervising various projects if they evolve around classification, prediction, or forecasting tasks on time-series data obtained from wearable on-body sensor systems, especially investigating the effects of data pre-processing, with a solid contribution of EEG.

Example EEG signals of myself in resting-state with closed eyes — Line-Plots: Example EEG signals (channels AF7, AF8, TP9, and TP10) of myself in resting-state with closed eyes, recorded using a Muse S device. No pre-processing was performed. Some aspects of the difficult nature of EEG data can be seen in this plot. Topographic Map: Schematic representation of the recording-sites.

Research Area: Mental State Classification with wearable EEG

The mental state of a person can be directly linked to human errors or subjective decisions [1] - [3]. Most straight-forward, it can be assessed with questionnaires. However, this type of exams may not suffice in case of acute dangerous situations. In general, the mental states such as emotions can be classified from physiological signals [4], which can be analyzed with sensors like camera systems, electrocardiogram (ECG), by using speech analysis, or by observing the heart-rate variability (HRV), amongst others [4] - [12]. Another measurement technique suitable for mental state classification is the EEG. Hence, one of the areas of my work focuses on investigating mental workload and stress using multiple modalities.

While clinical systems have been around for nearly a century, they might require a lot of time to set up, are limited in mobility and clinical systems are not designed to look inconspicuous. Wearable systems have been developed, which commonly offer only a limited amount of recording channels, usually do not allow to record data for periods of time longer than a day, and are more likely to experience signal distortion due to artefacts. Artefacts can distort the signal of interest significantly---hence, to optimize the signal quality, it is first and foremost advisable to minimize the influence of 'noise' due to artefacts---and can be of biological or technical origin [13]. Another area of my work focuses on developing analysis methods for the automated removal of artefacts.

Once the data is recorded, usually multiple steps are performed to increase the signal-to-noise ratio (SNR). So-called data pre-processing steps can be spatial filtering (e.g. by building a bipolar montage or interpolating bad channels), spectral filtering (e.g. to remove spectrally localized noise such as AC/DC noise), or temporal filtering (e.g. by means of interpolating recordings to filter out stimulus artefacts, if any). Afterwards, features can be extracted from the pre-processed data and feature-transformations might increase information content. Features are commonly of a) spectral nature, derived with Power Spectral Density, Fourier Transformation, or Wavelet Transformation, of b) amplitude-related nature like statistical measures, or c) entropy-related information [14] - [16]. The features are utilized for TSC model building, with TSC models commonly being threshold-functions, Machine Learning (ML) models, or Deep Learning (DL) models. While challenges like reproducability across studies due to model-, data-, and code-availability, are cumbersome for research on wearable EEG, many more challenges are faced in this field, such as system reliability, user privacy, interoperability between wearable devices, end-user training, and social inclusion, amongst others [17]. Another area of my work focuses on improving classification and forecasting algortihms for multi-modal time-series data.

We are working towards increaseing trust in wearable (EEG) systems. This includes identifying pressing challenges in the field, highlighting future perspectives, and supporting the development of wearable (EEG) applications outside of laboratory environments. In order to do so, multiple milestones have to be reached.

The many applications of multi-modal time-series

To establish a solid foundation for my future work on current pressing challenges, timely developments, and future trends in the field of mental state classification using wearable EEG, I conducted a systematic literature review titled Wearable Electroencephalography for Mental State Classification: A Systematic Literature Review together with Prof. Dr. Bert Arnrich, which is published as open-access. The search initially led to 602 publications, which were reduced to 76 items after duplicate removal, quality control and topic matching, subsequently analyzed in the categories data, recording device, questionnaire, stimulus, data pre-processing steps, task-handling, and training set-up used. Comparing the results and further notes, we found that most studies use a rather small frequency band (Delta to Gamma), use spectrum- or amplitude-based features, and utilize support vector machines, artificial neural networks, and threshold-based TSC models for classification of the time-series data. Additionally, inter-subject training paradigms are utilized seldomly with wearable EEG, whereas studies using clinical-grade EEG oftentimes tackle inter-subject variability directly [16]. Across publications, a window length of one second was mainly chosen for classification and spectral features were utilized the most. The achieved performance per time-series classification model was analyzed, finding linear discriminant analysis, decision trees, and k-nearest neighbors models outperform support-vector machines by a factor of up to 1.5. While TSC performance was found to vary strongly across architectures, training paradigms, problems, and recording devices uses, it was found that reproducibility across studies is one of the biggest challenges.

Comparison of wearable EEG devices. Left: EEG electrodes placed within an EEG cap, as utilized also in clinical-grade EEG setups, and available as wearable devices e.g. from OpenBCI. Right: somewhat inconspicuous wearable EEG device, offering less channels at the expense of being less versatile than clinical-grade EEG. The development of applications for wearable EEG devices is an open field with many challenges and opportunities, such as in the design of hidden wearable EEG devices.

This field is truly interdisciplinary in nature, and as such the cooperation with partners from different fields of expertise is essential. One such example is my master thesis Experimental Evaluation of data pre-processing methods for time series classification on brain activity data, in which I worked on experiments with varying input-formulation techniques on various challenging EEG datasets of lower- and higher-frequencies. Together with Dr. med. Gunnar Waterstraat, Prof. Dr. med. Gabriel Curio, Dr. Nico Steckhan, Sidratul Moontaha, and Prof. Dr. Bert Arnrich we investigated the research questions: 1) How much information increment is provided by different steps of a pre-processing pipeline?, 2) For within-subject training, which input formulation seems most promising, and which architecture (is) to use?, and 3) Exists an optimum for averaging consecutive trials, taking into account natural variance of (high-frequency Somatosensory Evoked Potentials) hfSEP and noise?, amongst others. Together with Dr. med. Gunnar Waterstraat, and supervised by Prof. Dr. med. Gabriel Curio and Prof. Dr. Bert Arnrich, we prepared the paper Optimization of Data Pre-Processing Methods for Time-Series Classification of Electroencephalography Data, which is currently undergoing the last steps of the publication process, and which increased my interest in the field of EEG. I would be interested in supervising similar master theses.

Workflow representation of my masters thesis. The impact of data pre-processing on three different datasets, two publicly available and one available upon reasonable request, was analyzed using different spectral and spatial filtering techniques as well as feature transformations. Out of the methods analyzed, optimal spatial filtering and subsequently applying the non-linear Hilbert Transform resulted in the highest, significant, TSC performance increments.

Teaching Activities

As of November 2022, I am supervising three master students.

In the Winter Term 2021/2022, Sidratul Moontaha and I supervised, guided by Prof. Dr. Bert Arnrich, the Digital Health - Connected Healthcare chair's Master Project called: Human Emotion and Activity Classification using Brain Activity Sensors. Amongst others, the developed test battery covers stimuli and questionnaire presenters, and multi-modal data can be recorded in parallel, such as Photoplethysmography, Electroencephalography, Acceleration, and Electrodermal Activity data. The multi-modal data can be extracted using a single platform, and synchronized using a shake detection tool. For evaluation, a baseline was recorded from eight participants in a controlled environment. Using Leave-One-Out Cross-Validation, the resampling of data, the ideal window size, and the applicability of Deep Learning for Mental Workload Classification were evaluated. In addition, participants were polled on the acceptance of using the wearable devices. This project gave our students the possibility to gather hands-on experience in experimental paradigm creation, data recording in a laboratory environment and outside-of-the-lab, and in classifying real-life data.

In June 2019, as an effort of public outreach, I gave a talk at my former high school, the Julius-Spiegelberg-Gymnasium Vechelde, about why computer sciences are cool, to make pupils curious about computer sciences.

Graphical abstract of the winter term's master project 2021/2022. Students were expected to work on emotion, activity, and joint classification of both, analyze the impact their data pre-processing approaches have, and get in touch with medical experts to validate their findings.

Publications

Wearable electroencephalography and multi-modal mental state classification: A systematic literature review; Christoph Anders, Bert Arnrich; Computers in Biology and Medicine; 2022-11; https://doi.org/10.1016/j.compbiomed.2022.106088

Towards Multi-Modal Recordings in Daily Life: A Baseline Assessment of an Experimental Framework; Christoph Anders, Sidratul Moontaha, Bert Arnrich; Pervasive Health and Smart Sensing at the Information Society; 2022-10; https://is.ijs.si/wp-content/uploads/2022/is2022zborniki/IS2022_Volume-H.pdf

And three more publications are currently in preparation or revision.

References

0. Gopher, D., & Donchin, E. (1986). Workload: An examination of the concept. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, Vol. 2. Cognitive processes and performance (pp. 1–49). John Wiley & Sons.

1. A. Bener, E. Yildirim, T. Ozkan, and T. Lajunen, “Driver ¨ sleepiness, fatigue, careless behavior and risk of motor vehicle crash and injury: Population based case and control study,” vol. 4, no. 5, pp. 496–502. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S209575641730212X

2. J. M. Twenge, T. E. Joiner, M. L. Rogers, and G. N. Martin, “Increases in depressive symptoms, suicide-related outcomes, and suicide rates among u.s. adolescents after 2010 and links to increased new media screen time,” vol. 6, no. 1, pp. 3–17, publisher: SAGE Publications Inc. [Online]. Available: https://doi.org/10.1177/2167702617723376

3. E. Frank and A. D. Dingle, “Self-reported depression and suicide attempts among u.s. women physicians,” vol. 156, no. 12, pp. 1887– 1894, publisher: American Psychiatric Publishing. [Online]. Available: https://ajp.psychiatryonline.org/doi/full/10.1176/ajp.156.12.1887

4. L. Shu, J. Xie, M. Yang, Z. Li, Z. Li, D. Liao, X. Xu, and X. Yang, “A review of emotion recognition using physiological signals,” vol. 18, no. 7, p. 2074, number: 7 Publisher: Multidisciplinary Digital Publishing Institute. [Online]. Available: https://www.mdpi.com/1424-8220/18/7/2074

5. B. Mandal, L. Li, G. S. Wang, and J. Lin, “Towards detection of bus driver fatigue based on robust visual analysis of eye state,” vol. 18, no. 3, pp. 545–557, conference Name: IEEE Transactions on Intelligent Transportation Systems.

6. R. Bhardwaj, P. Natrajan, and V. Balasubramanian, “Study to determine the effectiveness of deep learning classifiers for ECG based driver fatigue classification,” in 2018 IEEE 13th International Conference on Industrial and Information Systems (ICIIS), pp. 98–102, ISSN: 2164-7011.

7. N. Cummins, B. Vlasenko, H. Sagha, and B. Schuller, “Enhancing speech-based depression detection through gender dependent vowel-level formant features,” in Artificial Intelligence in Medicine, ser. Lecture Notes in Computer Science, A. ten Teije, C. Popow, J. H. Holmes, and L. Sacchi, Eds. Springer International Publishing, pp. 209–214.

8. D. G. Amen, P. Krishnamani, S. Meysami, A. Newberg, and C. A. Raji, “Classification of depression, cognitive disorders, and co-morbid depression and cognitive disorders with perfusion SPECT neuroimaging,” vol. 57, no. 1, pp. 253–266, publisher: IOS Press. [Online]. Available: https://content.iospress.com/articles/journalof-alzheimers-disease/jad161232

9. D. Palacios, V. Rodellar, C. Lazaro, A. G ´ omez, and P. G ´ omez, ´ “An ICA-based method for stress classification from voice samples,” vol. 32, no. 24, pp. 17 887–17 897. [Online]. Available: https://doi.org/10.1007/s00521-019-04549-3

10. R. Castaldo, L. Montesinos, P. Melillo, C. James, and L. Pecchia, “Ultra-short term HRV features as surrogates of short term HRV: a case study on mental stress detection in real life,” vol. 19, no. 1, p. 12. [Online]. Available: https://doi.org/10.1186/s12911-019-0742-y

11. R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain, “Speech emotion recognition using deep learning techniques: A review,” vol. 7, pp. 117 327–117 345, conference Name: IEEE Access.

12. N. Jain, S. Kumar, A. Kumar, P. Shamsolmoali, and M. Zareapoor, “Hybrid deep neural networks for face emotion recognition,” vol. 115, pp. 101–106. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167865518301302

13. G. Waterstraat, T. Fedele, M. Burghoff, H.-J. Scheer, and G. Curio, “Recording human cortical population spikes non-invasively–an EEG tutorial,” vol. 250, pp. 74–84.

14. P. Sarma, P. Tripathi, M. P. Sarma, and K. K. Sarma, “Pre-processing and feature extraction techniques for EEG- BCI applications- a review of recent research,” p. 8.

15. A. Craik, Y. He, and J. L. Contreras-Vidal, “Deep learning for electroencephalogram (EEG) classification tasks: a review,” vol. 16, no. 3, p. 031001. [Online]. Available: https://iopscience.iop.org/article/10.1088/1741-2552/ab0ab5

16. Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep learning-based electroencephalography analysis: a systematic review,” vol. 16, no. 5, p. 051001. [Online]. Available: https://iopscience.iop.org/article/10.1088/1741-2552/ab260c

17. M. Chan, D. Esteve, J.-Y. Fourniols, C. Escriba, and ` E. Campo, “Smart wearable systems: Current status and future challenges,” vol. 56, no. 3, pp. 137–156. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0933365712001182