Hasso-Plattner-InstitutSDG am HPI
Hasso-Plattner-InstitutDSG am HPI

Human Mental State Analysis using wearable Electroencephalography

Christoph Anders

Chair for Digital Health - Connected Healthcare
Hasso Plattner Institute

Office: Campus III Building G2, Room G-2.1.21
Tel.: +49 331 5509-3412
Email: Christoph.Anders(at)hpi.de 
Links: Homepage

Supervisor: Prof. Dr. Bert Arnrich

Starting Date: 01.04.2021

We analyze wearable electroencephalography (EEG) devices on their capabilities of classifying and predicting general mental states of humans. Amongst others, of special interest are a) utilizing multiple physiological signals recorded in real-world scenarios for multi-modal setups, b) quantifying the impact of data pre-processing on time-series classification (TSC) model performance as well as on intra- and inter-subject variability, and c) supporting the development of applicaitons for wearable EEG outside of laboratory environments.

Research Area: Mental State Classification with wearable EEG

The mental state of a person can be directly linked to human errors or subjective decisions [1] - [3]. Most straight-forward, it can be assessed with questionnaires. However, this type of exams may not suffice in case of acute dangerous situations. In general, the mental states such as emotions can be classified from physiological signals [4], which can be analyzed with sensors like camera systems, electrocardiogram (ECG), by using speech analysis, or by observing the heart-rate variability (HRV), amongst others [4] - [12]. Another measurement technique suitable for mental state classification is the EEG. While clinical EEG systems have been around for nearly a century, they might require a lot of time to set up, are limited in mobility and clinical EEG systems are not designed to look inconspicuous. Wearable EEG systems have been developed, which commonly offer only a limited amount of recording channels, usually do not allow to record data for periods of time longer than a day, and are more likely to experience signal distortion due to artefacts. Artefacts can distort the signal of interest significantly---hence, to optimize the signal quality, it is first and foremost advisable to minimize the influence of 'noise' due to artefacts---and can be of biological or technical origin [13]. 

Once the data is recorded, usually multiple steps are performed to increase the signal-to-noise ratio (SNR). So-called data pre-processing steps can be spatial filtering (e.g. by building a bipolar montage or interpolating bad channels), spectral filtering (e.g. to remove spectrally localized noise such as AC/DC noise), or temporal filtering (e.g. by means of interpolating recordings to filter out stimulus artefacts, if any). Afterwards, features can be extracted from the pre-processed data and feature-transformations might increase information content. Features are commonly of a) spectral nature, derived with Power Spectral Density, Fourier Transformation, or Wavelet Transformation, of b) amplitude-related nature like statistical measures, or c) entropy-related information [14] - [16]. The features are utilized for TSC model building, with TSC models commonly being threshold-functions, Machine Learning (ML) models, or Deep Learning (DL) models. While challenges like reproducability across studies due to model-, data-, and code-availability, are cumbersome for research on wearable EEG, many more challenges are faced in this field, such as system reliability, user privacy, interoperability between wearable devices, end-user training, and social inclusion, amongst others [17].

We are working towards increaseing trust in wearable EEG systems. This includes identifying pressing challenges in the field, highlighting future perspectives, and supporting the development of wearable EEG applications outside of laboratory environments. In order to do so, multiple milestones have to be reached. This includes development of a 'test-battery' for wearable EEG devices and related pipelines as well as the development of a similarity-score for mental state elicitation in a safe laboratory environment as opposed to real-life recordings. Finally, the exemplary development of an application for non-medical, assistive use based on wearable EEG, e.g. in elderly-care, would allow for evaluation of afforementioned efforts.

Current Research: From multi-modal time-series data to a wide range of assistive applications

To establish a solid foundation for my future work on current pressing challenges, timely developments, and future trends in the field of mental state classification using wearable EEG, I conducted a systematic literature review titled Wearable Electroencephalography for Mental State Classification: A Systematic Literature Review together with Prof. Dr. Bert Arnrich. The search initially led to 461 publications, which were reduced to 54 items after duplicate removal, quality control and topic matching, subsequently analyzed in the categories data, recording device, questionnaire, stimulus, data pre-processing steps, task-handling, and training set-up used. Comparing the results and further notes, we found that most studies use a rather small frequency band (Delta to Gamma), use spectrum- or amplitude-based features, and utilize support vector machines, artificial neural networks, and threshold-based TSC models for classification of the time-series data. Additionally, inter-subject training paradigms are utilized seldomly with wearable EEG, whereas studies using clinical-grade EEG oftentimes tackle inter-subject variability directly [16]. While TSC performance was found to vary strongly across architectures, training paradigms, problems, and recording devices uses, it was found that reproducibility across studies is one of the biggest challenges. This literature review supported my idea of creating a 'test-battery' for wearable EEG devices and related pipelines, which I will tackle within future projects.

Comparison of wearable EEG devices. Left: EEG electrodes placed within an EEG cap, as utilized also in clinical-grade EEG setups, and available as wearable devices e.g. from OpenBCI. Right: somewhat inconspicuous wearable EEG device, offering less channels at the expense of being less versatile than clinical-grade EEG. The development of applicaitons for wearable EEG devices is an open field with many challenges and opportunities.

In my master thesis Experimental Evaluation of data pre-processing methods for time series classification on brain activity data, I worked on experiments with varying input-formulation techniques on various challenging EEG datasets of lower- and higher-frequencies. Together with Dr. med. Gunnar Waterstraat, Prof. Dr. med. Gabriel Curio, Dr. Nico Steckhan, Sidratul Moontaha, and Prof. Dr. Bert Arnrich we investigated the research questions: 1) How much information increment is provided by different steps of a pre-processing pipeline?, 2) For within-subject training, which input formulation seems most promising, and which architecture (is) to use?, and 3) Exists an optimum for averaging consecutive trials, taking into account natural variance of (high-frequency Somatosensory Evoked Potentials) hfSEP and noise?, amongst others. Together with Dr. med. Gunnar Waterstraat, and supervised by Prof. Dr. med. Gabriel Curio and Prof. Dr. Bert Arnrich, we prepared the paper Optimization of Data Pre-Processing Methods for Time-Series Classification of Electroencephalography Data, which is currently undergoing the last steps of the publication process, and which increased my interest in the field of EEG.

Workflow representation of my masters thesis. The impact of data pre-processing on three different datasets, two publicly available and one available upon reasonable request, was analyzed using different spectral and spatial filtering techniques as well as feature transformations. Out of the methods analyzed, optimal spatial filtering and subsequently applying the non-linear Hilbert Transform resulted in the highest, significant, TSC performance increments.

Teaching Activities

Together with Sidratul Moontaha, and guided by Prof. Dr. Bert Arnrich, I am supervising the Digital Health - Connected Healthcare chair's Master Project for the Winter Term 2021/2022 called: Human Emotion and Activity Classification using Brain Activity Sensors. Amongst others, we hope to set a baseline for identification of the impact of different activities and emotions on classifying either emotions, activites, or both jointly in an intra- and inter-subject approach, giving our students the possibility to gather hands-on experience in experimental paradigm creation, data recording in a laboratory environment and outside-of-the-lab, and in classifying real-life data.

Graphical abstract of the winter term's master project 2021/2022. Students are expected to work on emotion, activity, and joint classification of both, analyze the impact their data pre-processing approaches have, and get in touch with medical experts to validate their findings.