Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
 

30.05.2022

Evaluation and demo paper accepted at VLDB 2022

We are excited to announce that our paper titled "Anomaly Detection in Time Series: A Comprehensive Evaluation" and the corresponding demo paper titled "TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms" have been accepted at the VLDB Conference 2022 and will appear in PVLDB Vol. 15, No. 9 and 12 respectively.

Anomaly Detection in Time Series: A Comprehensive Evaluation

Authors

Sebastian Schmidl, Phillip Wenig, Thorsten Papenbrock

Abstract

Detecting anomalous subsequences in time series data is an important task in areas ranging from manufacturing processes over finance applications to health care monitoring. An anomaly can indicate important events, such as production faults, delivery bottlenecks, system defects, or heart flicker, and is therefore of central interest. Because time series are often large and exhibit complex patterns, data scientists have developed various specialized algorithms for the automatic detection of such anomalous patterns. The number and variety of anomaly detection algorithms has grown significantly in the past and, because many of these solutions have been developed independently and by different research communities, there is no comprehensive study that systematically evaluates and compares the different approaches. For this reason, choosing the best detection technique for a given anomaly detection task is a difficult challenge.

This comprehensive, scientific study carefully evaluates most state-of-the-art anomaly detection algorithms. We collected and re-implemented 71 anomaly detection algorithms from different domains and evaluated them on 976 time series datasets. The algorithms have been selected from different algorithm families and detection approaches to represent the entire spectrum of anomaly detection techniques. In the paper, we provide a concise overview of the techniques and their commonalities; we evaluate their individual strengths and weaknesses and, thereby, consider factors, such as effectiveness, efficiency, and robustness. Our experimental results should ease the algorithm selection problem and open up new research directions.

Synthetic multivariate time series with a correlation anomaly and the scoring of k-Means.
Synthetic univariate time series resembling an ECG signal with a subsequence anomaly (pattern shift), a point anomaly (extremum), and the scorings of LSTM-AD and Sub-LOF.

TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms

Authors

Phillip Wenig, Sebastian Schmidl, Thorsten Papenbrock

Abstract

Detecting anomalous subsequences in time series is an important task in time series analytics because it serves the identification of special events, such as production faults, delivery bottlenecks, system defects, or heart flicker. Consequently, many algorithms have been developed for the automatic detection of such anomalous patterns. The enormous number of approaches (i.e., more than 158 as of today), the lack of properly labeled test data, and the complexity of time series anomaly benchmarking have, though, led to a situation where choosing the best detection technique for a given anomaly detection task is a difficult challenge.

In this demonstration, we present TimeEval, an extensible, scalable and automatic benchmarking toolkit for time series anomaly detection algorithms. TimeEval includes an extensive data generator and supports both interactive and batch evaluation scenarios. With our novel toolkit, we aim to ease the evaluation effort and help the community to provide more meaningful evaluations.

Architecture of the TimeEval toolkit.