The Enterprise Stream Processing Benchmark

Motivation

Why another benchmark?

The ever increasing amount of data that is produced nowadays, from smart homes to smart factories, gives rise to completely new challenges and opportunities. Terms like "Internet of Things" (IoT) and "Big Data" have gained traction to describe the creation and analysis of these new data masses. Furthermore, new technologies and systems were developed that are able to handle and analyze data streams, i.e., data arriving with high frequency and in large volume.

In recent years, e.g., a lot of distributed Data Stream Processing Systems were developed, whose usage represents one way of analyzing data streams. Although a broad variety of systems or system architectures is generally a good thing, the bigger the choice, the harder it is to choose.

Benchmarking is a common and proven approach to identify the best system for a specific set of needs. However, currently, no satisfying benchmark for modern data stream processing architectures exists. Particularly when an enterprise context, i.e., where data streams have to be combined with historical and transactional data, existing benchmarks have shortcomings. The Enterprise Streaming Benchmark (ESB), which is to be developed, will attempt to tackle this issue.

Objectives

We aim to create a relevant, real-world application benchmark with a focus on data stream processing architectures in an enterprise context. This involves including business or transactional data stored in a traditional database into the analysis process. In order to ease usage of the developed benchmark, we will develop a comprehensive toolkit for supporting implementation as well as setup and execution. Particularly, it will comprise tools for system setup, data ingestion, data validation and benchmark result analysis.

With regard to the domain, we will focus on an industrial manufacturing context. So the defined benchmark queries will answer questions in that area. Our primary data source will be sensor data from, e.g., industrial machinery. Therefore, we aim to work with real-world data.

Contact

Contact: Guenter Hesse

Github repository: https://github.com/guenter-hesse/ESPBench

Research Area: In-Memory Data Management for Enterprise Systems

Selected Publications

Application of Data Stream Processing Technologies in Industry 4.0: What is Missing?Hesse, Guenter; Sinzig, Werner; Matthies, Christoph; Uflacker, Matthias (2019). 304-310.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019application, abstract = {Industry 4.0 is becoming more and more important for manufacturers as the developments in the area of Internet of Things advance. Another technology gaining more attention is data stream processing systems. Although such streaming frameworks seem to be a natural fit for Industry 4.0 scenarios, their application in this context is still low. The contributions in this paper are threefold. Firstly, we present industry findings that we derived from site inspections with a focus on Industry 4.0. Moreover, our view on Industry 4.0 and important related aspects is elaborated. As a third contribution, we illustrate our opinion on why data stream processing technologies could act as an enabler for Industry 4.0 and point out possible obstacles on this way.}, author = {Hesse, Guenter and Sinzig, Werner and Matthies, Christoph and Uflacker, Matthias}, booktitle = {Proceedings of the 8th International Conference on Data Science, Technology and Applications - Volume 1: DATA}, keywords = {data_stream_processing esb industry4.0}, organization = {INSTICC}, pages = {304-310}, publisher = {SciTePress}, title = {Application of Data Stream Processing Technologies in Industry 4.0: What is Missing?}, year = 2019 }
Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems.Hesse, Guenter; Matthies, Christoph; Glass, Kelvin; Huegle, Johannes; Uflacker, Matthias (2019). 1381-1392.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019quantitative, abstract = {With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.}, author = {Hesse, Guenter and Matthies, Christoph and Glass, Kelvin and Huegle, Johannes and Uflacker, Matthias}, booktitle = {2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)}, keywords = {data_stream_processing esb imported}, pages = {1381-1392}, title = {Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems}, year = 2019 }
Adding Value by Combining Business and Sensor Data: An Industry 4.0 Use Case.Hesse, Guenter; Matthies, Christoph; Sinzig, Werner; Uflacker, Matthias G. Li, Yang, J., Gama, J., Natwichai, J., Tong, Y. (reds.) (2019). (Vol. 11448) 528-532.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019adding, abstract = {Industry 4.0 and the Internet of Things are recent developments that have lead to the creation of new kinds of manufacturing data. Linking this new kind of sensor data to traditional business information is crucial for enterprises to take advantage of the data's full potential. In this paper, we present a demo which allows experiencing this data integration, both vertically between technical and business contexts and horizontally along the value chain. The tool simulates a manufacturing company, continuously producing both business and sensor data, and supports issuing ad-hoc queries that answer specific questions related to the business. In order to adapt to different environments, users can configure sensor characteristics to their needs.}, author = {Hesse, Guenter and Matthies, Christoph and Sinzig, Werner and Uflacker, Matthias}, booktitle = {Database Systems for Advanced Applications. DASFAA 2019. Lecture Notes in Computer Science}, editor = {Li, Guoliang and Yang, Jun and Gama, Joao and Natwichai, Juggapong and Tong, Yongxin}, keywords = {business_data data_integration esb industry4.0 sensor_data}, month = {April}, pages = {528-532}, publisher = {Springer International Publishing}, title = {Adding Value by Combining Business and Sensor Data: An Industry 4.0 Use Case}, volume = 11448, year = 2019 }
Senska - Towards an Enterprise Streaming Benchmark.Hesse, Guenter; Reissaus, Benjamin; Matthies, Christoph; Lorenz, Martin; Kraus, Milena; Uflacker, Matthias (2018). 25-40.
[ BibTeX ] [ Download ]

@inproceedings{hesse2018senska, abstract = {In the light of growing data volumes and continuing digitization in fields such as Industry 4.0 or Internet of Things, data stream processing have gained popularity and importance. Especially enterprises can benefit from this development by augmenting their vital, core business data with up-to-date streaming information. Enriching this transactional data with detailed information from high-frequency data streams allows answering new analytical questions as well as improving current analyses, e.g., regarding predictive maintenance. Comparing such data stream processing architectures for use in an enterprise context, i.e., when combining streaming and business data, is currently a challenging task as there is no suitable benchmark.}, address = {Cham}, author = {Hesse, Guenter and Reissaus, Benjamin and Matthies, Christoph and Lorenz, Martin and Kraus, Milena and Uflacker, Matthias}, booktitle = {TPC Technology Conference (TPCTC)}, keywords = {benchmarking data_stream_processing esb senska}, pages = {25-40}, title = {Senska - Towards an Enterprise Streaming Benchmark}, type = {Publication}, year = 2018 }
A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium.Hesse, Guenter; Matthies, Christoph; Reissaus, Benjamin; Uflacker, Matthias in DEBS '17 (2017). 359-362.
[ BibTeX ] [ Download ]

@inproceedings{Guenter2017a, abstract = {Against the backdrop of ever-growing data volumes and trends like the Internet of Things (IoT) or Industry 4.0, Data Stream Processing Systems (DSPSs) or data stream processing architectures in general receive a greater interest. Continuously analyzing streams of data allows immediate responses to environmental changes. A challenging task in that context is assessing and comparing data stream processing architectures in order to identify the most suitable one for certain settings. The present paper provides an overview about performance benchmarks that can be used for analyzing data stream processing applications. By describing shortcomings of these benchmarks, the need for a new application benchmark in this area, especially for a benchmark covering enterprise architectures, is highlighted. A key role in such an enterprise context is the combination of streaming data and business data, which is barely covered in current data stream processing benchmarks. Furthermore, first ideas towards the development of a solution, i.e., a new application benchmark that is able to fill the existing gap, are depicted.}, author = {Hesse, Guenter and Matthies, Christoph and Reissaus, Benjamin and Uflacker, Matthias}, booktitle = {ACM International Conference on Distributed and Event-based Systems (DEBS)}, keywords = {Benchmark_Development Data_Stream_Processing Internet_of_Things Performance_Benchmarking Stream_Processing esb}, pages = {359-362}, series = {DEBS '17}, title = {A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium}, year = 2017 }
Conceptual Survey on Data Stream Processing Systems.Hesse, Guenter; Lorenz, Martin in IEEE International Conference on Parallel and Distributed Systems (ICPADS) (2015). 797-802.
[ BibTeX ] [ Download ]

@article{Guenter2015a, author = {Hesse, Guenter and Lorenz, Martin}, journal = {IEEE International Conference on Parallel and Distributed Systems (ICPADS)}, keywords = {Flink Samza Spark Storm Stream_Processing esb}, pages = {797-802}, title = {Conceptual Survey on Data Stream Processing Systems}, year = 2015 }