The Enterprise Stream Processing Benchmark

Motivation

Why another benchmark?

The ever increasing amount of data that is produced nowadays, from smart homes to smart factories, gives rise to completely new challenges and opportunities. Terms like "Internet of Things" (IoT) and "Big Data" have gained traction to describe the creation and analysis of these new data masses. Furthermore, new technologies and systems were developed that are able to handle and analyze data streams, i.e., data arriving with high frequency and in large volume.

In recent years, e.g., a lot of distributed Data Stream Processing Systems were developed, whose usage represents one way of analyzing data streams. Although a broad variety of systems or system architectures is generally a good thing, the bigger the choice, the harder it is to choose.

Benchmarking is a common and proven approach to identify the best system for a specific set of needs. However, currently, no satisfying benchmark for modern data stream processing architectures exists. Particularly when an enterprise context, i.e., where data streams have to be combined with historical and transactional data, existing benchmarks have shortcomings. The Enterprise Streaming Benchmark (ESB), which is to be developed, will attempt to tackle this issue.

Objectives

We aim to create a relevant, real-world application benchmark with a focus on data stream processing architectures in an enterprise context. This involves including business or transactional data stored in a traditional database into the analysis process. In order to ease usage of the developed benchmark, we will develop a comprehensive toolkit for supporting implementation as well as setup and execution. Particularly, it will comprise tools for system setup, data ingestion, data validation and benchmark result analysis.

With regard to the domain, we will focus on an industrial manufacturing context. So the defined benchmark queries will answer questions in that area. Our primary data source will be sensor data from, e.g., industrial machinery. Therefore, we aim to work with real-world data.

Contact

Contact: Guenter Hesse

Github repository: https://github.com/guenter-hesse/ESPBench

Research Area: In-Memory Data Management for Enterprise Systems

Selected Publications

ESPBench: The Enterprise Stream Processing Benchmark.Hesse, Guenter; Matthies, Christoph; Perscheid, Michael; Uflacker, Matthias; Plattner, Hasso (2021). 201–212.
[ BibTeX ] [ Download ]

@inproceedings{hesse2021espbench, abstract = {Growing data volumes and velocities in fields such as Industry 4.0 or the Internet of Things have led to the increased popularity of data stream processing systems. Enterprises can leverage these developments by enriching their core business data and analyses with up-to-date streaming data. Comparing streaming architectures for these complex use cases is challenging, as existing benchmarks do not cover them. ESPBench is a new enterprise stream processing benchmark that fills this gap. We present its architecture, the benchmarking process, and the query workload. We employ ESPBench on three state-of-the-art stream processing systems, Apache Spark, Apache Flink, and Hazelcast Jet, using provided query implementations developed with Apache Beam. Our results highlight the need for the provided ESPBench toolkit that supports benchmark execution, as it enables query result validation and objective latency measures.}, author = {Hesse, Guenter and Matthies, Christoph and Perscheid, Michael and Uflacker, Matthias and Plattner, Hasso}, booktitle = {ACM/SPEC Internation Conference on Performance Engineering (ICPE)}, keywords = {benchmarking esb imported}, pages = {201–212}, title = {ESPBench: The Enterprise Stream Processing Benchmark}, year = 2021 }
How Fast Can We Insert? An Empirical Performance Evaluation of Apache Kafka.Hesse, Guenter; Matthies, Christoph; Uflacker, Matthias (2021). 641-648.
[ BibTeX ] [ Download ]

@inproceedings{hesse2021insert, abstract = {Message brokers see widespread adoption in modern IT landscapes, with Apache Kafka being one of the most employed platforms. These systems feature well-defined APIs for use and configuration and present flexible solutions for various data storage scenarios. Their ability to scale horizontally enables users to adapt to growing data volumes and changing environments. However, one of the main challenges concerning message brokers is the danger of them becoming a bottleneck within an IT architecture. To prevent this, knowledge about the amount of data a message broker using a specific configuration can handle needs to be available. In this paper, we propose a monitoring architecture for message brokers and similar Java Virtual Machine-based systems. We present a comprehensive performance analysis of the popular Apache Kafka platform using our approach. As part of the benchmark, we study selected data ingestion scenarios with respect to their maximum data ingestion rates. The results show that we can achieve an ingestion rate of about 420,000 messages/second on the used commodity hardware and with the developed data sender tool.}, author = {Hesse, Guenter and Matthies, Christoph and Uflacker, Matthias}, booktitle = {IEEE International Conference on Parallel and Distributed Systems (ICPADS)}, keywords = {esb imported kafka}, pages = {641-648}, title = {How Fast Can We Insert? An Empirical Performance Evaluation of Apache Kafka}, year = 2021 }
Application of Data Stream Processing Technologies in Industry 4.0: What is Missing?Hesse, Guenter; Sinzig, Werner; Matthies, Christoph; Uflacker, Matthias (2019). 304-310.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019application, abstract = {Industry 4.0 is becoming more and more important for manufacturers as the developments in the area of Internet of Things advance. Another technology gaining more attention is data stream processing systems. Although such streaming frameworks seem to be a natural fit for Industry 4.0 scenarios, their application in this context is still low. The contributions in this paper are threefold. Firstly, we present industry findings that we derived from site inspections with a focus on Industry 4.0. Moreover, our view on Industry 4.0 and important related aspects is elaborated. As a third contribution, we illustrate our opinion on why data stream processing technologies could act as an enabler for Industry 4.0 and point out possible obstacles on this way.}, author = {Hesse, Guenter and Sinzig, Werner and Matthies, Christoph and Uflacker, Matthias}, booktitle = {International Conference on Data Science, Technology and Applications (DATA)}, keywords = {data_stream_processing esb industry4.0}, organization = {INSTICC}, pages = {304-310}, publisher = {SciTePress}, title = {Application of Data Stream Processing Technologies in Industry 4.0: What is Missing?}, year = 2019 }
Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems.Hesse, Guenter; Matthies, Christoph; Glass, Kelvin; Huegle, Johannes; Uflacker, Matthias (2019). 1381-1392.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019quantitative, abstract = {With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.}, author = {Hesse, Guenter and Matthies, Christoph and Glass, Kelvin and Huegle, Johannes and Uflacker, Matthias}, booktitle = {IEEE International Conference on Distributed Computing Systems (ICDCS)}, keywords = {data_stream_processing esb imported}, pages = {1381-1392}, title = {Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems}, year = 2019 }
Adding Value by Combining Business and Sensor Data: An Industry 4.0 Use Case.Hesse, Guenter; Matthies, Christoph; Sinzig, Werner; Uflacker, Matthias G. Li, Yang, J., Gama, J., Natwichai, J., Tong, Y. (reds.) (2019). (Vol. 11448) 528-532.
[ BibTeX ] [ Download ]

@inproceedings{hesse2019adding, abstract = {Industry 4.0 and the Internet of Things are recent developments that have lead to the creation of new kinds of manufacturing data. Linking this new kind of sensor data to traditional business information is crucial for enterprises to take advantage of the data's full potential. In this paper, we present a demo which allows experiencing this data integration, both vertically between technical and business contexts and horizontally along the value chain. The tool simulates a manufacturing company, continuously producing both business and sensor data, and supports issuing ad-hoc queries that answer specific questions related to the business. In order to adapt to different environments, users can configure sensor characteristics to their needs.}, author = {Hesse, Guenter and Matthies, Christoph and Sinzig, Werner and Uflacker, Matthias}, booktitle = {International Conference on Database Systems for Advanced Applications (DASFAA)}, editor = {Li, Guoliang and Yang, Jun and Gama, Joao and Natwichai, Juggapong and Tong, Yongxin}, keywords = {business_data data_integration esb industry4.0 sensor_data}, month = {April}, pages = {528-532}, publisher = {Springer International Publishing}, title = {Adding Value by Combining Business and Sensor Data: An Industry 4.0 Use Case}, volume = 11448, year = 2019 }
Senska - Towards an Enterprise Streaming Benchmark.Hesse, Guenter; Reissaus, Benjamin; Matthies, Christoph; Lorenz, Martin; Kraus, Milena; Uflacker, Matthias (2018). 25-40.
[ BibTeX ] [ Download ]

@inproceedings{hesse2018senska, abstract = {In the light of growing data volumes and continuing digitization in fields such as Industry 4.0 or Internet of Things, data stream processing have gained popularity and importance. Especially enterprises can benefit from this development by augmenting their vital, core business data with up-to-date streaming information. Enriching this transactional data with detailed information from high-frequency data streams allows answering new analytical questions as well as improving current analyses, e.g., regarding predictive maintenance. Comparing such data stream processing architectures for use in an enterprise context, i.e., when combining streaming and business data, is currently a challenging task as there is no suitable benchmark.}, address = {Cham}, author = {Hesse, Guenter and Reissaus, Benjamin and Matthies, Christoph and Lorenz, Martin and Kraus, Milena and Uflacker, Matthias}, booktitle = {TPC Technology Conference (TPCTC)}, keywords = {benchmarking data_stream_processing esb senska}, pages = {25-40}, title = {Senska - Towards an Enterprise Streaming Benchmark}, type = {Publication}, year = 2018 }
A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium.Hesse, Guenter; Matthies, Christoph; Reissaus, Benjamin; Uflacker, Matthias in DEBS '17 (2017). 359-362.
[ BibTeX ] [ Download ]

@inproceedings{Guenter2017a, abstract = {Against the backdrop of ever-growing data volumes and trends like the Internet of Things (IoT) or Industry 4.0, Data Stream Processing Systems (DSPSs) or data stream processing architectures in general receive a greater interest. Continuously analyzing streams of data allows immediate responses to environmental changes. A challenging task in that context is assessing and comparing data stream processing architectures in order to identify the most suitable one for certain settings. The present paper provides an overview about performance benchmarks that can be used for analyzing data stream processing applications. By describing shortcomings of these benchmarks, the need for a new application benchmark in this area, especially for a benchmark covering enterprise architectures, is highlighted. A key role in such an enterprise context is the combination of streaming data and business data, which is barely covered in current data stream processing benchmarks. Furthermore, first ideas towards the development of a solution, i.e., a new application benchmark that is able to fill the existing gap, are depicted.}, author = {Hesse, Guenter and Matthies, Christoph and Reissaus, Benjamin and Uflacker, Matthias}, booktitle = {ACM International Conference on Distributed and Event-based Systems (DEBS)}, keywords = {Benchmark_Development Data_Stream_Processing Internet_of_Things Performance_Benchmarking Stream_Processing esb}, pages = {359-362}, series = {DEBS '17}, title = {A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium}, year = 2017 }
Conceptual Survey on Data Stream Processing Systems.Hesse, Guenter; Lorenz, Martin in IEEE International Conference on Parallel and Distributed Systems (ICPADS) (2015). 797-802.
[ BibTeX ] [ Download ]

@article{Guenter2015a, author = {Hesse, Guenter and Lorenz, Martin}, journal = {IEEE International Conference on Parallel and Distributed Systems (ICPADS)}, keywords = {Flink Samza Spark Storm Stream_Processing esb}, pages = {797-802}, title = {Conceptual Survey on Data Stream Processing Systems}, year = 2015 }