1.
Ihde, N., Marten, P., Eleliemy, A., Poerwawinata, G., Silva, P., Tolovski, I., Ciorba, M.F., Rabl, T.: A Survey of Big Data, High Performance Computing, and Machine Learning Benchmarks. In: Nambiar, R. and Poess, M. (eds.) Lecture Notes in Computer Science. pp. 1–21 (2021).
In recent years, there has been a convergence of Big Data (BD), High Performance Computing (HPC), and Machine Learning (ML) systems. This convergence is due to the increasing complexity of long data analysis pipelines on separated software stacks. With the increasing complexity of data analytics pipelines comes a need to evaluate their systems, in order to make informed decisions about technology selection, sizing and scoping of hardware. While there are many benchmarks for each of these domains, there is no convergence of these efforts. As a first step, it is also necessary to understand how the individual benchmark domains relate. In this work, we analyze some of the most expressive and recent benchmarks of BD, HPC, and ML systems. We propose a taxonomy of those systems based on individual dimensions such as accuracy metrics and common dimensions such as workload type. Moreover, we aim at enabling the usage of our taxonomy in identifying adapted benchmarks for their BD, HPC, and ML systems. Finally, we identify challenges and research directions related to the future of converged BD, HPC, and ML system benchmarking.
2.
Böther, M., Kißig, O., Benson, L., Rabl, T.: Drop It In Like It’s Hot: An Analysis of Persistent Memory as a Drop-in Replacement for NVMe SSDs. International Workshop on Data Managment on New Hardware (DAMON’21), June 20–25, 2021, Virtual Event, China. ACM (2021).
Solid-state drives (SSDs) have improved database system performance significantly due to the higher bandwidth that they provide over traditional hard disk drives. Persistent memory (PMem) is a new storage technology that offers DRAM-like speed at SSD-like capacity. Due to its byte-addressability, research has mainly treated PMem as a replacement of, or an addition to DRAM, e.g., by proposing highly-optimized, DRAM-PMem-hybrid data structures and system designs. However, PMem can also be used via a regular file system interface and standard Linux I/O operations. In this paper, we analyze PMem as a drop-in replacement for Non-Volatile Memory Express (NVMe) SSDs and evaluate possible performance gains while requiring no or only minor changes to existing applications. This drop-in approach speeds-up database systems like Postgres, without requiring any code changes. We systematically evaluate PMem and NVMe SSDs in three database microbenchmarks and the widely used TPC-H benchmark on Postgres. Our experiments show that PMem outperforms a RAID of four NVMe SSDs in read-intensive OLAP workloads by up to 4x without any modifications while achieving similar performance in write-intensive workloads. Finally, we give four practical insights to aid decision-making on when to use PMem as an SSD drop-in replacement and how to optimize for it.
3.
Benson, L., Makait, H., Rabl, T.: Viper: An Efficient Hybrid PMem-DRAM Key-Value Store. Proceedings of the VLDB Endowment. 14, 1544–1556 (2021).
Key-value stores (KVSs) have found wide application in modern software systems. For persistence, their data resides in slow secondary storage, which requires KVSs to employ various techniques to increase their read and write performance from and to the underlying medium. Emerging persistent memory (PMem) technologies offer data persistence at close-to-DRAM speed, making them a promising alternative to classical disk-based storage. However, simply drop-in replacing existing storage with PMem does not yield good results, as block-based access behaves differently in PMem than on disk and ignores PMem's byte addressability, layout, and unique performance characteristics. In this paper, we propose three PMem-specific access patterns and implement them in a hybrid PMem-DRAM KVS called Viper. We employ a DRAM-based hash index and a PMem-aware storage layout to utilize the random-write speed of DRAM and efficient sequential-write performance PMem. Our evaluation shows that Viper significantly outperforms existing KVSs for core KVS operations while providing full data persistence. Moreover, Viper outperforms existing PMem-only, hybrid, and disk-based KVSs by 4--18x for write workloads, while matching or surpassing their get performance.
4.
Böther, M., Rabl, T.: Scale-Down Experiments on TPCx-HS. Big Data in Emergent Distributed Environments (BiDEDE’21), June 20, 2021, Virtual Event, China. ACM (2021).
The Transaction Processing Performance Council's (TPC) benchmarks are the standard for evaluating data processing performance and are extensively used in academia and industry. Official TPC results are usually produced on high-end deployments, making transferability to commodity hardware difficult. Recent performance improvements on low-power ARM CPUs have made low-end computers, such as the Raspberry Pi, a candidate platform for distributed, low-scale data processing. In this paper, we conduct a feasibility study of executing scaled-down big data workloads on low-power ARM clusters. To this end, we run the TPCx-HS benchmark on two Raspberry Pi clusters. TPCx-HS is the ideal candidate for hardware comparisons and understanding hardware characteristics for data processing workloads because TPCx-HS results do not depend on specific software implementations and the benchmark has limited options for workload-specific tuning. Our evaluation shows that Pis exhibit similar behavior to large-scale big data systems in terms of price performance and relative throughput to performance results. Current generation Pi clusters are becoming a reasonable choice for GB-scale data processing due to the increasing amount of available memory, while older versions struggle with stable execution of high-load scenarios.
5.
Gévay, G.E., Rabl, T., Breß, S., Madai-Tahy, L., Quiané-Ruiz, J.-A., Markl, V.: Efficient Control Flow in Dataflow Systems: When Ease-of-Use Meets High Performance. 37th IEEE International Conference on Data Engineering. (2021).
6.
Daase, B., Bollmeier, L.J., Benson, L., Rabl, T.: Maximizing Persistent Memory Bandwidth Utilization for OLAP Workloads. Proceedings of the 2021 International Conference on Management of Data (SIGMOD ’21), June 20--25, 2021, Virtual Event, China. ACM (2021).
Modern database systems for online analytical processing (OLAP) typically rely on in-memory processing. Keeping all active data in DRAM severely limits the data capacity and makes larger deployments much more expensive than disk-based alternatives. Byte-addressable persistent memory (PMEM) is an emerging storage technology that bridges the gap between slow-but-cheap SSDs and fast-but-expensive DRAM. Thus, research and industry have identified it as a promising alternative to pure in-memory data warehouses. However, recent work shows that PMEM's performance is strongly dependent on access patterns and does not always yield good results when simply treated like DRAM. To characterize PMEM's behavior in OLAP workloads, we systematically evaluate PMEM on a large, multi-socket server commonly used for OLAP workloads. Our evaluation shows that PMEM can be treated like DRAM for most read access but must be used differently when writing. To support our findings, we run the Star Schema Benchmark on PMEM and DRAM. We show that PMEM is suitable for large, read-heavy OLAP workloads with an average query runtime slowdown of 1.66x compared to DRAM. Following our evaluation, we present 7 best practices on how to maximize PMEM's bandwidth utilization in future system designs.
7.
Menon, P., M. Qadah, T., Rabl, T., Sadoghi, M., Jacobsen, H.-A.: LogStore: A Workload-aware, Adaptable Key-Value Store on Hybrid Storage Systems. 37th IEEE International Conference on Data Engineering. (2021).
8.
Traub, J., Grulich, P.M., Cuéllar, A.R., Breß, S., Katsifodimos, A., Rabl, T., Markl, V.: Scotty: General and Efficient Open-Source Window Aggregation for Stream Processing Systems. Transactions on Database Systems. 46, 46 (2021).
