Ilin Tolovski presented the paper A Survey of Big Data, High Performance Computing, and Machine Learning Benchmarks by Nina Ihde, Paula Marten, Ahmed Eleliemy, Gabrielle Poerwawinata, Pedro Silva, Ilin Tolovski, Florina M. Ciorba, and Tilmann Rabl. The paper is the result of a research collaboration in the Horizon 2020 DAPHNE Project between the Hasso-Plattner-Institute and the High Performance Computing Group from University of Basel.
In recent years, there has been a convergence of Big Data (BD), High Performance Computing (HPC), and Machine Learning (ML) systems. This convergence is due to the increasing complexity of long data analysis pipelines on separated software stacks. With the increasing complexity of data analytics pipelines comes a need to evaluate their systems, in order to make informed decisions about technology selection, sizing and scoping of hardware. While there are many benchmarks for each of these domains, there is no convergence of these efforts. As a first step, it is also necessary to understand how the individual benchmark domains relate. In this work, we analyze some of the most expressive and recent benchmarks of BD, HPC, and ML systems. We propose a taxonomy of those systems based on individual dimensions such as accuracy metrics and common dimensions such as workload type. Moreover, we aim at enabling the usage of our taxonomy in identifying adapted benchmarks for their BD, HPC, and ML systems. Finally, we identify challenges and research directions related to the future of converged BD, HPC, and ML system benchmarking.
Besides session chairing at the conference, Tilmann Rabl co-chaired the Ph.D. Workshop on Monday, August 16, and Ilin Tolovski, Rafael Moczalla and Tilmann Rabl were local chairs for the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC) on Friday, August 20.