Current Stream Processing Engines (SPEs) can process millions of records across hundreds of nodes to analyze an ever growing amount of real-time data. However, the most widely used SPEs, such as Apache Flink, Spark Streaming, or Storm are all JVM-based and do not utilize the servers'
hardware efficiently. Recent work [1] shows that many hardware-specific optimizations can be made to improve the efficiency of each individual node (i.e., scale-up instead of scale-out). These optimizations include, e.g., query compilation, use of modern storage devices, or use of RDMA.
We are currently developing a new SPE for modern hardware at our group, which tackles some of these challenges. In this project, you will actively develop components of this new system, which we plan to open source soon. The project is still in very early stages, so there are many topics to choose from, depending on your interests. Some ideas are: efficient data distribution between operators via RDMA, efficient hardware-optimized streaming joins and aggregations, efficient checkpointing and recovery, efficient query compilation via architecture-aware optimizations, query plan optimization, and many more.
Students will learn the inner workings of stream processors and data management systems in general, with a particular focus on query compilation and modern hardware. It is targeting students interested in acquiring skills in data management, stream processing, data flows, compilers, and low- level systems programming. The project will be implemented in C++.
Information on query compiled SPEs can be found in the recent SIGMOD ’20 paper Grizzly [2]. General information and an introduction on stream processing can be found in the O’Reilly blog posts by Tyler Akidau [3, 4] and the stream processing book [5].
If you have any questions, please reach out to Lawrence Benson.
