In many big data scenarios, data comes with high speed as a never-ending stream of events. For these data streams, decisions need to be made often on the fly. Unfortunately, common algorithms are rarely applicable in scenarios with streaming data. Most algorithms were designed for offline settings, i.e., the entire data set needs to be scanned and processed (multiple times), before a decision can be made. Therefore, novel algorithms on data streams are needed.
In this seminar, we implement, evaluate (and at best improve) streaming algorithms from current research projects. We will look at data stream mining, recommendations for data streams, and algorithms for graph streams where edges and vertices arrive in a streaming fashion. Students will develop data streaming techniques and implement prototypes based on current research projects: Each team, consisting of two students, chooses and presents a challenging research task and implements the proposed solution using the streaming framework Apache Kafka with Kafka Streams. Students may select one of the papers listed here:
- Chaitanya Manapragada, Geoffrey I. Webb, and Mahsa Salehi, Extremely Fast Decision Tree, KDD 2018.
See also https://github.com/chaitanya-m/kdd2018.git - Kai Sheng Tai, Vatsal Sharan, Peter Bailis, and Gregory Valiant, Sketching Linear Classifiers over Data Streams, SIGMOD 2018.
See also https://github.com/stanford-futuredata/wmsketch - Yang Zhou, Tong Yang, Jie Jiang, Bin Cui, Minlan Yu, Xiaoming Li, and Steve Uhlig, Cold Filter: A Meta-Framework for Faster and More Accurate Stream Processing, SIGMOD 2018.
See also https://github.com/streamclassifier/ColdFilter - Aneesh Sharma, Jerry Jiang, Praveen Bommannavar, Brian Larso, and Jimmy Lin, GraphJet: Real-Time Content Recommendations at Twitter, VLDB 2016.
See also https://github.com/twitter/GraphJet - Xiangmin Zhou, Dong Qin, Xiaolu Lu, Lei Chen, and Yanchun Zhang, Online Social Media Recommendation over Streams, ICDE 2019.
- Dhivya Eswaran, Christos Faloutsos, Sudipto Guha, and Nina Mishra, SpotLight: Detecting Anomalies in Streaming Graphs. KDD 2018.
This is a first selection of current research about data streams (tbc). We welcome further suggestions.
This is a project seminar: There will be a few weekly lectures including an introductory lecture and an invited talk from industry about Stream Processing with Apache Kafka. Teams will frequently meet with the assigned supervisor.
Now that the project finished successfully, the teams' implementations are available online:
