Talk "Disco: Efficient Distributed Window Aggregation" at Flink Forward 2020

Join Flink Forward 2020 (it's free) and check out our talk Disco: Efficient Distributed Window Aggregation by Lawrence Benson on Thursday, 22 October at 20:00. 

The talk is available on Youtube. Check it out here: https://www.youtube.com/watch?v=EC6kz64CqMY

Modern stream processing engines (SPEs) provide complex window types and user-defined aggregation functions to analyze streams. While SPEs run in central data centers, wireless sensors networks (WSNs) perform distributed aggregations close to the data sources, which is beneficial especially in modern IoT setups. However, WSNs support only basic aggregations and windows. To bridge the gap between complex central aggregations and simple distributed analysis, we present Disco, a distributed complex window aggregation approach.

Disco is an outlook into a core challenge that future SPEs face and a solution for how to tackle it. Disco processes complex window types on multiple independent nodes while efficiently aggregating incoming data streams. Our evaluation shows that Disco’s throughput scales linearly with the number of nodes and that Disco already outperforms a centralized solution in a two-node setup. Furthermore, Disco reduces the network cost significantly compared to the centralized approach.

The talk is based on our research paper Disco: Efficient Distributed Window Aggregation, which was published at EDBT 2020 (https://openproceedings.org/2020/conf/edbt/paper_300.pdf).

The code for Disco is open source and available on Github: https://github.com/hpides/disco.