While one might argue whether parallel query execution is required for transactional query processing, analytical queries on large data sets need to run in parallel to achieve acceptable performance for real-time analytics. With 80 cores and counting per computing node, sequential processing of queries is not an option, as we cannot expect single queries to fully utilize all available resources throughout execution. Several challenges arise if we allow intra-query parallelism, e.g., what if we need to run two queries in parallel? A naïve approach would be to divide the cores by two and run each query on half the cores. However, due to architectural restrictions (e.g., data partitioning, memory channels, bandwidth restrictions), operator implementations (e.g., sequential code, sync points) and data characteristics (e.g., table sizes, number of distinct values) most queries / operators will most likely show sub-linear scaling curves. And what happens if one query finishes earlier, or if further queries arrive during execution? Much research has been conducted in the area of parallel query execution in the area of disk-based database management systems (e.g., [RM95], [WC04]). However, we see three main changes that do not allow to simply adopt previous work to In-Memory Databases:
- More processing units: We have much more parallel processing units in one computing node (many-core CPUs) and see a trend for further growth
- Faster execution time: Queries are executed much faster in an In-Memory Database (rather seconds compared to minutes or hours); hence the overhead of running a complex scheduling algorithm easily accounts for a significant portion of total execution time of a query
- No disk: Much of the previous work has been focused on avoiding the disk I/O bottleneck. Disk is not a bottleneck for complex, read-intensive queries on In-Memory databases anymore
Due to these changes, existing work on parallel query processing has to be revisited for In-Memory databases and potentially adapted.
