We presented our research on using RDMA Multicasts to improve database operations at the 2016 OpenFabrics Workshop in Monterey, CA. In our talk, we demonstrated how databases can profit from hardware-assisted multicast operations and what impact this has on the performance of operations. This research is done as a collaborative project of our group and SAP Labs in Palo Alto.
For more information, please contact Markus Dreseler.
Abstract:
Modern distributed database systems heavily rely on the underlying networking infrastructure. In order to minimize the impact of network operations on query processing performance, high-throughput and low-latency network technologies such as InfiniBand and RDMA are being utilized. The resulting implications on expensive data processing operations such as joins are well studied.
With a constantly growing number of nodes per distributed database system, pieces of information have to be shared with multiple nodes and group communication operations become more relevant. Although such operations are well known and frequently used in the high-performance computing community (mainly due to MPI), there is little traction in the database (and big data) community. It also seems as if there is little information available that quantifies the benefits from hardware-assisted multicasts, which in turn makes it more difficult to justify the engineering efforts and monetary investment. This talk focuses on using RDMA multicast in the context of distributed database operations and covers the following three aspects:
- Which operations in a distributed database system can benefit from RDMA multicast?
- Which frameworks and tools are available for a database systems engineer to make use of RDMA multicast?
- Presentation of experimental results that quantify a) the benefits of RDMA multicast (in comparison to uni- cast/broadcast) and b) the associated costs (such as establishing a multicast group).
