In-memory databases are widely used in the software industry due to their high query processing performance. A key reason for this performance is storing the majority of the data on local DRAM, which results in low data access latencies compared to slower secondary storage devices, such as flash SSDs. However, for analytical databases, DRAM capacity is a limited resource since the size of data to be processed continuously grows. In traditional data centers, the basic unit is a monolithic server, accommodating small quantities of compute, storage, and network resources. This co-location of the different resources prevents independent scaling of individual resource types and, thus, makes it hard to fulfill the highly elastic and growing memory demands from data-intensive applications. Furthermore, scaling out either computing or storage capacity in a co-located hardware setup implies the need to over-provision the other resource type. One way to tackle these elasticity and utilization shortcomings is a composable data center architecture, which separates traditional server resources into independent compute, storage, and network pools.
Memory disaggregation is a promising technique for achieving elasticity and improving resource utilization. In a research project collaborating with Seagate Technology, we investigate how disaggregated memory can be utilized for an in-memory database to expand the local memory capacity of a database server. Disaggregated memory can be considered as an additional tier in the memory hierarchy, with certain bandwidth and latency qualities. Our aspects of interest are memory management mechanisms to efficiently store data on disaggregated memory and autonomous data placement algorithms. For our research, we use Hyrise as an exemplary in-memory database and a hardware prototype for disaggregated memory provided by Seagate Technology.
Contact: Dr. Daniel Ritter, Marcel Weisgut, Martin Boissier, Dr. Michael Perscheid
Research Area: Autonomous Data Management
