In-Memory Data Management on Modern Hardware
High performance in-memory computing will change how enterprises work. Currently, enterprise data is split into two databases for performance reasons. Usually, disk-based row-oriented database systems are used for operational data and column-oriented databases are used for analytics (e.g. “sum of all sales in China grouped by product”). While analytical databases are often kept in-memory, they are often also mixed with disk-based storage media. Transactional data and analytical data are not stored in the same database: analytical data resides in separate data warehouses, to which it is replicated in batch jobs. As a result, flexible real-time reporting is not possible and leaders are forced to make decisions based on insufficient information in very short time frames. This is about to change, since hardware architectures have evolved dramatically during the past decade. Multi-core architectures and the availability of large amounts of main memory at low costs are about to set new breakthroughs in the software industry. It has become possible to store data sets of entire Fortune 500 companies in main memory. At the same time, orders of magnitude faster performance than with disk-based systems can be achieved. Traditional disks are one of the last remaining mechanical devices in a world of silicon and are about to become what tape drives are today: a device only necessary for backup. With in-memory computing and hybrid databases using both row and column-oriented storage where appropriate, transactional and analytical processing can be unified.
At the research group “Enterprise Platform and Integration Concepts“ of Prof. Dr. Hasso Plattner at the Hasso Plattner Institute, we are conducting research projects with the goal of revolutionizing enterprise systems and applications based on them. One of our projects focuses on building an in-memory hybrid database that unifies the advantages of column- and row-oriented database systems; another project analyzes how in-memory databases can be used in a Software-as-a-Service environment. In corporation projects with SAP and using real customer data, we showed that with in-memory column-oriented databases the time for business transactions, like dunning, could be reduced from 20 minutes to one second. We are also augmenting Available-to-Promise applications with real time analytics and flexible order fulfillment. Our vision is that in-memory computing enables completely new ways of how businesses are run and operated through new business applications.