Prof. Dr. Felix Naumann

Distributed Data Management

Lecturer: Dr. Thorsten Papenbrock


The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization.

Big data analytics and, hence, data management are a multi-million dollar markets that grow constantly! Data and the ability to control and use it is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements.

In this lecture, we take a look at various technologies involved in building distributed, data-intensive systems. We discuss theoretical concepts (data models, encoding, replication, ...) as well as some of their practical implementations (Akka, MapReduce, Spark, ...).

Topic list

  • Foundations of Distributed Data Management
  • OLAP and OLTP
  • Encoding and Evolution
  • Hands-on Akka
  • Data Models and Query Languages
  • Storage and Retrieval
  • Replication
  • Partitioning
  • Batch Processing
  • Hands-on Spark
  • Distributed Systems
  • Consistency and Consensus
  • Transaction Processing
  • Stream Processing
  • Hands-on Flink
  • Services and Containerization
  • Cloud-based Data Systems


Thelecure is taking place in room F-E.06 (Campus II) Mondays at 1:30 PM and Tuesday at 03:15 PM. The lecture is held in English and a recording is availabe on tele-TASK.

21.10.Encoding and Communication
22.10.Encoding and Communication
28.10.Hands-on Akka Actor Programming
29.10.<no lecture> (HPI Dies Academicus)
04.11.Hands-on Akka Actor Programming
05.11.Hands-on Akka Actor Programming
11.11.Data Models and Query Languages
12.11.Storage and Retrieval
25.11.Distributed Systems
26.11.Distributed Systems
02.12.Consistency and Consensus
09.12.Batch Processing
10.12.<no lecture> (sick)
16.12.Batch Processing
17.12.Batch Processing
Christmas break
06.01.Assignment 1-3 Evaluation
07.01.Hands-on Spark Batch Processing
13.01.Hands-on Spark Batch Processing (tpch.zip)
14.01.<no lecture> (guest)
20.01.Stream Processing
21.01.Stream Processing
27.01.Distributed DBMSs
28.01.Distributed DBMSs
03.02.Distributed Query Optimization
04.02.Assignment 4 Evaluation and Lecture Summary



The grade will determined in an exam. The exam will take place on February 21 in the time from 14:00 to 17:00 AM in HS 1. The prerequisite for admission to the exam is the successful completion of the exercises.

To practice for the exam, you can use the exams of WS_2017/18 (solution), WS_2018/19 (solution), and WS_2019/20 (solution). You can also find check-yourself questions in the slides (solution).


Course book:

  • Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017, 978-1449373320

Further reading:

  • Distributed Systems, Maarten van Steen and Andrew S. Tanenbaum, 2017, 978-1543057386
  • Web-Scale Data Management for the Cloud, Wolfgang Lehner and Kai-Uwe Sattler, 2013, 1489997717
  • Introduction to Parallel Computing, Zbigniew J. Czech, 2017, 978-1107174399
  • Principles of Distributed Database Systems, M. Tamer Özsu and Patrick Valduriez, 2011, 978-1441988331
  • Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services, Brendan Burns, 2017, 978-1491983645
  • Spark: Big Data Cluster Computing in Production, Ilya Ganelin and Ema Orhian and Kai Sasaki and Brennon York, 2016, 978-1119254010
  • Reactive Messaging Patterns with the Actor Model, Vaughn Vernon, 2015, 978-0133846836
  • Mining Massive Datasets, Jure Leskovec and Anand Rajaraman and Jeffrey David Ullman, 2014, 978-1107077232
  • Algorithmische Geometrie, Rolf Klein, 2005, 978-3540209560