Distributed Data Management (Sommersemester 2021)

Dozent: Dr. Thorsten Papenbrock (Information Systems)
Website zum Kurs: https://hpi.de/naumann/teaching/current-courses/ss-21/distributed-data-management.html

Allgemeine Information

• Semesterwochenstunden: 4
• ECTS: 6
• Benotet: Ja
• Einschreibefrist: 18.03.2021 - 09.04.2021
• Lehrform: Vorlesung / Übung
• Belegungsart: Wahlpflichtmodul
• Lehrsprache: Englisch

Studiengänge, Modulgruppen & Module

Beschreibung

The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization.

Big data analytics and management are a multi-million dollar markets that grow constantly! The ability to control and utilize large amounts of data is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements.

In this lecture, we take a look at various technologies involved in building distributed, data-intensive systems. We start by discussing fundamental concepts in distributed computing, such das data models, encoding formats, messaging, data replication and partitioning, fault tollerance, and batch- and stream processing. In between, we consider different practical systems from the Big Data Landscape, such as Akka and Spark. In the end, we concentrate on data management aspects, such as distributed database management system architectures and distributed query optimization.

Voraussetzungen

To take this class, a basic understanding of object-oriented programming, data structures and relational databases is required. If you cannot programm fluently in at least one object-oriented language (e.g. Java, C#, C++, Python, Ruby etc.) taking this class will be hard.

During the exercises, participants will need to program in Java and Scala. In preparation of the course, we therefore recomment to get familiar with these two languages or refresh your knowledge. We expect no deep knowledge, but you should know the basic language constructs and be able to create, for instance, a Maven Java programm and an SBT Scala programm that both read a file and count all words in these two files.

Literatur

Course books:

• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017, 978-1449373320
• Distributed Systems, Maarten van Steen and Andrew S. Tanenbaum, 2017, 978-1543057386
• Principles of Distributed Database Systems, M. Tamer Özsu and Patrick Valduriez, 2011, 978-1441988331

Further reading:

• Web-Scale Data Management for the Cloud, Wolfgang Lehner and Kai-Uwe Sattler, 2013, 1489997717
• Introduction to Parallel Computing, Zbigniew J. Czech, 2017, 978-1107174399
• Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services, Brendan Burns, 2017, 978-1491983645
• Spark: Big Data Cluster Computing in Production, Ilya Ganelin and Ema Orhian and Kai Sasaki and Brennon York, 2016, 978-1119254010
• Reactive Messaging Patterns with the Actor Model, Vaughn Vernon, 2015, 978-0133846836
• Mining Massive Datasets, Jure Leskovec and Anand Rajaraman and Jeffrey David Ullman, 2014, 978-1107077232
• Algorithmische Geometrie, Rolf Klein, 2005, 978-3540209560

Lern- und Lehrformen

Lectures and Exercises

Leistungserfassung

The final grade will be determined in a written exam. Please see the course's web page for more details.

Termine

Please see the course's web page.

Zurück