Distributed Data Management (Sommersemester 2020)

Lecturer: Dr. Thorsten Papenbrock (Information Systems)
Course Website: https://hpi.de/en/naumann/teaching/teaching/ss-20/distributed-data-management-vl-master.html

General Information

Weekly Hours: 4
Credits: 6
Graded: yes
Enrolment Deadline: 06.04.2020 - 22.04.2020
Teaching Form: Lecture / Exercise
Enrolment Type: Compulsory Elective Module
Course Language: English

Programs, Module Groups & Modules

Data Engineering MA

Description

IMPORTANT NOTE 1: This lecture is a catch-up course for HPI students in the "Data Engineering" track that start their studies in SS2020. The lecture is given annually, but because we switched from winder- to summer-periodes, this iteration of the lecture is only for these students. It is one of their core modules and its next iteration in one year is too late; hence, the extra iteration this semester.

IMPORTANT NOTE 2: Due to the current COVID-19 situation, we need to start this lecture in online-mode. This means that we use teleTask recordings for the teaching sessions and jitsi web-meetings for on-boarding, question, hands-on and homework sessions. Please find the detailed descriptions of our schedule on our web page.

The free lunch is over! Computer systems up until the turn of the century became constantly faster without any particular effort simply because the hardware they were running on increased its clock speed with every new release. This trend has changed and today's CPUs stall at around 3 GHz. The size of modern computer systems in terms of contained transistors (cores in CPUs/GPUs, CPUs/GPUs in compute nodes, compute nodes in clusters), however, still increases constantly. This caused a paradigm shift in writing software: instead of optimizing code for a single thread, applications now need to solve their given tasks in parallel in order to expect noticeable performance gains. Distributed computing, i.e., the distribution of work on (potentially) physically isolated compute nodes is the most extreme method of parallelization.

Big data analytics and, hence, data management are a multi-million dollar markets that grow constantly! Data and the ability to control and use it is the most valuable ability of today's computer systems. Because data volumes grow so rapidly and with them the complexity of questions they should answer, data analytics, i.e., the ability of extracting any kind of information from the data becomes increasingly difficult. As data analytics systems cannot hope for their hardware getting any faster to cope with performance problems, they need to embrace new software trends that let their performance scale with the still increasing number of processing elements.

In this lecture, we take a look at various technologies involved in building distributed, data-intensive systems. We discuss theoretical concepts (data models, encoding, replication, ...) as well as some of their practical implementations (Akka, MapReduce, Spark, ...).

Topic list:

Introduction
Foundations
Encoding & Communication
Akka Actor Programming
Data Models & Query Languages
Storage & Retrieval
Replication
Partitioning
Distributed Systems
Consistency & Consensus
Transactions
Batch Processing
Spark Batch Processing
Stream Processing
Distributed DBMS
Distributed Query Optimization

Literature

Course book:

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017, 978-1449373320

Learning

Lectures and Exercises

Examination

The grade will determined in an oral exam. The prerequisite for admission to the exam is the successful completion of the exercises.

Dates

Please see web page.

Zurück