Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

Data Profiling

Note: Please register yourself for the openHPI course online and also contact the Studienreferat to let them know that you intend to take the course.

Description

Prof. Dr. Felix Naumann
Tobias Bleifuß (exercises)

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

Additional information

  • Lectures will be given in English.
  • Slides, lecture videos and other material will be made available on openHPI (only accessible for HPI students with their student email addresses).
  • Please register for the openHPI course online and also contact the Studienreferat to let them know that you intend to take the course.

Schedule

This semester the lecture will take place in a flipped hybrid classroom mode. The main lecture will be available as videos on openHPI. We will meet weekly for Q&A, exercises and discussions. You can attend these weekly meetings either in person Tuesdays 13:30 in F-E.06, or online via Zoom. The Zoom link will be distributed through the openHPI platform.
With the begin of the lockdown period in Brandenburg, we have switched to an online-only mode for the time being.

DateTopicVideos
03.11.2020no meeting 
10.11.2020Q&A: Data Profiling Introduction & ExerciseIntroduction (up to Video 5.4)
17.11.2020Q&A: Unique Column CombinationsUnique column combinations (up to Video 6.6)
24.11.2020Q&A: Cardinality estimation and FDs (part 1)Cardinality estimation (Videos 6.7 and 6.8) and functional dependencies (up to Video 7.5)
01.12.2020Q&A: FDs part 2HPIValid (Videos 6.9-6.11) and TANE (Videos 7.6 - 7.13)
08.12.2020Exercise presentations on UCCs and new exercise on FDs 
15.12.2020Q&A: HyFD and INDs part 1fdep and HyFD (Videos 7.14 - 7.15) and unary INDs (Videos 8.1 - 8.7)
22.12.2020Academic Christmas holidays 
29.12.2020Academic Christmas holidays 
05.01.2021Q&A: INDs part 2n-ary INDs (Videos 8.8 - 8.10)
12.01.2021Exercise presentations on FDs and new exercise on INDs 
19.01.2021Q&A Order DependenciesODs (Videos 9.1 - 9.10)
26.01.2021Q&A Denial ConstraintsDCs (Videos 10.1 - 10.5)
02.02.2021Presentations: Exercise on INDs 
09.02.2021Q&A Probabilistic keys (and semantics)Prob. Keys (Videos 11.1 - 11.5)
16.2.2021, 14 UhrExam preparation 
23.2.2021, 13 Uhr
Hörsaalgebäude
Written exam 

Literature

The course largely follows the following textbook, which we will supply:

Data Profiling - Synthesis Lectures on Data Management
Ziawasch Abedjan, Lukasz Golab, Felix Naumann, Thorsten Papenbrock

In addition, each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See also the following two articles for an overview on data profiling:

Exam

We plan to have a written exam on February 23 at 1pm in HS1. To qualify for the exam you need to successfully complete the exercises (success defined during exercises).