Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
 

Data Profiling (lecture, master's course)

Prof. Dr. Felix Naumann
Youri Kaminsky (exercises)

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the dataset at hand.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

In this lecture, we will study efficient algorithms and data structures to handle the typically vast search spaces of data profiling tasks. The techniques are applicable not only to data profiling, but provide general ideas of algorithmically dealing with large-scale data.

Lectures will be given in English.

Schedule

DateTopic
Mo 17.04.2023Introduction and motivation
Tu 18.04.2023 
Mo 24.04.2023 
Tu 25.04.2023 
Mo 01.05.2023Feiertag - Tag der Arbeit
Tu 02.05.2023 
Mo 08.05.2023 
Tu 09.05.2023 
Mo 15.05.2023 
Tu 16.05.2023 
Mo 22.05.2023 
Tu 23.05.2023 
Mo 29.05.2023Pfingstmontag
Tu 30.05.2023 
Mo 05.06.2023 
Tu 06.06.2023 
Mo 12.06.2023 
Tu 13.06.2023 
Mo 19.06.2023 
Tu 20.06.2023 
Mo 26.06.2023 
Tu 27.06.2023 
Mo 03.07.2023 
Tu 04.07.2023 
Mo 10.07.2023 
Tu 11.07.2023 
Mo 17.07.2023 
Tu 18.07.2023 
Mo 24.07.2023 
Tu 25.07.2023 

Literature

The course largely follows the following textbook, which we will supply:

Data Profiling - Synthesis Lectures on Data Management
Ziawasch Abedjan, Lukasz Golab, Felix Naumann, Thorsten Papenbrock

In addition, each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See also the following two articles for an overview on data profiling:

Exam

We plan to have a written exam on February 23 at 1pm in HS1. To qualify for the exam you need to successfully complete the exercises (success defined during exercises).