Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

Description

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

Additional information

  • Lectures can be given in English.
  • Slides will be made available on the HPI-internal materials-folder.

Schedule

Schedule:

  • Mondays at 15:15 in HS 2 and
  • Thursdays at 13:30 in HS 2

The following schedule is subject to change.

DateTopicSlides
Mo 17.04.  Easter Monday
Th 20.04.Big Data IntroductionPDF
Mo 24.04.Exercise
Th 27.04.Data Profiling IntroductionPDF
Mo 01.05.Labor Day
Th 04.05.Data Profiling Introduction
ROOM: Hauptgebäude, D-Space 3. OG
Mo 08.05.Basic Data StructuresPDF
Th 11.05.Unique Column Combinations
Mo 15.05.no lecture
Th 18.05.no lecture
Mo 22.05.Data Profiling Experiments PDF
Th 25.05.Ascension Day
Mo 29.05.Exercise
Th 01.06.Functional DependenciesPDF
Mo 05.06.Pentecost
Th 08.06.Functional Dependencies
Mo 12.06.Functional Dependencies (Thorsten Papenbrock)PDF
Th 15.06.Inclusion Dependencies
Mo 19.06.Exercise
Th 22.06.Inclusion DependenciesPDF
Mo 26.06.Inclusion Dependencies (Sebastian Kruse)PDF
Th 29.06.SemanticsPDF
Mo 03.07.Thomas Bläsius: Complexity of Data Profiling
Th 06.07.
Mo 10.07.Exercise
Th 13.07.Bachelorpodium
Mo 17.07.Denial Constraints (Tobias Bleifuß)PDF
Th 20.07.Cardinality Estimation (Hazar Harmouch)PDF
Tu 25.07.
17 Uhr in HS2
Giuseppe Polese: Approximate Data Profiling
Th 27.07.
17 Uhr in HS2
Data profiling challenges and outlook / Exam preparation
Tu 08.08.Exam in HS 2

Literature

The course does not follow a textbook. Each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See the following two articles for an overview on data profiling:

These two books (mostly on data mining) are also of general interest to the lecture:

  • Jiawei Han, Micheline Kamber, Jian Pei: Data Mining: Concepts and Techniques
  • Dorian Pyle: Data Preparation for Data Mining

Exam

A written exam will take place on August 8, 2017 in HS 2.