Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

Description

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

Additional information

  • Lectures are given in English.
  • Slides will be made available on the HPI-internal materials-folder.

Schedule - to be updated

Schedule:

  • Mondays at 15:15 in HS 2 and
  • Thursdays at 13:30 in HS 2

The following schedule is subject to change.

DateTopicSlides
Mo 17.04.  Easter Monday
Th 20.04.Big Data Introduction
Mo 24.04.Exercise
Th 27.04.Data Profiling Introduction
Mo 01.05.

Labor Day

Th 04.05.

new room

ACHTUNG: Hauptgebäude, D-Space 3. OG
Mo 08.05.
Th 11.05.
Mo 15.05.no lecture
Th 18.05.no lecture
Mo 22.05.
Th 25.05.Ascension Day
Mo 29.05.Exercise
Th 01.06.
Mo 05.06.Pentecost
Th 08.06.
Mo 12.06.
Th 15.06.
Mo 19.06.
Th 22.06.
Mo 26.06.
Th 29.06.
Mo 03.07.
Th 06.07.
Mo 10.07.
Th 13.07.
Mo 17.07.
Th 20.07.
Mo 24.07.Data profiling challenges and outlook
Th 27.07.Exam preparation
TBDExam

Literature

The course does not follow a textbook. Each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See the following two articles for an overview on data profiling:

These two books (mostly on data mining) are also of general interest to the lecture:

  • Jiawei Han, Micheline Kamber, Jian Pei: Data Mining: Concepts and Techniques
  • Dorian Pyle: Data Preparation for Data Mining

Exam

A written exam will take place on ???.