Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

Data Profiling

Description

Prof. Dr. Felix Naumann

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

Additional information

  • Lectures can be given in English.
  • Slides will be made available on the HPI-internal materials-folder.
     

Schedule

In this semester we plan to prepare the main lecture as videos and reserve a weekly lecture hall for Q&A, exercises and discussions. We will meet in person Tuesdays 13:30 in HS 1; these meetings shall also be available online. Details on how the course is planned will follow.

DateTopicSlides
 Big Data Introduction 
 Exercise 
   
   
   

Literature

The course largely follows the following textbook, which we will supply:

Data Profiling - Synthesis Lectures on Data Management
Ziawasch Abedjan, Lukasz Golab, Felix Naumann, Thorsten Papenbrock

In addition, each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See also the following two articles for an overview on data profiling:

Exam

We plan a written exam. To qualify for the exam you need to successfully complete the exercises (success defined during exercises).