Data Profiling

Description

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

Additional information

• Lectures can be given in English.
• Slides will be made available on the HPI-internal materials-folder.

Schedule

Schedule:

• Mondays at 15:15 in HS 2 and
• Thursdays at 13:30 in HS 2

The following schedule is subject to change.

Literature

The course does not follow a textbook. Each lecture references various scientific articles and other sources of information. Good sources to find those articles are

• DBLP
• ACM's Digital Library
• Google Scholar
• Author's homepages

See the following two articles for an overview on data profiling:

• A short introductory article: Felix Naumann, Data Profiling Revisited, SIGMOD Record 2013
• A survey: Ziawasch Abedjan, Lukasz Golab and Felix Naumann: Profiling relational data: a survey, VLDB Journal vol 24(4), pages 557 - 581, 2015

These two books (mostly on data mining) are also of general interest to the lecture:

• Jiawei Han, Micheline Kamber, Jian Pei: Data Mining: Concepts and Techniques
• Dorian Pyle: Data Preparation for Data Mining

Exam

A written exam will take place on August 8, 2017 in HS 2.