Prof. Dr. Felix Naumann

Data Profiling (lecture, master's course)

Prof. Dr. Felix Naumann
Youri Kaminsky (exercises)

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the dataset at hand.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing. Topics include an introduction, data structures, unique column combinations, functional dependencies, inclusion dependencies, order dependencies, denial constraints, and semantic interpretation of profiling results.

In this lecture, we will study efficient algorithms and data structures to handle the typically vast search spaces of data profiling tasks. The techniques are applicable not only to data profiling, but provide general ideas of algorithmically dealing with large-scale data. The topics include:

  • Foundations: data structures and basic algorithms
  • Single-column profiling
  • Discovery of unique column combinations
  • Functional dependency discovery
  • Order dependency discovery
  • Inclusion dependency discovery
  • Query optimization using dependencies
  • Semantic data profiling
  • Denial constraint discovery
  • Violation detection


  • Lecture hall: L.E-03
  • Mondays 13:30 - 15:00 
  • Tuesdays 15:15 - 16:45
  • Lectures will be given in English.
  • The exercises will be coordinated in Moodle.
Mo 17.04.2023Organization & introduction
Tu 18.04.2023
ROOM L.1-06
Introduction & motivation
Mo 24.04.2023Data structures
Tu 25.04.2023Metanome intro & Data structures
Mo 01.05.2023Feiertag - Tag der Arbeit
Tu 02.05.2023Data structures + Q&A
Mo 08.05.2023Unique Column Combinations (UCCs)
Tu 09.05.2023Lecture & Exercise: UCCs
Mo 15.05.2023Functional Dependencies (FDs)
Tu 16.05.2023Functional Dependencies (FDs)
Mo 22.05.2023Functional Dependencies (FDs)
Tu 23.05.2023Functional Dependencies (FDs)
Mo 29.05.2023Pfingstmontag
Tu 30.05.2023Order Dependencies (Sebastian Schmidl)
Mo 05.06.2023Order Dependencies (Sebastian Schmidl)
Tu 06.06.2023Exercise: UCCs reports and FD task
Mo 12.06.2023Inclusion Dependencies
Tu 13.06.2023Inclusion Dependencies
Mo 19.06.2023Query Optimization using Dependencies (Daniel Lindner)
Tu 20.06.2023no lecture
Mo 26.06.2023Inclusion Dependencies
Tu 27.06.2023on your own: Inclusion Dependencies (BINDER)
Mo 03.07.2023Denial Constraints
Tu 04.07.2023Denial Constraints
Mo 10.07.2023Exercise: FD reports and IND task
Tu 11.07.2023DC Violation Detection (Youri Kaminsky)
Mo 17.07.2023Cardinality estimation (Hazar Harmouch)
Tu 18.07.2023Genuine dependencies
Mo 24.07.2023no lecture
Tu 25.07.2023Exercise: IND reports; Exam prep Q&A
Tu 08.08.2023
10am - 12 am in F.E-06
Written exam


The course largely follows the following textbook, which we will supply on demand:

Data Profiling - Synthesis Lectures on Data Management
Ziawasch Abedjan, Lukasz Golab, Felix Naumann, Thorsten Papenbrock

In addition, each lecture references various scientific articles and other sources of information. Good sources to find those articles are

See also the following two articles for an overview on data profiling:


We plan to have a written exam on tbd at tbd in tbd. To qualify for the exam you need to successfully complete the exercises (success defined during exercises).