Data Profiling and Data Cleansing (Sommersemester 2013)

Lecturer: Prof. Dr. Felix Naumann (Information Systems)

General Information

Weekly Hours: 4
Credits: 6
Graded: yes
Enrolment Deadline: 10.2.2013 - 30.4.2013
Teaching Form: VU
Enrolment Type: Compulsory Elective Module

Programs, Module Groups & Modules

IT-Systems Engineering BA

IT-Systems Engineering MA

IT-Systems Engineering A
IT-Systems Engineering B
IT-Systems Engineering C
IT-Systems Engineering D

Description

According to Wikipedia, data profiling is the process of examining the data available in an existing data source [...] and collecting statistics and information about that data. It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.

Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing.

Many of the insights gained during data profiling point to deficiencies of the data. Profiling reveals data errors, such as inconsistent formatting within a column, missing values, or outliers. Profiling results can also be used to measure and monitor the general quality of a dataset, for instance by determining the number of records that do not conform to previously established constraints. The second part of the lecture examines various methods and algorithms to improve the quality of data, with an emphasis on the many existing duplicate detection approaches.

Literature

siehe Webseite

Learning

Vorlesung und Übung

Examination

Erfolgreiche Teilnahme an der Übung
Mündliche Prüfung oder Klausur am Ende der Vorlesungszeit

Dates

Tuesdays / Thursdays 9:15 - 10:45 in HS 1

Zurück