Prof. Dr. Felix Naumann


Stratosphere is a joint DFG project conducted by the Technische Universität Berlin, Humboldt Universität Berlin, and the Hasso-Plattner-Institut. It explores how the elasticity of clouds can be exploited for processing analytic queries massively in parallel. Unlike most traditional DBMS, Stratosphere inherently supports text-based and semi-structured data.

Official Project Site

The sub-projects at HPI focus on data quality improvements of linked open data, efficient and scalable data profiling, and knowledge discoevry.

Data Cleansing

We defined the declarative data cleansing language Meteor, implement the underlying basic operations, and develop cost estimations for the operations. Furthermore, we provide test data sets and example queries to evaluate the efficiency and effectivity of the data cleansing process.

Data Profiling

Detecting dependencies in the evergrowing amounts of data has a high computational complexity. One way to cope with this complexity is to distribute the computational work among multiple interconnected computers. However, most existing data profiling algorithms are not designed for parallel execution on computer clusters but rather to run on a single machine. Therefore, we research distributed modifications of existing algorithms as well as new algorithms that can be efficiently executed on computer clusters and that scale out on the number of the cluster nodes.

Knowledge Discovery

Driven by applications such as social media analytics, Web search, advertising, recommendation, mobile sensoring, genomic sequencing, astronomical observations, etc., the need for scalable learning, mining, and knowledge discovery methods is steadily growing. Often the challenge is to automatically process and analyze TBs of evolving data. Extracting value (e.g., understanding the underlying structure and making predictions) from such data, before it is outdated, is a major concern. Therefore, the goal is to enable the scalability of such applications based on Stratosphere.

Please contact Felix Naumann, Toni Grütze (Knowledge Discovery on Stratosphere), or Sebastian Kruse (Data Profiling on Stratosphere) for further questions.

Former members


Scaling Out the Discovery of Inclusion Dependencies

Kruse, Sebastian; Papenbrock, Thorsten; Naumann, Felix in Proceedings of the conference on Database Systems for Business, Technology, and Web (BTW) page 445-454 . 2015 .

Inclusion dependencies are among the most important database dependencies. In addition to their most prominent application – foreign key discovery – inclusion dependencies are an important input to data integration, query optimization, and schema redesign. With their discovery being a recurring data profiling task, previous research has proposed different algorithms to discover all inclusion dependencies within a given dataset. However, none of the proposed algorithms is designed to scale out, i.e., none can be distributed across multiple nodes in a computer cluster to increase the performance. So on large datasets with many inclusion dependencies, these algorithms can take days to complete, even on high-performance computers. We introduce SINDY, an algorithm that efficiently discovers all unary inclusion dependencies of a given relational dataset in a distributed fashion and that is not tied to main memory requirements. We give a practical implementation of SINDY that builds upon the map-reduce-style framework Stratosphere and conduct several experiments showing that SINDY can process huge datasets by several factors faster than its competitors while scaling with the number of cluster nodes.
Further Information
Tags discovery disctributed inclusion_dependencies isg parallel profiling