Metadata Management System
Contact: Sebastian Kruse
What is this project about?
The Metadata Management System (MDMS) is an open source project that aims to unlock the potentials of data profiling results, i.e., metadata. Data profiling algorithms reveal latent properties of datasets, e.g., functional and inclusion dependencies, that are a prerequisite to many data management tasks, such as data integration, query optimization, and schema reverse engineering. However, the plain availability of oftentimes sheer amounts of metadata is in general not sufficient - instead the metadata require further processing. This is where the MDMS comes into play: It allows to store metadata in a structured manner and offers analyzing, interaction, and visualization capabilities, so as to detect relevant bits of the metadata and gain actual insights from the combination of different metadata types.
What are the tasks in this project?
The MDMS is evolving constantly. Not all of the components of the system are fully implemented yet. Also, we are seeking to employ modern technologies for the different components, from Apache Cassandra as data layer over Apache Flink and Spark as analytics layer to Apache Zeppelin and Jupyter as interaction and visualization tools. In this project, you will have to evaluate such technologies w.r.t. their suitability for the MDMS; (re-)implement parts of the MDMS; and demonstrate the capabilities of your solution with practical examples. Also, you are encouraged to contribute and discuss the overall design and concepts of the system.
What are the prerequisites to join this project?
MDMS has been implemented with Java and Scala and you should be somewhat proficient in one of these languages. We don't expect you to know all the different systems mentioned above, but you should be interested in learning and adopting such technologies.
What do I learn from the project?
There are clearly three things you can take away from the project: (1) You get to work on open source software. (2) You learn a lot about recent technologies in the database/information systems community. (3) You can familiarize yourselves with the topics of our chair, especially data profiling, which you might benefit from during your studies and your Master's theses.