Jiang, Lan, Gerardo Vitagliano, and Felix Naumann. “Structure Detection in Verbose CSV Files”. In International Conference on Extending Database Technology (EDBT), 193–204, 2021. https://edbt2021proceedings.github.io/docs/p32.pdf.
Numerous data are stored in semi-structured files with ad-hoc layout. Such data are valuable digital assets for various data-driven applications. This work introduces the notion of verbose CSV files. Verbose CSV files include content serving different purposes in various positions. They are designed for human visual inspection or statistical report collection. An important preliminary task for extracting information from such files is structure detection, in particular classifying lines or cells by their purpose. As manual efforts are infeasible and error-prone for large files or large sets of files, automatic approaches are desirable. This work addresses both the line and the cell classification problems on verbose CSV files. Strudel is a supervised learning approach based on a random forest classifier combined with a set of novel features that fall into three categories: content features, contextual features, and computational features. We annotated five real-world datasets from various domains, on which we tested our approach. Our in-depth experiments show the advantages of Strudel over baseline and state-of-the-art approaches in both line and cell classification tasks.
Hameed, Mazhar, and Felix Naumann. “Data Preparation: A Survey of Commercial Tools”. SIGMOD Record 49, no. 3 (2020).
Raw data are often messy: they follow different encodings, records are not well structured, values do not adhere to patterns, etc. Such data are in general not fit to be ingested by downstream applications, such as data analytics tools, or even by data management systems. The act of obtaining information from raw data relies on some data preparation process. Data preparation is integral to advanced data analysis and data management, not only for data science but for any data-driven applications. Existing data preparation tools are operational and useful, but there is still room for improvement and optimization. With increasing data volume and its messy nature, the demand for prepared data increases day by day. To cater to this demand, companies and researchers are developing techniques and tools for data preparation. To better understand the available data preparation systems, we have conducted a survey to investigate (1) prominent data preparation tools, (2) distinctive tool features, (3) the need for preliminary data processing even for these tools and, (4) features and abilities that are still lacking. We conclude with an argument in support of automatic and intelligent data preparation beyond traditional and simplistic techniques.
Jiang, Lan, Gerardo Vitagliano, and Felix Naumann. “A Scoring-Based Approach for Data Preparator Suggestion”. In Lernen, Wissen, Daten, Analysen (LWDA), 2454:6–9, 2019.
Self-service data preparation enables end users to prepare data by themselves. However, the plethora of possible data preparation steps can overwhelm the user. We introduce a score-based preparator ranking approach to propose preparator candidates in a context-specific manner. To this end, we give scoring functions for a selected set of preparators and outline future work towards a full-fledged data preparation system.
