Numerous data are stored in semi-structured files with ad-hoc layout. Such data are valuable digital assets for various data-driven applications. This work introduces the notion of verbose CSV files. Verbose CSV files include content serving different purposes in various positions. They are designed for human visual inspection or statistical report collection. An important preliminary task for extracting information from such files is structure detection, in particular classifying lines or cells by their purpose. As manual efforts are infeasible and error-prone for large files or large sets of files, automatic approaches are desirable. This work addresses both the line and the cell classification problems on verbose CSV files. We use the following figure to demonstrate a typical verbose CSV file and the six different type of cell and line classes.

This work "Structure Detection in Verbose CSV Files" has been published at EDBT'21.

To address the line/cell classication problem, we propose the Structure Detection in Verbose CSV Files (Strudel) approach, which is grounded on a multi-class random forest classifier. The following fiure shows the architecture of the approach. It first detects the dialect of a text file, and creates a verbose CSV file from it, based on the dialect. Then Strudel classifies first lines and then cells therein with the proposed feature sets. Cells of different types are distinguished by colors. We propose sophisticated features to model the individual classes for both classification tasks. The features can be categorized into three groups: 1) content features parsing the values of cells or lines, such as cell length and amount of words; 2) contextual features comparing the inspected cell or line with its neighbors, such as the similarity of data types between lines/cells; 3) computational features seeking to connect lines/cells with each other by inspecting arithmetic correlations between them.