In data-driven workflows, practicioners typically resort to a plethora of different systems:

•  Wrangling tools, such as Trifacta, can be used for the early preparation and cleaning stages;
•  Spreadsheet tools, such as Microsoft Excel, are often used to carry out data exploration and analysis;
•  Programming frameworks, such as Pandas, Tensorflow, PyTorch, allow users to perform statistical and machine-learning operations;
•  Business intelligence software, such as Tableau, to create interactive dashboards and data visualizations;
•  Database management systems, such as MySQL,  are traditionally used to efficiently organize and store data;

For its flexibility, ease of use and parsing, the de-facto standard file format for the data flow between heterogeneous systems is CSV.
Albeit the structure of CSV files is defined in the RFC4180 document, often times files have a different structure, for example due to different locale specifications (e.g. comma used for floating point delimiters in Germany).
Moreover, data files may contain metadata information such as preambles or footnotes that should not be considered as part of the tabular content of CSV files.

The results of our survey highlight how often real-world files do not follow the standard specification of the RFC4180 document with respect to different structural features.

The key question that arises from these results is: how do different tools handle loading files with such non-standard features?

For each loading step we calculate three measures:

• Success: a binary measure if the system was able to load the file in memory.
• Completeness: a score in [0,1] if all input cells are present in the output.
• Coinciseness: a score [0,1] if no spurious cells are introduced in the data loading process.

The final benchmark score is obtained summing up the three different score: as Pollock is composed of a source file with 125 pollutions, the maximum score obtainable by a system is 375.

We experimented applying Pollock to benchmark four systems: a data science programming framework (DS), a business intelligence tool (BI), a spreadsheet software (SS), and a database management software (DB).
The results can be seen in the table below.

Pollock is an open source project. Its code is available on the project page at GitHub.

1. G. Vitagliano, L. Jiang, M. Hameed, F. Naumann: Pollock: A Data Loading Benchmark. In preparation

This research project is conducted at the Data Preparation group. If you have any question, interesting idea, or request, do not hesitate to contact Gerardo Vitagliano.