Deploying a machine learning pipeline is a resource-demanding task that requires a combination of data and software engineering expertise. However, even with meticulous testing, the risk of encountering run-time errors during pipeline operation remains a significant concern. In this presentation, our focus lies in addressing the run-time errors caused by mismatches between the data to be processed and the code responsible for its processing. We present strategies for the early detection of these mismatches through static and dynamic analysis, leveraging techniques from JSON Schema reasoning. Specifically, we showcase our recent contributions to essential tasks such as schema validation, schema extraction, and checking schema containment. Furthermore, we provide an outlook on the challenges introduced by the latest drafts of JSON Schema. Lastly, we conclude with a discussion on application domains for our contributions, extending beyond the fortification of machine learning pipelines against run-time errors.

By connecting two operators and batching the pipeline together, it's possible to determine the compatibility of schemas for inputs and outputs. In LALE it is used between connected operators to make sure the first operators output is always a valid input for the second operator as we know both operators interfaces.

Not elimination

The algorithm with "Not elimination" [2]: transforming JSON Schemas so that they no longer contain the "not" keyword, which is used for negation. This is significant because "not" can make schemas difficult to understand and reason about. For example, Professor Scherzinger thought that the `not` seem like optional - if its `not required`.

The most common use of 'not' in JSON Schema is to indicate `not required`, an older method of specifying that certain properties are not allowed. Regarding boolean operators, the research analyzed the total occurrences and the number of files in which various JSON Schema keywords appear, essentially assessing how often operators in the JSON Schema language are used. While 'not' is not frequently used, it is nonetheless very important. Moreover, there are operators that represent forms of negation, such as 'oneOf', which is a union of choices (like A|B|C, meaning A and not B and C). Although 'not' is not used frequently, different forms of negation, like 'oneOf', are commonly employed.

An Empirical Study on the "Usage of Not" in Real-World JSON Schema Documents [2]

Eliminating these negations can be done by pushing the not down in an algebraic form of the schema Professor Scherzinger and her chair designed.

Witness generation

The LALE containment checker supports no recursion and barely no negation so Professor Scherzinger and others developed a solution [3] that understands recursion and negation by reducing containment checks to witness generation:

A witness for a schema S that is a subset of schema S’ should always be a witness for S’. Now the algorithm tries to generate a witness for S∧¬S’ and if there is a witness found S is not contained in S’ if the algorithm can’t find one it is contained. The new containment checker for classic JSON Schema is the first complete algorithm for this task as before there was only the complexity of EXPTIME complete known.

The algorithm consists of three parts: the not elimination, rewriting the algebra in DNF and a bottom-up witness generation. The not elimination is necessary for the bottom-up generation because otherwise witnesses are generated in multiple iterations and in the last step the algorithm has to restart because there is a not in a higher hierarchical order and a witness for not: X cannot be generated from a witness for X.

They have developed a new, straightforward algorithm specifically for generating JSON Schema witnesses. They thoroughly tested this algorithm to assess how well it works and how fast it is. These tests were conducted using a variety of schema collections, including thousands of schemas from real-world scenarios. Their analysis focuses on how completely the algorithm can handle the JSON Schema language (with the exception of the "uniqueItems" operator) and whether it can perform efficiently in a practical amount of time on many real-world examples (succeeded 99 % of the GitHubs JSON Schemas).

Witness Generation for JSON Schema [3]

Schema Extraction

If we receive data without an accompanying schema, we have the ability to extract a schema directly from the data itself. Then, we can perform containment-checking by comparing these extracted schemas. Current advanced methods for schema extraction focus mostly on understanding the structure of JSON data [4], like how it's nested and whether it contains arrays or objects. Professor Scherzinger and her team, however, are working on identifying 'tagged unions' within JSON Schema. Tagged unions are a design pattern in JSON Schema where a specific property within an object (referred to as the tag) suggests different subschemas for the other properties in the object based on its value. The team aims to formalize these relationships as conditional functional dependencies and represent them using the JSON Schema operators 'if-then-else'. [5, 6]

Extracting JSON Schemas with tagged unions [6]

Conclusion

JSON Schema is widely used in various fields, such as NoSQL document stores for validating inputs, Web APIs for ensuring correct data formats and interactions, and in the creation of forms based on JSON data. This broad application spectrum highlights the versatility and importance of JSON Schema in managing and validating data across different technological domains.

Prof. Dr. Stefanie Scherzinger's research focuses on advanced JSON Schema applications, including witness generation algorithms and containment checks. She explores schema extraction, particularly identifying tagged unions in JSON Schema. Her work also involves addressing machine learning pipeline issues through early error detection using JSON Schema reasoning. Additionally, she is contributing to the development of JTutor, a JSON Schema validator, and has made significant advancements in debugging and accuracy in schema validation.

Resources

[1] Lecture "Detecting Data-Code Mismatches in Machine Learning Pipelines" (tele-task.de)

[2] An Empirical Study on the “Usage of Not” in Real-World JSON Schema Documents (link.springer.com)

[3] Witness Generation for JSON Schema (arxiv.org)

[4] Schema Extraction and Structural Outlier Detection for JSON-based NoSQL Data Stores (researchgate.net)

[5] Tagger: A Tool for the Discovery of Tagged Unions in JSON Schema Extraction (uni-regensburg.de)

[6] Extracting JSON Schemas with Tagged Unions (arxiv.org)