Art archives are a rich source of information in several ways: proving the provenance of certain art pieces, facilitating research on art history, and understanding an artist in the context of their work. These archives typically comprise various kinds of heterogeneous documents: auction catalogs, personal correspondence, books, exhibition catalogs, bills, certificates, studies, theses, etc. Many of these archives are not easily accessible as they are not yet digitized. Even the ones that are available in digitized form are hard to explore with general text mining tools.

In this project, we aim to facilitate access to a large collection of art related documents. To this end, we need to adapt standard NLP tools to cater to the unique challenges of the art domain. The ultimate goal is to generate a knowledge graph which can be easily explored by art historians. The knowledge graph would also serve as a backbone for semantic search functionality and for new ways to represent art entities, e.g. as embeddings in a high dimensional space. Modern deep learning methods will be developed to manage and visualize large collections of art historical and scholarly documents.

This project is being carried out in close collaboration with the Wildenstein Plattner Institute who have been generous to share their interesting data with us for facilitating this research, while also providing regular feedback and suggestions.

Identification of titles of artworks as named entities is a complex task due to the challenges of this domain. Existing NER tools are not able to perform well for this task due to lack of availability of domain specific training data. In this project, we develop techniques to generate in a semi-automatic manner a large corpus of good quality training data with annotations for artwork titles. Retraining of existing NER tools on this training dataset shows considerable improvement over baseline.

This work was presented at the TPDL 2019 conference held in Oslo, Norway. An extended version with further experiments and analysis is available here.

A demo of the NER models that were trained on the generated corpus to detect artworks is publicly available at this link.

Cultural heritage data can facilitate search and browsing, help art historians to track the provenance of artworks and enable wider semantic text exploration for digital cultural resources. However, this information is contained in images of artworks as well as textual descriptions, or annotations accompanied with the images. During the digitization of such resources, the valuable associations between the images and texts are frequently lost. We want to retrieve the associations between images and texts for artworks from art-historic archives. Machine learning will be used to generate text descriptions for the extracted images on the one hand, and to detect descriptive phrases and titles of images from the text on the other hand. Finally, embeddings can align both, the descriptions and the images.

This ongoing project is in collaboration with Wildenstein Plattner Institute and Christian Bartz (PhD student at the Internet Technology and Systems chair) . The project description paper titled "Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis" was published at the Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020), co-located with LREC 2020 conference. The results of this project were published at the International Workshop on Fine Art Pattern Extraction and Recognition in conjunction with the 25th International Conference on Pattern Recognition (ICPR 2020) (2020).

Knowledge graphs are widely used for applications such as search, questions answering and recommendations. Large scale, general purpose knowledge graphs, having millions of facts, have been constructed through automated techniques from publicly available datasets such as Wikipedia. However, these knowledge graphs are typically incomplete and often fail to correctly capture the semantics of the data. This holds true particularly for domain-specific data, where the generic techniques for automated knowledge graph creation often fail due to novel challenges, such as lack of training data, semantic ambiguities and absence of representative ontologies.
This project focusses on automated knowledge graph construction for the cultural heritage domain. We investigate the research challenges encountered during the creation of an ontology and a knowledge graph from digitized collections of cultural heritage data based on machine learning approaches.

More details can be found in the ESWC 2020 PhD Symposium paper and the Text2KG@ESWC 2022 workshop paper.

• SoSe 2022 - Knowledge Graphs (Seminar)
• SoSe 2021 - Generating Art using GANs (Master Project)
• WiSe 2020 - Practical Introduction to Deep Learning for Computer Vision (Seminar)
• SoSe 2020 - Multimodal Analysis for Cultural Data (Master Project)
• WiSe 2019 - Museumserlebnisse mit Datenanalysen optimieren (Bachelor Project)
• SoSe 2019 - Text Visualisation in Practice (Project Seminar)

• Nitisha Jain, Alejandro Sierra-Múnera, Philipp Schmidt, Julius Streit, Simon Thormeyer, Maria Lomaeva, Ralf Krestel: Generating Domain-Specific Knowledge Graphs: Challenges with Open Information Extraction. Proceedings of the International Workshop on Knowledge Graph Generation from Text at ESWC, 2022 (to appear)
• Nitisha Jain, Christian Bartz, Tobias Bredow, Emanuel Metzenthin, Jona Otholt, Ralf Krestel: Semantic Analysis of Cultural Heritage Data: Aligning Paintings and Descriptions in Art-Historic Collections. Proceedings of the International Workshop on Fine Art Pattern Extraction and Recognition (FAPER@ICPR), 2020
  [Paper][Springer][DOI:10.1007/978-3-030-68796-0_37]
• Nitisha Jain: Multimodal Knowledge Graphs for Semantic Analysis of Cultural Heritage Data. Invited Talk at the Workshop on Knowledge Bases and Multiple Modalities (KBMM@AKBC), 2020
  [Paper][Slides]
• Nitisha Jain: Domain-Specific Knowledge Graph Construction for Semantic Analysis. Proceedings of the Extended Semantic Web Conference (ESWC), 2020
  [Paper][URL][DOI:10.1007/978-3-030-62327-2_40]
• Nitisha Jain, Christian Bartz, Ralf Krestel: Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis. Proceedings of the International Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI@LREC), 2020
  [Paper][URL]
• Nitisha Jain, Ralf Krestel: Who is Mona L.? Identifying Mentions of Artworks in Historical Archives. International Conference on Theory and Practice of Digital Libraries (TPDL), 2019
  [Paper][Springer][DOI:10.1007/978-3-030-30760-8_10]