Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

CAart: Cognitive Analysis of art resources and texts

Art archives are a rich source of information for multiple reasons: proving the provenance of certain art pieces, facilitating research on art history, and understanding a particular artist with regard to the context of his or her work. These archives typically comprise of various kinds of heterogeneous documents: auction catalogs, personal correspondence, books, exhibition catalogs, bills, certificates, studies, theses, etc. Many of these archives are not easily accessible as they are not yet digitized. Even the ones that are available in digitized form are hard to explore with general text mining tools.

In this project, we aim to facilitate access to a large collection of art related documents. To this end, we need to adapt standard NLP tools to cater to the unique challenges of the art domain. The ultimate goal is to generate a knowledge graph which can be easily explored by art historians. The knowledge graph would also serve as a backbone for semantic search functionality and for new ways to represent art entities, e.g. as embeddings in a high dimensional space. Modern deep learning methods will be developed to manage and visualize large collections of art historical and scholarly documents.

Subprojects

NER4Art

Identification of titles of artworks as named entities is a complex task due to the challenges of this domain. Existing NER tools are not able to perform well for this task due to lack of availability of domain specific training data. In this project, we develop techniques to generate in a semi-automatic manner a large corpus of good quality training data with annotations for artwork titles. Retraining of existing NER tools on this training dataset shows considerable improvement over baseline.

This work was presented at the TPDL 2019 conference held in Oslo, Norway.

Multimodal Analysis of Art-historic Archives

Cultural heritage data can facilitate search and browsing, help art historians to track the provenance of artworks and enable wider semantic text exploration for digital cultural resources. However, this information is contained in images of artworks as well as textual descriptions, or annotations accompanied with the images. During the digitization of such resources, the valuable associations between the images and texts are frequently lost. We want to retrieve the associations between images and texts for artworks from art-historic archives. Machine learning will be used to generate text descriptions for the extracted images on the one hand, and to detect descriptive phrases and titles of images from the text on the other hand. Finally, embeddings can align both, the descriptions and the images.

This ongoing project is in collaboration with Christian Bartz (PhD student at the Internet Technology and Systems chair). The project description paper titled "Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis" was published at the Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020), co-located with LREC 2020 conference.

Domain-Specific Knowledge Graph Construction for Semantic Analysis

Knowledge graphs are widely used for applications such as search, questions answering and recommendations. Large scale, general purpose knowledge graphs, having millions of facts, have been constructed through automated techniques from publicly available datasets such as Wikipedia. However, these knowledge graphs are typically incomplete and often fail to correctly capture the semantics of the data. This holds true particularly for domain-specific data, where the generic techniques for automated knowledge graph creation often fail due to novel challenges, such as lack of training data, semantic ambiguities and absence of representative ontologies.
This project focusses on automated knowledge graph construction for the cultural heritage domain. We investigate the research challenges encountered during the creation of an ontology and a knowledge graph from digitized collections of cultural heritage data based on machine learning approaches.

More details can be found in the ESWC 2020 PhD Symposium paper.

Associated Activities

Project-Related Publications

  • Jain, N.: Domain-Specific Knowledge Graph Construction for Semantic Analysis.Extended Semantic Web Conference (ESWC 2020) Ph.D. Symposium (2020).
     
  • Jain, N., Bartz, C., Krestel, R.: Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis.1st International Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020), co-located with LREC 2020 conference (2020).
     
  • Jain, N., Krestel, R.: Who is Mona L.? Identifying Mentions of Artworks in Historical Archives.International Conference on Theory and Practice of Digital Libraries (TPDL 2019). p. 115--122. Springer (2019).