Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
  
 

20.03.2020

Workshop paper at LREC 2020

Our short paper titled "Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis" has been accepted to the Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020), co-located with LREC 2020 conference. This paper describes our ongoing project in collaboration with Christian Bartz, PhD student at the Internet Technology and Systems chair. The preprint of the paper can be viewed here.

Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis

Authors - Nitisha Jain, Christian Bartz, Ralf Krestel

Abstract - Cultural heritage data plays a pivotal role in the understanding of human history and culture. A wealth of information is buried in art-historic archives which can be extracted via their digitization and analysis. This information can facilitate search and browsing, help art historians to track the provenance of artworks and enable wider semantic text exploration for digital cultural resources. However, this information is contained in images of artworks as well as textual descriptions, or annotations accompanied with the images. During the digitization of such resources, the valuable associations between the images and texts are frequently lost. In this project description, we propose an approach to retrieve the associations between images and texts for artworks from art-historic archives. To this end, we use machine learning to generate text descriptions for the extracted images on the one hand, and to detect descriptive phrases and titles of images from the text on the other hand. Finally, we use embeddings to align both, the descriptions and the images.