Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
 

10.11.2020

Paper accepted at FAPER workshop in conjunction with ICPR 2020

Our full paper titled "Semantic Analysis of Cultural Heritage Data: Aligning Paintings and Descriptions in Art-Historic Collections" has been accepted to the International Workshop on Fine Art Pattern Extraction and Recognition (FAPER), co-located with ICPR 2020 conference. This paper describes our work from the SoSe 2020 Master Project in collaboration with Christian Bartz and Masters students Tobias Bredow, Emanuel Metzenthin and Jona Otholt.

Semantic Analysis of Cultural Heritage Data: Aligning Paintings and Descriptions in Art-Historic Collections

Art-historic documents often contain multimodal data in terms of images of artworks and metadata, descriptions, or interpretations thereof. Most research efforts have focused either on image analysis or text analysis independently since the associations between the two modes are usually lost during digitization. In this work, we focus on the task of alignment of images and textual descriptions in art-historic digital collections. To this end, we reproduce an existing approach that learns alignments in a semi-supervised fashion. We identify several challenges while automatically aligning images and texts, specifically for the cultural heritage domain, which limit the scalability of previous works. To improve the performance of alignment, we introduce various enhancements to extend the existing approach that show promising results.