Improving Object Detection in Art Images Using Only Style Transfer

12 Feb 2021  ·  David Kadish, Sebastian Risi, Anders Sundnes Løvlie ·

Despite recent advances in object detection using deep learning neural networks, these neural networks still struggle to identify objects in art images such as paintings and drawings. This challenge is known as the cross depiction problem and it stems in part from the tendency of neural networks to prioritize identification of an object's texture over its shape. In this paper we propose and evaluate a process for training neural networks to localize objects - specifically people - in art images. We generate a large dataset for training and validation by modifying the images in the COCO dataset using AdaIn style transfer. This dataset is used to fine-tune a Faster R-CNN object detection network, which is then tested on the existing People-Art testing dataset. The result is a significant improvement on the state of the art and a new way forward for creating datasets to train neural networks to process art images.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Object Detection PeopleArt FasterRCNN (trained on StyleCOCO) mAP 36 # 5
mAP@0.5 68 # 5
mAP@0.75 33 # 5