Sketch Me That Shoe

We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketch-photo pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep triplet-ranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for fine-grained cross-domain ranking tasks.

PDF Abstract


Introduced in the Paper:


Used in the Paper:

ImageNet Sketch Chairs

Results from the Paper

Results from Other Papers

Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Sketch-Based Image Retrieval Chairs Chairs net + R@1 72.2 # 3
R@10 99.0 # 1
Sketch-Based Image Retrieval Chairs BoW-HOG + rankSVM R@1 28.9 # 8
R@10 67.0 # 8
Sketch-Based Image Retrieval Chairs Sketch-a-Net + rankSVM R@1 47.4 # 7
R@10 82.5 # 7
Sketch-Based Image Retrieval Chairs Dense-HOG + rankSVM R@1 52.6 # 6
R@10 93.8 # 4
Sketch-Based Image Retrieval Chairs Shoes net + R@1 65.0 # 4
R@10 92.8 # 5
Sketch-Based Image Retrieval Handbags BoW-HOG + rankSVM R@1 2.4 # 8
R@10 10.7 # 8
Sketch-Based Image Retrieval Handbags Sketch-a-Net + rankSVM R@1 9.5 # 7
R@10 44.1 # 6
Sketch-Based Image Retrieval Handbags Dense-HOG + rankSVM R@1 15.5 # 6
R@10 40.5 # 7
Sketch-Based Image Retrieval Handbags Shoes net + R@1 23.2 # 5
R@10 59.5 # 4
Sketch-Based Image Retrieval Handbags Chairs net + R@1 26.2 # 4
R@10 58.3 # 5


No methods listed for this paper. Add relevant methods here