Recall@k Surrogate Loss with Large Batches and Similarity Mixup

25 Aug 2021  ·  Yash Patel, Giorgos Tolias, Jiri Matas ·

This work focuses on learning deep visual representation models for retrieval by exploring the interplay between a new loss function, the batch size, and a new regularization approach. Direct optimization, by gradient descent, of an evaluation metric, is not possible when it is non-differentiable, which is the case for recall in retrieval. A differentiable surrogate loss for the recall is proposed in this work. Using an implementation that sidesteps the hardware constraints of the GPU memory, the method trains with a very large batch size, which is essential for metrics computed on the entire retrieval database. It is assisted by an efficient mixup regularization approach that operates on pairwise scalar similarities and virtually increases the batch size further. The suggested method achieves state-of-the-art performance in several image retrieval benchmarks when used for deep metric learning. For instance-level recognition, the method outperforms similar approaches that train using an approximation of average precision.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Metric Learning CARS196 Recall@k Surrogate loss (ResNet-50) R@1 88.3 # 10
Image Retrieval iNaturalist Recall@k Surrogate loss (ResNet-50) R@1 71.8 # 2
R@5 84.7 # 2
R@16 91.9 # 2
R@32 94.3 # 2
Metric Learning Stanford Online Products Recall@k Surrogate Loss (ResNet-50) R@1 82.7 # 5
Vehicle Re-Identification VehicleID Large Recall@k Surrogate loss (ResNet-50) Rank-1 93.8 # 1
Rank-5 96.6 # 1
Vehicle Re-Identification VehicleID Medium Recall@k Surrogate loss (ResNet-50) Rank-1 94.6 # 1
Rank-5 96.9 # 1
Vehicle Re-Identification VehicleID Small Recall@k Surrogate loss (ResNet-50) Rank-1 95.7 # 1
Rank-5 97.9 # 1