Optimal Transport for Long-Tailed Recognition with Learnable Cost Matrix

It is attracting attention to the long-tailed recognition problem, a burning issue that has become very popular recently. Distinctive from conventional recognition is that it posits that the allocation of the training set is supremely distorted, with a balanced validation and test set. Predictably, there are severe challenges posed to the generalisation behaviour of the model, due to the distribution shift between the training and test sets. Approaches to the challenge revolve into two groups: firstly, training-aware methods, with the aim of enhancing the generalisability of the model by exploiting its potential in the training period; and secondly, post-hoc correction, liberally coupled with training-aware methods, which is intended to refine the predictions to the extent possible in the post-processing stage, offering the advantages of simplicity and effectiveness. In this paper, we introduce an alternative direction to do the post-hoc correction, which goes beyond the statistical methods. Mathematically, we approach this issue from the perspective of optimal transport (OT), yet, choosing the exact cost matrix when applying OT is challenging and requires expert knowledge of various tasks. To overcome this limitation, we propose to employ linear mapping to adaptively learn the cost matrix without necessary configurations. Testing our methods in practice, our experiments reveals that, along with high efficiency and excellent performance, our method surpasses all previous methods and has the best performance to date.