Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

ICCV 2021  ·  Yongming Rao, Guangyi Chen, Jiwen Lu, Jie zhou ·

Attention mechanism has demonstrated great potential in fine-grained visual recognition tasks. In this paper, we present a counterfactual attention learning method to learn more effective attention based on causal inference. Unlike most existing methods that learn visual attention based on conventional likelihood, we propose to learn the attention with counterfactual causality, which provides a tool to measure the attention quality and a powerful supervisory signal to guide the learning process. Specifically, we analyze the effect of the learned visual attention on network prediction through counterfactual intervention and maximize the effect to encourage the network to learn more useful attention for fine-grained image recognition. Empirically, we evaluate our method on a wide range of fine-grained recognition tasks where attention plays a crucial role, including fine-grained image categorization, person re-identification, and vehicle re-identification. The consistent improvement on all benchmarks demonstrates the effectiveness of our method. Code is available at

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Fine-Grained Image Classification CUB-200-2011 CAL Accuracy 90.6 # 8
Person Re-Identification DukeMTMC-reID CAL Rank-1 90 # 34
mAP 80.5 # 38
Fine-Grained Image Classification FGVC Aircraft CAL Accuracy 94.2 # 9
Person Re-Identification Market-1501 CAL Rank-1 95.5 # 42
mAP 89.5 # 45
Person Re-Identification MSMT17 CAL(ResNet50) Rank-1 84.2 # 20
mAP 64 # 20
Fine-Grained Image Classification Stanford Cars CAL Accuracy 95.5% # 10
Vehicle Re-Identification VehicleID Large CAL mAP 80.9 # 1
Rank-1 75.1 # 9
Vehicle Re-Identification VehicleID Medium CAL mAP 83.8 # 1
Rank-1 78.2 # 8
Vehicle Re-Identification VehicleID Small CAL mAP 87.8 # 4
Rank-1 82.5 # 10
Vehicle Re-Identification VeRi-776 CAL mAP 74.3 # 13
Rank-1 95.4 # 9
Rank5 97.9 # 5