Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark

5 Jun 2023  ·  Shuyu Yang, Yinan Zhou, Yaxiong Wang, Yujiao Wu, Li Zhu, Zhedong Zheng ·

In this paper, we introduce a large Multi-Attribute and Language Search dataset for text-based person retrieval, called MALS, and explore the feasibility of performing pre-training on both attribute recognition and image-text matching tasks in one stone. In particular, MALS contains 1,510,330 image-text pairs, which is about 37.5 times larger than prevailing CUHK-PEDES, and all images are annotated with 27 attributes. Considering the privacy concerns and annotation costs, we leverage the off-the-shelf diffusion models to generate the dataset. To verify the feasibility of learning from the generated data, we develop a new joint Attribute Prompt Learning and Text Matching Learning (APTM) framework, considering the shared knowledge between attribute and text. As the name implies, APTM contains an attribute prompt learning stream and a text matching learning stream. (1) The attribute prompt learning leverages the attribute prompts for image-attribute alignment, which enhances the text matching learning. (2) The text matching learning facilitates the representation learning on fine-grained details, and in turn, boosts the attribute prompt learning. Extensive experiments validate the effectiveness of the pre-training on MALS, achieving state-of-the-art retrieval performance via APTM on three challenging real-world benchmarks. In particular, APTM achieves a consistent improvement of +6.96%, +7.68%, and +16.95% Recall@1 accuracy on CUHK-PEDES, ICFG-PEDES, and RSTPReid datasets by a clear margin, respectively.

PDF Abstract

Results from the Paper

 Ranked #1 on Text based Person Retrieval on RSTPReid (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Text based Person Retrieval CUHK-PEDES APTM R@1 76.53 # 1
R@10 94.15 # 2
R@5 90.04 # 2
mAP 66.91 # 3
Text based Person Retrieval ICFG-PEDES APTM mAP 41.22 # 2
R@1 68.51 # 1
Pedestrian Attribute Recognition PA-100K APTM Accuracy 80.17 # 5
Text based Person Retrieval RSTPReid APTM R@1 67.50 # 1
R@5 85.70 # 3
R@10 91.45 # 1