Word4Per: Zero-shot Composed Person Retrieval

25 Nov 2023  ·  Delong Liu, Haiwen Li, Zhicheng Zhao, Fei Su, Yuan Dong ·

Searching for specific person has great social benefits and security value, and it often involves a combination of visual and textual information. Conventional person retrieval methods, whether image-based or text-based, usually fall short in effectively harnessing both types of information, leading to the loss of accuracy. In this paper, a whole new task called Composed Person Retrieval (CPR) is proposed to jointly utilize both image and text information for target person retrieval. However, the supervised CPR requires very costly manual annotation dataset, while there are currently no available resources. To mitigate this issue, we firstly introduce the Zero-shot Composed Person Retrieval (ZS-CPR), which leverages existing domain-related data to resolve the CPR problem without expensive annotations. Secondly, to learn ZS-CPR model, we propose a two-stage learning framework, Word4Per, where a lightweight Textual Inversion Network (TINet) and a text-based person retrieval model based on fine-tuned Contrastive Language-Image Pre-training (CLIP) network are learned without utilizing any CPR data. Thirdly, a finely annotated Image-Text Composed Person Retrieval (ITCPR) dataset is built as the benchmark to assess the performance of the proposed Word4Per framework. Extensive experiments under both Rank-1 and mAP demonstrate the effectiveness of Word4Per for the ZS-CPR task, surpassing the comparative methods by over 10\%. The code and ITCPR dataset will be publicly available at https://github.com/Delong-liu-bupt/Word4Per.

PDF Abstract

Results from the Paper


 Ranked #1 on Zero-shot Composed Person Retrieval on ITCPR dataset (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-shot Composed Person Retrieval ITCPR dataset Word4Per (fuse) Rank-1 47.502 # 1
mAP 56.944 # 1
Zero-shot Composed Person Retrieval ITCPR dataset Word4Per Rank-1 45.549 # 2
mAP 55.260 # 2

Methods


No methods listed for this paper. Add relevant methods here