Improving Visual Object Tracking through Visual Prompting

27 Sep 2024  Â·  Shih-Fang Chen, Jun-Cheng Chen, I-Hong Jhuo, Yen-Yu Lin ·

Learning a discriminative model to distinguish a target from its surrounding distractors is essential to generic visual object tracking. Dynamic target representation adaptation against distractors is challenging due to the limited discriminative capabilities of prevailing trackers. We present a new visual Prompting mechanism for generic Visual Object Tracking (PiVOT) to address this issue. PiVOT proposes a prompt generation network with the pre-trained foundation model CLIP to automatically generate and refine visual prompts, enabling the transfer of foundation model knowledge for tracking. While CLIP offers broad category-level knowledge, the tracker, trained on instance-specific data, excels at recognizing unique object instances. Thus, PiVOT first compiles a visual prompt highlighting potential target locations. To transfer the knowledge of CLIP to the tracker, PiVOT leverages CLIP to refine the visual prompt based on the similarities between candidate objects and the reference templates across potential targets. Once the visual prompt is refined, it can better highlight potential target locations, thereby reducing irrelevant prompt information. With the proposed prompting mechanism, the tracker can generate improved instance-aware feature maps through the guidance of the visual prompt, thus effectively reducing distractors. The proposed method does not involve CLIP during training, thereby keeping the same training complexity and preserving the generalization capability of the pretrained foundation model. Extensive experiments across multiple benchmarks indicate that PiVOT, using the proposed prompting method can suppress distracting objects and enhance the tracker.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Object Tracking AVisT PiVOT-L Success Rate 62.2 # 1
Visual Object Tracking LaSOT PiVOT-L AUC 73.4 # 6
Normalized Precision 84.7 # 2
Precision 82.1 # 2
Visual Object Tracking NeedForSpeed PiVOT-L AUC 0.682 # 2
Visual Object Tracking OTB-2015 PiVOT-L Precision 0.946 # 1
AUC 0.712 # 4

Methods