PLAR: Prompt Learning for Action Recognition

21 May 2023  ·  Xijun Wang, Ruiqi Xian, Tianrui Guan, Dinesh Manocha ·

We present a new general learning approach, Prompt Learning for Action Recognition (PLAR), which leverages the strengths of prompt learning to guide the learning process. Our approach is designed to predict the action label by helping the models focus on the descriptions or instructions associated with actions in the input videos. Our formulation uses various prompts, including learnable prompts, auxiliary visual information, and large vision models to improve the recognition performance. In particular, we design a learnable prompt method that learns to dynamically generate prompts from a pool of prompt experts under different inputs. By sharing the same objective with the task, our proposed PLAR can optimize prompts that guide the model's predictions while explicitly learning input-invariant (prompt experts pool) and input-specific (data-dependent) prompt knowledge. We evaluate our approach on datasets consisting of both ground camera videos and aerial videos, and scenes with single-agent and multi-agent actions. In practice, we observe a 3.17-10.2% accuracy improvement on the aerial multi-agent dataset Okutamam and a 1.0-3.6% improvement on the ground camera single-agent dataset Something Something V2. We plan to release our code on the WWW.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Action Recognition Okutama-Action PLAR with bbox (Ours) Accuracy 75.93 # 1
Action Recognition Okutama-Action PLAR without bbox (Ours) Accuracy 71.54 # 2
Action Recognition Something-Something V2 PLAR Top-1 Accuracy 67.3 # 60
Top-5 Accuracy 91 # 45


No methods listed for this paper. Add relevant methods here