Anticipative Video Transformer

ICCV 2021  ·  Rohit Girdhar, Kristen Grauman ·

We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies--both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR'21 challenge.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Anticipation EPIC-KITCHENS-100 AVT+ Recall@5 15.9 # 2
Action Anticipation EPIC-KITCHENS-100 (test) AVT++ recall@5 16.7 # 1
Action Anticipation EPIC-KITCHENS-100 (test) AVT+ recall@5 12.6 # 2
Action Anticipation EPIC-KITCHENS-55 (Seen test set (S1)) AVT+ Top 1 Accuracy - Verb 34.36 # 2
Top 1 Accuracy - Noun 20.16 # 3
Top 1 Accuracy - Act. 16.84 # 1
Top 5 Accuracy - Verb 80.03 # 1
Top 5 Accuracy - Noun 51.57 # 2
Top 5 Accuracy - Act. 36.52 # 1
Action Anticipation EPIC-KITCHENS-55 (Unseen test set (S2) AVT+ Top 1 Accuracy - Verb 30.66 # 1
Top 1 Accuracy - Noun 15.64 # 1
Top 1 Accuracy - Act. 10.41 # 1
Top 5 Accuracy - Verb 72.17 # 1
Top 5 Accuracy - Noun 40.76 # 1
Top 5 Accuracy - Act. 24.27 # 1

Methods