TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

29 Oct 2023  ·  Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu sun, Lu Hou ·

Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

PDF Abstract

Results from the Paper

 Ranked #1 on Video Retrieval on Condensed Movies (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Video Retrieval ActivityNet TESTA (ViT-B/16) text-to-video R@1 54.8 # 11
text-to-video R@5 80.8 # 9
text-to-video R@10 89.6 # 9
Video Question Answering ActivityNet-QA TESTA (ViT-B/16) Accuracy 45 # 17
Video Retrieval Condensed Movies TESTA (ViT-B/16) text-to-video R@1 24.9 # 1
text-to-video R@5 46.5 # 1
text-to-video R@10 55.1 # 1
Video Retrieval DiDeMo TESTA (ViT-B/16) text-to-video R@1 61.2 # 7
text-to-video R@5 87.2 # 4
text-to-video R@10 91.5 # 3
Video Retrieval QuerYD TESTA (ViT-B/16) text-to-video R@1 83.4 # 1
text-to-video R@10 95.3 # 1
text-to-video R@5 93.8 # 1


No methods listed for this paper. Add relevant methods here