Temporal Relational Reasoning in Videos

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.

PDF Abstract ECCV 2018 PDF ECCV 2018 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Classification Charades MultiScale TRN MAP 25.2 # 36
Action Recognition In Videos Jester (Gesture Recognition) MultiScale TRN Val 95.31 # 4
Hand Gesture Recognition Jester test Multiscale TRN Top 1 Accuracy 94.78 # 2
Action Classification Moments in Time TRN-Multiscale Top 1 Accuracy 28.27 # 25
Top 5 Accuracy 53.87 # 16
Action Recognition In Videos Something-Something V1 2-Stream TRN Top 1 Accuracy 42.01 # 3
Action Recognition Something-Something V1 2-Stream TRN Top 1 Accuracy 42.01 # 64
Action Recognition Something-Something V1 M-TRN Top 1 Accuracy 34.4 # 67
Action Recognition In Videos Something-Something V2 2-Stream TRN Top-1 Accuracy 55.52 # 3
Top-5 Accuracy 83.06 # 3