Hidden Two-Stream Convolutional Networks for Action Recognition

2 Apr 2017  ·  Yi Zhu, Zhenzhong Lan, Shawn Newsam, Alexander G. Hauptmann ·

Analyzing videos of human actions involves understanding the temporal relationships among video frames. State-of-the-art action recognition approaches rely on traditional optical flow estimation methods to pre-compute motion information for CNNs... Such a two-stage approach is computationally expensive, storage demanding, and not end-to-end trainable. In this paper, we present a novel CNN architecture that implicitly captures motion information between adjacent frames. We name our approach hidden two-stream CNNs because it only takes raw video frames as input and directly predicts action classes without explicitly computing optical flow. Our end-to-end approach is 10x faster than its two-stage baseline. Experimental results on four challenging action recognition datasets: UCF101, HMDB51, THUMOS14 and ActivityNet v1.2 show that our approach significantly outperforms the previous best real-time approaches. read more

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Action Recognition HMDB-51 Hidden Two-Stream Average accuracy of 3 splits 78.7 # 17
Action Recognition UCF101 Hidden Two-Stream 3-fold Accuracy 97.1 # 15


No methods listed for this paper. Add relevant methods here