Understanding Dynamic Scenes using Graph Convolution Networks

We present a novel Multi-Relational Graph Convolutional Network (MRGCN) based framework to model on-road vehicle behaviors from a sequence of temporally ordered frames as grabbed by a moving monocular camera. The input to MRGCN is a multi-relational graph where the graph's nodes represent the active and passive agents/objects in the scene, and the bidirectional edges that connect every pair of nodes are encodings of their Spatio-temporal relations. We show that this proposed explicit encoding and usage of an intermediate spatio-temporal interaction graph to be well suited for our tasks over learning end-end directly on a set of temporally ordered spatial relations. We also propose an attention mechanism for MRGCNs that conditioned on the scene dynamically scores the importance of information from different interaction types. The proposed framework achieves significant performance gain over prior methods on vehicle-behavior classification tasks on four datasets. We also show a seamless transfer of learning to multiple datasets without resorting to fine-tuning. Such behavior prediction methods find immediate relevance in a variety of navigation tasks such as behavior planning, state estimation, and applications relating to the detection of traffic violations over videos.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Test results ApolloScape MRGCN LSTM Accuracy 72 # 3
Motion Segmentation ApolloScape Rel-Att-GCN Accuracy 89 # 2
Motion Segmentation ApolloScape MRGCN Accuracy 86 # 3
Motion Segmentation ApolloScape Rule Based Accuracy 90 # 1
Motion Segmentation ApolloScape MRGCN-LSTM Accuracy 72 # 4
Motion Segmentation ApolloScape St-RNN Accuracy 63 # 5
Test results ApolloScape Rel-Att GCN Accuracy 89 # 1
Test results ApolloScape MRGCN Accuracy 86 # 2
Test results Honda MRGCN Accuracy 81 # 2
Test results Honda Rel-Att GCN Accuracy 85 # 1
Test results Honda MRGCN LSTM Accuracy 60 # 3
Test results ImageCLEF-DA Rel-Att GCN Accuracy 97 # 1
Test results Indian MRGCN LSTM Accuracy 84 # 3
Test results Indian MRGCN Accuracy 93 # 2
Test results Indian Rel-Att GCN Accuracy 99 # 1
Test results KITTI MRGCN LSTM Accuracy 89 # 3
Test results KITTI MRGCN Accuracy 95 # 2
Test results KITTI Rel-Att GCN Accuracy 99 # 1


No methods listed for this paper. Add relevant methods here