In this paper, we focus on heterogeneous features learning for RGB-D activity recognition. We find that features from different channels (RGB, depth) could share some similar hidden structures, and then propose a joint learning model to simultaneously explore the shared and feature-specific components as an instance of heterogeneous multi-task learning. The proposed model formed in a unified framework is capable of: 1) jointly mining a set of subspaces with the same dimensionality to exploit latent shared features across different feature channels, 2) meanwhile, quantifying the shared and feature-specific components of features in the subspaces, and 3) transferring feature-specific intermediate transforms (i-transforms) for learning fusion of heterogeneous features across datasets. To efficiently train the joint model, a three-step iterative optimization algorithm is proposed, followed by a simple inference model. Extensive experimental results on four activity datasets have demonstrated the efficacy of the proposed method. Anew RGB-D activity dataset focusing on human-object interaction is further contributed, which presents more challenges for RGB-D activity benchmarking.

PDF Abstract IEEE Transactions 2016 PDF IEEE Transactions 2016 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Skeleton Based Action Recognition NTU RGB+D Dynamic Skeletons Accuracy (CV) 65.2 # 112
Accuracy (CS) 60.2 # 112
Skeleton Based Action Recognition NTU RGB+D 120 Dynamic Skeletons Accuracy (Cross-Subject) 50.8% # 66
Accuracy (Cross-Setup) 54.7% # 65
Skeleton Based Action Recognition SYSU 3D Dynamic Skeletons Accuracy 75.5% # 8


No methods listed for this paper. Add relevant methods here