Learning Clip Representations for Skeleton-based 3D Action Recognition

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence, and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a Multi-task Convolutional Neural Network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and feature learning method for 3D action recognition compared to existing techniques.

Original languageEnglish
Pages (from-to)2842-2855
JournalIEEE Transactions on Image Processing
Volume27
Issue number6
DOIs
Publication statusPublished - Jun 2018

Fingerprint

Neural networks
Recurrent neural networks

Cite this

@article{510559a3affc430394f1c9e1fee4704f,
title = "Learning Clip Representations for Skeleton-based 3D Action Recognition",
abstract = "This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence, and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a Multi-task Convolutional Neural Network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and feature learning method for 3D action recognition compared to existing techniques.",
keywords = "3D Action Recognition, Clip Representation, CNN, Computational modeling, Feature extraction, Hidden Markov models, Multi-task Learning, Robustness, Skeleton, Task analysis, Three-dimensional displays",
author = "Qiuhong Ke and Mohammed Bennamoun and Senjian An and Ferdous Sohel and Farid Boussaid",
year = "2018",
month = "6",
doi = "10.1109/TIP.2018.2812099",
language = "English",
volume = "27",
pages = "2842--2855",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
number = "6",

}

Learning Clip Representations for Skeleton-based 3D Action Recognition. / Ke, Qiuhong; Bennamoun, Mohammed; An, Senjian; Sohel, Ferdous; Boussaid, Farid.

In: IEEE Transactions on Image Processing, Vol. 27, No. 6, 06.2018, p. 2842-2855.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Learning Clip Representations for Skeleton-based 3D Action Recognition

AU - Ke, Qiuhong

AU - Bennamoun, Mohammed

AU - An, Senjian

AU - Sohel, Ferdous

AU - Boussaid, Farid

PY - 2018/6

Y1 - 2018/6

N2 - This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence, and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a Multi-task Convolutional Neural Network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and feature learning method for 3D action recognition compared to existing techniques.

AB - This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence, and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a Multi-task Convolutional Neural Network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and feature learning method for 3D action recognition compared to existing techniques.

KW - 3D Action Recognition

KW - Clip Representation

KW - CNN

KW - Computational modeling

KW - Feature extraction

KW - Hidden Markov models

KW - Multi-task Learning

KW - Robustness

KW - Skeleton

KW - Task analysis

KW - Three-dimensional displays

UR - http://www.scopus.com/inward/record.url?scp=85042870944&partnerID=8YFLogxK

U2 - 10.1109/TIP.2018.2812099

DO - 10.1109/TIP.2018.2812099

M3 - Article

VL - 27

SP - 2842

EP - 2855

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 6

ER -