Skeleton-based action recognition through attention guided heterogeneous graph neural network

Tianchen Li, Pei Geng, Xuequan Lu, Wanqing Li, Lei Lyu

Research output: Contribution to journalArticlepeer-review

Abstract

Previous graph convolutional networks typically use homogeneous graphs to explore the hidden dependencies between joints in skeleton-based action recognition. Consequently, these networks treat both physical and nonphysical connections between joints as a single edge attribute and model them simultaneously, blurring the essential distinction between kinematic and interactive relationships. We propose an Attention Guided Heterogeneous Graph Neural Network (AG-HGNN), introducing a novel heterogeneous skeleton graph (HSG). In HSG, joints are represented as vertices; the physical and the nonphysical connections are represented as actual and virtual edges, respectively. Vertices and actual or virtual edges constitute actual and virtual metapaths, respectively. To comprehensively capture the relational patterns of each metapath, we design a rotary self-attention convolution module to understand the semantic associations between joints by introducing the rotary position embedding. Additionally, we propose an inter-frame adaptive temporal convolution module to adaptively adjust the weight of frames in the metapath, enabling spatial convolution to capture temporal dynamics. Among these metapaths, we develop a semantic aggregation module to determine the importance of each metapath in fusing the semantic associations revealed through metapaths. Experiments on three public datasets demonstrate that our proposed AG-HGNN achieves outstanding results.

Original languageEnglish
Article number112868
Number of pages12
JournalKnowledge-Based Systems
Volume309
DOIs
Publication statusPublished - 30 Jan 2025

Fingerprint

Dive into the research topics of 'Skeleton-based action recognition through attention guided heterogeneous graph neural network'. Together they form a unique fingerprint.

Cite this