TY - JOUR
T1 - Skeleton-based action recognition through attention guided heterogeneous graph neural network
AU - Li, Tianchen
AU - Geng, Pei
AU - Lu, Xuequan
AU - Li, Wanqing
AU - Lyu, Lei
N1 - Publisher Copyright:
© 2024 Elsevier B.V.
PY - 2025/1/30
Y1 - 2025/1/30
N2 - Previous graph convolutional networks typically use homogeneous graphs to explore the hidden dependencies between joints in skeleton-based action recognition. Consequently, these networks treat both physical and nonphysical connections between joints as a single edge attribute and model them simultaneously, blurring the essential distinction between kinematic and interactive relationships. We propose an Attention Guided Heterogeneous Graph Neural Network (AG-HGNN), introducing a novel heterogeneous skeleton graph (HSG). In HSG, joints are represented as vertices; the physical and the nonphysical connections are represented as actual and virtual edges, respectively. Vertices and actual or virtual edges constitute actual and virtual metapaths, respectively. To comprehensively capture the relational patterns of each metapath, we design a rotary self-attention convolution module to understand the semantic associations between joints by introducing the rotary position embedding. Additionally, we propose an inter-frame adaptive temporal convolution module to adaptively adjust the weight of frames in the metapath, enabling spatial convolution to capture temporal dynamics. Among these metapaths, we develop a semantic aggregation module to determine the importance of each metapath in fusing the semantic associations revealed through metapaths. Experiments on three public datasets demonstrate that our proposed AG-HGNN achieves outstanding results.
AB - Previous graph convolutional networks typically use homogeneous graphs to explore the hidden dependencies between joints in skeleton-based action recognition. Consequently, these networks treat both physical and nonphysical connections between joints as a single edge attribute and model them simultaneously, blurring the essential distinction between kinematic and interactive relationships. We propose an Attention Guided Heterogeneous Graph Neural Network (AG-HGNN), introducing a novel heterogeneous skeleton graph (HSG). In HSG, joints are represented as vertices; the physical and the nonphysical connections are represented as actual and virtual edges, respectively. Vertices and actual or virtual edges constitute actual and virtual metapaths, respectively. To comprehensively capture the relational patterns of each metapath, we design a rotary self-attention convolution module to understand the semantic associations between joints by introducing the rotary position embedding. Additionally, we propose an inter-frame adaptive temporal convolution module to adaptively adjust the weight of frames in the metapath, enabling spatial convolution to capture temporal dynamics. Among these metapaths, we develop a semantic aggregation module to determine the importance of each metapath in fusing the semantic associations revealed through metapaths. Experiments on three public datasets demonstrate that our proposed AG-HGNN achieves outstanding results.
KW - Graph neural network
KW - Heterogeneous graph
KW - Self-attention mechanism
KW - Skeleton-based action recognition
UR - http://www.scopus.com/inward/record.url?scp=85212343988&partnerID=8YFLogxK
U2 - 10.1016/j.knosys.2024.112868
DO - 10.1016/j.knosys.2024.112868
M3 - Article
AN - SCOPUS:85212343988
SN - 0950-7051
VL - 309
JO - Knowledge-Based Systems
JF - Knowledge-Based Systems
M1 - 112868
ER -