Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction

    Research output: Contribution to journalArticle

    6 Citations (Scopus)

    Abstract

    Predicting an interaction before it is fully executed is very important in applications such as human-robot interaction and video surveillance. In a two-human interaction scenario, there often contextual dependency structure between the global interaction context of the two humans and the local context of the different body parts of each human. In this paper, we propose to learn the structure of the interaction contexts, and combine it with the spatial and temporal information of a video sequence for a better prediction of the interaction class. The structural models, including the spatial and the temporal models, are learned with Long Short Term Memory (LSTM) networks to capture the dependency of the global and local contexts of each RGB frame and each optical flow image, respectively. LSTM networks are also capable of detecting the key information from the global and local interaction contexts. Moreover, to effectively combine the structural models with the spatial and temporal models for interaction prediction, a ranking score fusion method is also introduced to automatically compute the optimal weight of each model for score fusion. Experimental results on the BIT-Interaction and the UT-Interaction datasets clearly demonstrate the benefits of the proposed method.

    Original languageEnglish
    JournalIEEE Transactions on Multimedia
    DOIs
    Publication statusE-pub ahead of print - 29 Nov 2017

    Fingerprint

    Fusion reactions
    Human robot interaction
    Optical flows
    Long short-term memory

    Cite this

    @article{47cd47b308524c22b6c8d200282abf03,
    title = "Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction",
    abstract = "Predicting an interaction before it is fully executed is very important in applications such as human-robot interaction and video surveillance. In a two-human interaction scenario, there often contextual dependency structure between the global interaction context of the two humans and the local context of the different body parts of each human. In this paper, we propose to learn the structure of the interaction contexts, and combine it with the spatial and temporal information of a video sequence for a better prediction of the interaction class. The structural models, including the spatial and the temporal models, are learned with Long Short Term Memory (LSTM) networks to capture the dependency of the global and local contexts of each RGB frame and each optical flow image, respectively. LSTM networks are also capable of detecting the key information from the global and local interaction contexts. Moreover, to effectively combine the structural models with the spatial and temporal models for interaction prediction, a ranking score fusion method is also introduced to automatically compute the optimal weight of each model for score fusion. Experimental results on the BIT-Interaction and the UT-Interaction datasets clearly demonstrate the benefits of the proposed method.",
    keywords = "Australia, Computational modeling, Context modeling, Hidden Markov models, Interaction Prediction, Interaction Structure, LSTM, Matrix converters, Predictive models, Ranking Score Fusion, Streaming media",
    author = "Qiuhong Ke and Mohammed Bennamoun and Senjian An and Ferdous Sohel and Farid Boussaid",
    year = "2017",
    month = "11",
    day = "29",
    doi = "10.1109/TMM.2017.2778559",
    language = "English",
    journal = "IEEE Transactions on Multimedia",
    issn = "1520-9210",
    publisher = "IEEE, Institute of Electrical and Electronics Engineers",

    }

    TY - JOUR

    T1 - Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction

    AU - Ke, Qiuhong

    AU - Bennamoun, Mohammed

    AU - An, Senjian

    AU - Sohel, Ferdous

    AU - Boussaid, Farid

    PY - 2017/11/29

    Y1 - 2017/11/29

    N2 - Predicting an interaction before it is fully executed is very important in applications such as human-robot interaction and video surveillance. In a two-human interaction scenario, there often contextual dependency structure between the global interaction context of the two humans and the local context of the different body parts of each human. In this paper, we propose to learn the structure of the interaction contexts, and combine it with the spatial and temporal information of a video sequence for a better prediction of the interaction class. The structural models, including the spatial and the temporal models, are learned with Long Short Term Memory (LSTM) networks to capture the dependency of the global and local contexts of each RGB frame and each optical flow image, respectively. LSTM networks are also capable of detecting the key information from the global and local interaction contexts. Moreover, to effectively combine the structural models with the spatial and temporal models for interaction prediction, a ranking score fusion method is also introduced to automatically compute the optimal weight of each model for score fusion. Experimental results on the BIT-Interaction and the UT-Interaction datasets clearly demonstrate the benefits of the proposed method.

    AB - Predicting an interaction before it is fully executed is very important in applications such as human-robot interaction and video surveillance. In a two-human interaction scenario, there often contextual dependency structure between the global interaction context of the two humans and the local context of the different body parts of each human. In this paper, we propose to learn the structure of the interaction contexts, and combine it with the spatial and temporal information of a video sequence for a better prediction of the interaction class. The structural models, including the spatial and the temporal models, are learned with Long Short Term Memory (LSTM) networks to capture the dependency of the global and local contexts of each RGB frame and each optical flow image, respectively. LSTM networks are also capable of detecting the key information from the global and local interaction contexts. Moreover, to effectively combine the structural models with the spatial and temporal models for interaction prediction, a ranking score fusion method is also introduced to automatically compute the optimal weight of each model for score fusion. Experimental results on the BIT-Interaction and the UT-Interaction datasets clearly demonstrate the benefits of the proposed method.

    KW - Australia

    KW - Computational modeling

    KW - Context modeling

    KW - Hidden Markov models

    KW - Interaction Prediction

    KW - Interaction Structure

    KW - LSTM

    KW - Matrix converters

    KW - Predictive models

    KW - Ranking Score Fusion

    KW - Streaming media

    UR - http://www.scopus.com/inward/record.url?scp=85036530091&partnerID=8YFLogxK

    U2 - 10.1109/TMM.2017.2778559

    DO - 10.1109/TMM.2017.2778559

    M3 - Article

    JO - IEEE Transactions on Multimedia

    JF - IEEE Transactions on Multimedia

    SN - 1520-9210

    ER -