MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Muhammad Bilal Shaikh; Douglas Chai; Syed Mohammed Shamsul Islam; Naveed Akhtar

doi:10.1109/VCIP56404.2022.10008833

MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, Naveed Akhtar

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

6 Citations (Scopus)

Abstract

Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

Original language	English
Title of host publication	2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022
Place of Publication	USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
ISBN (Electronic)	9781665475921
DOIs	https://doi.org/10.1109/VCIP56404.2022.10008833
Publication status	Published - 2022
Event	2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022 - Suzhou, China Duration: 13 Dec 2022 → 16 Dec 2022

Publication series

Name	2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

Conference

Conference	2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022
Country/Territory	China
City	Suzhou
Period	13/12/22 → 16/12/22

Access to Document

10.1109/VCIP56404.2022.10008833

Cite this

Shaikh, M. B., Chai, D., Islam, S. M. S., & Akhtar, N. (2022). MAiVAR: Multimodal Audio-Image and Video Action Recognizer. In 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022 (2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/VCIP56404.2022.10008833

Shaikh, Muhammad Bilal ; Chai, Douglas ; Islam, Syed Mohammed Shamsul et al. / MAiVAR : Multimodal Audio-Image and Video Action Recognizer. 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022. USA : IEEE, Institute of Electrical and Electronics Engineers, 2022. (2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022).

@inproceedings{320adc19a6864612ac402216742198bc,

title = "MAiVAR: Multimodal Audio-Image and Video Action Recognizer",

abstract = "Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.",

author = "Shaikh, {Muhammad Bilal} and Douglas Chai and Islam, {Syed Mohammed Shamsul} and Naveed Akhtar",

note = "Funding Information: This work is jointly supported by Edith Cowan University (ECU) and Higher Education Commission (HEC) of Pakistan under Project #PM/HRDI-UESTPs/UETs-I/Phase-1/Batch-VI/2018. Dr. Akhtar is a recipient of Office of National Intelligence National Intelligence Postdoctoral Grant # NIPG-2021-001 funded by the Australian Government. Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022 ; Conference date: 13-12-2022 Through 16-12-2022",

year = "2022",

doi = "10.1109/VCIP56404.2022.10008833",

language = "English",

series = "2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

booktitle = "2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022",

address = "United States",

}

Shaikh, MB, Chai, D, Islam, SMS & Akhtar, N 2022, MAiVAR: Multimodal Audio-Image and Video Action Recognizer. in 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022. 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022, IEEE, Institute of Electrical and Electronics Engineers, USA, 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022, Suzhou, China, 13/12/22. https://doi.org/10.1109/VCIP56404.2022.10008833

MAiVAR: Multimodal Audio-Image and Video Action Recognizer. / Shaikh, Muhammad Bilal; Chai, Douglas; Islam, Syed Mohammed Shamsul et al.
2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022. USA: IEEE, Institute of Electrical and Electronics Engineers, 2022. (2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022).

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

TY - GEN

T1 - MAiVAR

T2 - 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

AU - Shaikh, Muhammad Bilal

AU - Chai, Douglas

AU - Islam, Syed Mohammed Shamsul

AU - Akhtar, Naveed

N1 - Funding Information: This work is jointly supported by Edith Cowan University (ECU) and Higher Education Commission (HEC) of Pakistan under Project #PM/HRDI-UESTPs/UETs-I/Phase-1/Batch-VI/2018. Dr. Akhtar is a recipient of Office of National Intelligence National Intelligence Postdoctoral Grant # NIPG-2021-001 funded by the Australian Government. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

AB - Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

UR - http://www.scopus.com/inward/record.url?scp=85147247031&partnerID=8YFLogxK

U2 - 10.1109/VCIP56404.2022.10008833

DO - 10.1109/VCIP56404.2022.10008833

M3 - Conference paper

AN - SCOPUS:85147247031

T3 - 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

BT - 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - USA

Y2 - 13 December 2022 through 16 December 2022

ER -

Shaikh MB, Chai D, Islam SMS, Akhtar N. MAiVAR: Multimodal Audio-Image and Video Action Recognizer. In 2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022. USA: IEEE, Institute of Electrical and Electronics Engineers. 2022. (2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022). doi: 10.1109/VCIP56404.2022.10008833

MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Spectral Centroid Images for Multi-class Human Action Analysis : A Benchmark Dataset

MFFCs for Multi-class Human Action Analysis : A Benchmark Dataset

Waveplot-based Dataset for Multi-class Human Action Analysis

Cite this

MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Datasets

Spectral Centroid Images for Multi-class Human Action Analysis : A Benchmark Dataset

MFFCs for Multi-class Human Action Analysis : A Benchmark Dataset

Waveplot-based Dataset for Multi-class Human Action Analysis

Cite this