MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, Naveed Akhtar

Research output: Chapter in Book/Conference paperConference paperpeer-review

5 Citations (Scopus)

Abstract

Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.
Original languageEnglish
Title of host publication2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022
Place of PublicationUSA
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISBN (Electronic)9781665475921
DOIs
Publication statusPublished - 2022
Event2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022 - Suzhou, China
Duration: 13 Dec 202216 Dec 2022

Publication series

Name2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

Conference

Conference2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022
Country/TerritoryChina
CitySuzhou
Period13/12/2216/12/22

Fingerprint

Dive into the research topics of 'MAiVAR: Multimodal Audio-Image and Video Action Recognizer'. Together they form a unique fingerprint.

Cite this