Self-supervised learning to detect key frames in videos

Xiang Yan, Syed Zulqarnain Gilani, Mingtao Feng, Liang Zhang, Hanlin Qin, Ajmal Mian

Research output: Contribution to journalArticle

Abstract

Detecting key frames in videos is a common problem in many applications such as video classification, action recognition and video summarization. These tasks can be performed more efficiently using only a handful of key frames rather than the full video. Existing key frame detection approaches are mostly designed for supervised learning and require manual labelling of key frames in a large corpus of training data to train the models. Labelling requires human annotators from different backgrounds to annotate key frames in videos which is not only expensive and time consuming but also prone to subjective errors and inconsistencies between the labelers. To overcome these problems, we propose an automatic self-supervised method for detecting key frames in a video. Our method comprises a two-stream ConvNet and a novel automatic annotation architecture able to reliably annotate key frames in a video for self-supervised learning of the ConvNet. The proposed ConvNet learns deep appearance and motion features to detect frames that are unique. The trained network is then able to detect key frames in test videos. Extensive experiments on UCF101 human action and video summarization VSUMM datasets demonstrates the effectiveness of our proposed method.

Original languageEnglish
Article number6941
Pages (from-to)1-18
Number of pages18
JournalSensors (Switzerland)
Volume20
Issue number23
DOIs
Publication statusPublished - 1 Dec 2020

Fingerprint Dive into the research topics of 'Self-supervised learning to detect key frames in videos'. Together they form a unique fingerprint.

Cite this