Vision Transformer Based Model for Describing a Set of Images as a Story

Research output: Chapter in Book/Conference paperConference paperpeer-review

5 Citations (Scopus)
33 Downloads (Pure)

Abstract

Visual Story-Telling is the process of forming a multi sentence story from a set of images. Appropriately including visual variation and contextual information captured inside the input images is one of the most challenging aspects of visual storytelling. Consequently, stories developed from a set of images often lack cohesiveness, relevance, and semantic relationship. In this paper, we propose a novel Vision Transformer Based Model for describing a set of images as a story. The proposed method extracts the distinct features of the input images using a Vision Transformer (ViT). Firstly, input images are divided into 16 × 16 patches and bundled into a linear projection of flattened patches. The transformation from a single image to multiple image patches captures the visual variety of the input visual patterns. These features are used as input to a Bidirectional-LSTM which is part of the sequence encoder. This captures the past and future image context of all image patches. Then, an attention mechanism is implemented and used to increase the discriminatory capacity of the data fed into the language model, i.e. a Mogrifier-LSTM. The performance of our proposed model is evaluated using the Visual Story-Telling dataset (VIST), and the results show that our model outperforms the current state of the art models.
Original languageEnglish
Title of host publicationAdvances in Artificial Intelligence - 35th Australasian Joint Conference on AI 2022
Subtitle of host publicationAdvances in Artificial Intelligence - 35th Australasian Joint Conference, AI 2022, Proceedings
EditorsHaris Aziz, Débora Corrêa, Tim French
Place of PublicationSwitzerland
PublisherSpringer Science + Business Media
Pages15-28
Number of pages14
Volume13728
ISBN (Electronic)978-3-031-22695-3
ISBN (Print)9783031226946
DOIs
Publication statusPublished - 2022
Event35th Australasian Joint Conference on Artificial Intelligence, AI 2022 - Perth, Australia
Duration: 5 Dec 20229 Dec 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13728 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference35th Australasian Joint Conference on Artificial Intelligence, AI 2022
Country/TerritoryAustralia
CityPerth
Period5/12/229/12/22

Fingerprint

Dive into the research topics of 'Vision Transformer Based Model for Describing a Set of Images as a Story'. Together they form a unique fingerprint.

Cite this