Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning

N. Sharif, M. A. A. K. Jalwana, M. Bennamoun, W. Liu, S. A. A. Shah

Research output: Chapter in Book/Conference paperConference paperpeer-review

1 Citation (Scopus)

Abstract

Image captioning is a challenging vision-to-language task, which has garnered a lot of attention over the past decade. The introduction of Encoder-Decoder based architectures expedited the research in this area and provided the backbone of the most recent systems. Moreover, leveraging relationships between objects for holistic scene understanding, which in turn improves captioning, has recently sparked interest among researchers. Our proposed model encodes the spatial and semantic proximity of object pairs into linguistically-aware relationship embeddings. Moreover, it captures the global semantics of the image using NASNet. This way, true semantic relations that are not apparent in visual content of an image can be learned, such that the decoder can attend to the most relevant object relations and visual features to generate more semantically-meaningful captions. Our experiments highlight the usefulness of linguistically-aware object relations as well as NASNet visual features for image captioning.
Original languageEnglish
Title of host publication2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ)
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages1-6
Number of pages6
DOIs
Publication statusPublished - 2020
Event35th International Conference on Image and Vision Computing New Zealand, IVCNZ - Wellington, New Zealand
Duration: 25 Nov 202027 Nov 2020
Conference number: 35

Conference

Conference35th International Conference on Image and Vision Computing New Zealand, IVCNZ
Abbreviated titleIVCNZ 2020
Country/TerritoryNew Zealand
CityWellington
Period25/11/2027/11/20

Fingerprint

Dive into the research topics of 'Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning'. Together they form a unique fingerprint.

Cite this