Relationship detection based on object semantic inference and attention mechanisms

Liang Zhang, Shuai Zhang, Peiyi Shen, Guangming Zhu, Syed Afaq Ali Shah, Mohammed Bennamoun

Research output: Chapter in Book/Conference paperConference paper

2 Citations (Scopus)

Abstract

Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as <man ride horse>, <horse, carry, bag>. To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

Original languageEnglish
Title of host publicationICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval
Place of PublicationUSA
PublisherAssociation for Computing Machinery (ACM)
Pages68-72
Number of pages5
ISBN (Electronic)9781450367653
DOIs
Publication statusPublished - 5 Jun 2019
Event2019 ACM International Conference on Multimedia Retrieval, ICMR 2019 - Ottawa, Canada
Duration: 10 Jun 201913 Jun 2019

Publication series

NameICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval

Conference

Conference2019 ACM International Conference on Multimedia Retrieval, ICMR 2019
CountryCanada
CityOttawa
Period10/06/1913/06/19

Fingerprint Dive into the research topics of 'Relationship detection based on object semantic inference and attention mechanisms'. Together they form a unique fingerprint.

Cite this