Relationship detection based on object semantic inference and attention mechanisms

Liang Zhang, Shuai Zhang, Peiyi Shen, Guangming Zhu, Syed Afaq Ali Shah, Mohammed Bennamoun

Research output: Chapter in Book/Conference paperConference paper

Abstract

Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as <man ride horse>, <horse, carry, bag>. To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

Original languageEnglish
Title of host publicationICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval
Place of PublicationUSA
PublisherAssociation for Computing Machinery (ACM)
Pages68-72
Number of pages5
ISBN (Electronic)9781450367653
DOIs
Publication statusPublished - 5 Jun 2019
Event2019 ACM International Conference on Multimedia Retrieval, ICMR 2019 - Ottawa, Canada
Duration: 10 Jun 201913 Jun 2019

Publication series

NameICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval

Conference

Conference2019 ACM International Conference on Multimedia Retrieval, ICMR 2019
CountryCanada
CityOttawa
Period10/06/1913/06/19

Fingerprint

Semantics
Image understanding
Genes

Cite this

Zhang, L., Zhang, S., Shen, P., Zhu, G., Shah, S. A. A., & Bennamoun, M. (2019). Relationship detection based on object semantic inference and attention mechanisms. In ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval (pp. 68-72). (ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval). USA: Association for Computing Machinery (ACM). https://doi.org/10.1145/3323873.3325025
Zhang, Liang ; Zhang, Shuai ; Shen, Peiyi ; Zhu, Guangming ; Shah, Syed Afaq Ali ; Bennamoun, Mohammed. / Relationship detection based on object semantic inference and attention mechanisms. ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval. USA : Association for Computing Machinery (ACM), 2019. pp. 68-72 (ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval).
@inproceedings{bb6ce975c08d423dbd51fd2a261b3335,
title = "Relationship detection based on object semantic inference and attention mechanisms",
abstract = "Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as , . To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.",
keywords = "Attention mechanism, Feature refinement, Relationship detection, Semantic module",
author = "Liang Zhang and Shuai Zhang and Peiyi Shen and Guangming Zhu and Shah, {Syed Afaq Ali} and Mohammed Bennamoun",
year = "2019",
month = "6",
day = "5",
doi = "10.1145/3323873.3325025",
language = "English",
series = "ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval",
publisher = "Association for Computing Machinery (ACM)",
pages = "68--72",
booktitle = "ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval",
address = "United States",

}

Zhang, L, Zhang, S, Shen, P, Zhu, G, Shah, SAA & Bennamoun, M 2019, Relationship detection based on object semantic inference and attention mechanisms. in ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval. ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval, Association for Computing Machinery (ACM), USA, pp. 68-72, 2019 ACM International Conference on Multimedia Retrieval, ICMR 2019, Ottawa, Canada, 10/06/19. https://doi.org/10.1145/3323873.3325025

Relationship detection based on object semantic inference and attention mechanisms. / Zhang, Liang; Zhang, Shuai; Shen, Peiyi; Zhu, Guangming; Shah, Syed Afaq Ali; Bennamoun, Mohammed.

ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval. USA : Association for Computing Machinery (ACM), 2019. p. 68-72 (ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval).

Research output: Chapter in Book/Conference paperConference paper

TY - GEN

T1 - Relationship detection based on object semantic inference and attention mechanisms

AU - Zhang, Liang

AU - Zhang, Shuai

AU - Shen, Peiyi

AU - Zhu, Guangming

AU - Shah, Syed Afaq Ali

AU - Bennamoun, Mohammed

PY - 2019/6/5

Y1 - 2019/6/5

N2 - Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as , . To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

AB - Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as , . To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

KW - Attention mechanism

KW - Feature refinement

KW - Relationship detection

KW - Semantic module

UR - http://www.scopus.com/inward/record.url?scp=85068075115&partnerID=8YFLogxK

U2 - 10.1145/3323873.3325025

DO - 10.1145/3323873.3325025

M3 - Conference paper

T3 - ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval

SP - 68

EP - 72

BT - ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval

PB - Association for Computing Machinery (ACM)

CY - USA

ER -

Zhang L, Zhang S, Shen P, Zhu G, Shah SAA, Bennamoun M. Relationship detection based on object semantic inference and attention mechanisms. In ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval. USA: Association for Computing Machinery (ACM). 2019. p. 68-72. (ICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval). https://doi.org/10.1145/3323873.3325025