MSG-Chart: Multimodal Scene Graph for ChartQA

Research output: Chapter in Book/Conference paperConference paperpeer-review

Abstract

Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart. This graph module can be easily integrated with different vision transformers as inductive bias. Our experiments demonstrate that incorporating the proposed graph module enhances the understanding of charts' elements' structure and semantics, thereby improving performance on publicly available benchmarks, ChartQA and OpenCQA.

Original languageEnglish
Title of host publicationCIKM 2024 - Proceedings of the 33rd ACM International Conference on Information and Knowledge Management
PublisherAssociation for Computing Machinery (ACM)
Pages3709-3713
Number of pages5
ISBN (Electronic)9798400704369
DOIs
Publication statusPublished - 21 Oct 2024
Event33rd ACM International Conference on Information and Knowledge Management - Boise, United States
Duration: 21 Oct 202425 Oct 2024

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings
ISSN (Print)2155-0751

Conference

Conference33rd ACM International Conference on Information and Knowledge Management
Abbreviated titleCIKM 2024
Country/TerritoryUnited States
CityBoise
Period21/10/2425/10/24

Fingerprint

Dive into the research topics of 'MSG-Chart: Multimodal Scene Graph for ChartQA'. Together they form a unique fingerprint.

Cite this