PDF-VQA: A New Dataset for Real-World VQA on PDF Documents

Yihao Ding; Siwen Luo; Hyunsuk Chung; Soyeon Caren Han

doi:10.1007/978-3-031-43427-3_35

PDF-VQA: A New Dataset for Real-World VQA on PDF Documents

Yihao Ding, Siwen Luo, Hyunsuk Chung, Soyeon Caren Han

Computer Science and Software Engineering

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

7 Citations (Scopus)

Abstract

Document-based Visual Question Answering examines the document understanding of document images in conditions of natural language questions. We proposed a new document-based VQA dataset, PDF-VQA, to comprehensively examine the document understanding from various aspects, including document element recognition, document layout structural understanding as well as contextual understanding and key information extraction. Our PDF-VQA dataset extends the current scale of document understanding that limits on the single document page to the new scale that asks questions over the full document of multiple pages. We also propose a new graph-based VQA model that explicitly integrates the spatial and hierarchically structural relationships between different document elements to boost the document structural understanding. The performances are compared with several baselines over different question types and tasks (The full dataset is released in https://github.com/adlnlp/pdfvqa).

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases
Subtitle of host publication	Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI
Editors	Gianmarco De Francisci Morales, Francesco Bonchi, Claudia Perlich, Natali Ruchansky, Nicolas Kourtellis, Elena Baralis
Place of Publication	Switzerland
Publisher	Springer Nature Switzerland AG
Pages	585-601
Number of pages	17
Edition	1
ISBN (Electronic)	9783031434273
ISBN (Print)	9783031434266
DOIs	https://doi.org/10.1007/978-3-031-43427-3_35
Publication status	Published - 18 Sept 2023
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Turin, Italy Duration: 18 Sept 2023 → 22 Sept 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14174 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Abbreviated title	ECML PKDD 2023
Country/Territory	Italy
City	Turin
Period	18/09/23 → 22/09/23

Access to Document

10.1007/978-3-031-43427-3_35

Cite this

Ding, Y., Luo, S., Chung, H., & Han, S. C. (2023). PDF-VQA: A New Dataset for Real-World VQA on PDF Documents. In G. De Francisci Morales, F. Bonchi, C. Perlich, N. Ruchansky, N. Kourtellis, & E. Baralis (Eds.), Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI (1 ed., pp. 585-601). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14174 LNAI). Springer Nature Switzerland AG. https://doi.org/10.1007/978-3-031-43427-3_35

Ding, Yihao ; Luo, Siwen ; Chung, Hyunsuk et al. / PDF-VQA : A New Dataset for Real-World VQA on PDF Documents. Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI. editor / Gianmarco De Francisci Morales ; Francesco Bonchi ; Claudia Perlich ; Natali Ruchansky ; Nicolas Kourtellis ; Elena Baralis. 1. ed. Switzerland : Springer Nature Switzerland AG, 2023. pp. 585-601 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a1abe094be76479697884b2d4d421acd,

title = "PDF-VQA: A New Dataset for Real-World VQA on PDF Documents",

abstract = "Document-based Visual Question Answering examines the document understanding of document images in conditions of natural language questions. We proposed a new document-based VQA dataset, PDF-VQA, to comprehensively examine the document understanding from various aspects, including document element recognition, document layout structural understanding as well as contextual understanding and key information extraction. Our PDF-VQA dataset extends the current scale of document understanding that limits on the single document page to the new scale that asks questions over the full document of multiple pages. We also propose a new graph-based VQA model that explicitly integrates the spatial and hierarchically structural relationships between different document elements to boost the document structural understanding. The performances are compared with several baselines over different question types and tasks (The full dataset is released in https://github.com/adlnlp/pdfvqa).",

keywords = "Document Information Extraction, Document Understanding, Visual Question Answering",

author = "Yihao Ding and Siwen Luo and Hyunsuk Chung and Han, {Soyeon Caren}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2023 ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2023",

month = sep,

day = "18",

doi = "10.1007/978-3-031-43427-3_35",

language = "English",

isbn = "9783031434266",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Nature Switzerland AG",

pages = "585--601",

editor = "{De Francisci Morales}, Gianmarco and Francesco Bonchi and Claudia Perlich and Natali Ruchansky and Nicolas Kourtellis and Elena Baralis",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

address = "Switzerland",

edition = "1",

}

Ding, Y, Luo, S, Chung, H & Han, SC 2023, PDF-VQA: A New Dataset for Real-World VQA on PDF Documents. in G De Francisci Morales, F Bonchi, C Perlich, N Ruchansky, N Kourtellis & E Baralis (eds), Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI. 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14174 LNAI, Springer Nature Switzerland AG, Switzerland, pp. 585-601, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Turin, Italy, 18/09/23. https://doi.org/10.1007/978-3-031-43427-3_35

PDF-VQA: A New Dataset for Real-World VQA on PDF Documents. / Ding, Yihao; Luo, Siwen; Chung, Hyunsuk et al.
Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI. ed. / Gianmarco De Francisci Morales; Francesco Bonchi; Claudia Perlich; Natali Ruchansky; Nicolas Kourtellis; Elena Baralis. 1. ed. Switzerland: Springer Nature Switzerland AG, 2023. p. 585-601 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14174 LNAI).

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

TY - GEN

T1 - PDF-VQA

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

AU - Ding, Yihao

AU - Luo, Siwen

AU - Chung, Hyunsuk

AU - Han, Soyeon Caren

PY - 2023/9/18

Y1 - 2023/9/18

N2 - Document-based Visual Question Answering examines the document understanding of document images in conditions of natural language questions. We proposed a new document-based VQA dataset, PDF-VQA, to comprehensively examine the document understanding from various aspects, including document element recognition, document layout structural understanding as well as contextual understanding and key information extraction. Our PDF-VQA dataset extends the current scale of document understanding that limits on the single document page to the new scale that asks questions over the full document of multiple pages. We also propose a new graph-based VQA model that explicitly integrates the spatial and hierarchically structural relationships between different document elements to boost the document structural understanding. The performances are compared with several baselines over different question types and tasks (The full dataset is released in https://github.com/adlnlp/pdfvqa).

AB - Document-based Visual Question Answering examines the document understanding of document images in conditions of natural language questions. We proposed a new document-based VQA dataset, PDF-VQA, to comprehensively examine the document understanding from various aspects, including document element recognition, document layout structural understanding as well as contextual understanding and key information extraction. Our PDF-VQA dataset extends the current scale of document understanding that limits on the single document page to the new scale that asks questions over the full document of multiple pages. We also propose a new graph-based VQA model that explicitly integrates the spatial and hierarchically structural relationships between different document elements to boost the document structural understanding. The performances are compared with several baselines over different question types and tasks (The full dataset is released in https://github.com/adlnlp/pdfvqa).

KW - Document Information Extraction

KW - Document Understanding

KW - Visual Question Answering

UR - http://www.scopus.com/inward/record.url?scp=85174443406&partnerID=8YFLogxK

UR - https://link.springer.com/book/10.1007/978-3-031-43427-3#bibliographic-information

UR - https://onesearch.library.uwa.edu.au/permalink/61UWA_INST/1vk1d8f/alma991487180402101

U2 - 10.1007/978-3-031-43427-3_35

DO - 10.1007/978-3-031-43427-3_35

M3 - Conference paper

AN - SCOPUS:85174443406

SN - 9783031434266

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 585

EP - 601

BT - Machine Learning and Knowledge Discovery in Databases

A2 - De Francisci Morales, Gianmarco

A2 - Bonchi, Francesco

A2 - Perlich, Claudia

A2 - Ruchansky, Natali

A2 - Kourtellis, Nicolas

A2 - Baralis, Elena

PB - Springer Nature Switzerland AG

CY - Switzerland

Y2 - 18 September 2023 through 22 September 2023

ER -

Ding Y, Luo S, Chung H, Han SC. PDF-VQA: A New Dataset for Real-World VQA on PDF Documents. In De Francisci Morales G, Bonchi F, Perlich C, Ruchansky N, Kourtellis N, Baralis E, editors, Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track - European Conference, ECML PKDD 2023, Proceedings Part VI. 1 ed. Switzerland: Springer Nature Switzerland AG. 2023. p. 585-601. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-43427-3_35