Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Francesco De Toni; Christopher Akiki; Javier de la Rosa; Clémentine Fourrier; Enrique Manjavacas; Stefan Schweter; Daniel van Strien

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Francesco De Toni
, Christopher Akiki
, Javier de la Rosa
, Clémentine Fourrier
, Enrique Manjavacas
, Stefan Schweter
, Daniel van Strien

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

Abstract

In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

Original language	English
Title of host publication	Proceedings of BigScience Episode #5
Subtitle of host publication	Workshop on Challenges & Perspectives in Creating Large Language Models
Editors	Angela Fan, Suzana Ilic, Thomas Wolf, Matthias Gallé
Place of Publication	USA
Publisher	Association for Computational Linguistics
Pages	75-83
Number of pages	9
ISBN (Electronic)	9781955917261
ISBN (Print)	9781955917261
Publication status	Published - May 2022
Event	BigScience Episode #5 – Challenges & Perspectives in Creating Large Language Models: ACL 2022 Workshop - Duration: 27 May 2022 → 27 May 2022

Workshop

Workshop	BigScience Episode #5 – Challenges & Perspectives in Creating Large Language Models
Period	27/05/22 → 27/05/22

Access to Document

Cite this

De Toni, F., Akiki, C., de la Rosa, J., Fourrier, C., Manjavacas, E., Schweter, S., & van Strien, D. (2022). Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. In A. Fan, S. Ilic, T. Wolf, & M. Gallé (Eds.), Proceedings of BigScience Episode #5: Workshop on Challenges & Perspectives in Creating Large Language Models (pp. 75-83). Association for Computational Linguistics. https://arxiv.org/abs/2204.05211

De Toni, Francesco ; Akiki, Christopher ; de la Rosa, Javier et al. / Entities, Dates, and Languages : Zero-Shot on Historical Texts with T0. Proceedings of BigScience Episode #5: Workshop on Challenges & Perspectives in Creating Large Language Models. editor / Angela Fan ; Suzana Ilic ; Thomas Wolf ; Matthias Gallé. USA : Association for Computational Linguistics, 2022. pp. 75-83

@inproceedings{62521b9bc1864143b5a7ad29604eb4ef,

title = "Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0",

abstract = "In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.",

keywords = "named entity recognition, zero-shot learning, large language models, digital humanities",

author = "\{De Toni\}, Francesco and Christopher Akiki and \{de la Rosa\}, Javier and Cl{\'e}mentine Fourrier and Enrique Manjavacas and Stefan Schweter and \{van Strien\}, Daniel",

year = "2022",

month = may,

language = "English",

isbn = "9781955917261",

pages = "75--83",

editor = "Angela Fan and Suzana Ilic and Thomas Wolf and Matthias Gall{\'e}",

booktitle = "Proceedings of BigScience Episode \#5",

publisher = "Association for Computational Linguistics",

note = "BigScience Episode \#5 – Challenges \& Perspectives in Creating Large Language Models : ACL 2022 Workshop ; Conference date: 27-05-2022 Through 27-05-2022",

}

De Toni, F, Akiki, C, de la Rosa, J, Fourrier, C, Manjavacas, E, Schweter, S & van Strien, D 2022, Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. in A Fan, S Ilic, T Wolf & M Gallé (eds), Proceedings of BigScience Episode #5: Workshop on Challenges & Perspectives in Creating Large Language Models. Association for Computational Linguistics, USA, pp. 75-83, BigScience Episode #5 – Challenges & Perspectives in Creating Large Language Models, 27/05/22. <https://arxiv.org/abs/2204.05211>

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. / De Toni, Francesco; Akiki, Christopher; de la Rosa, Javier et al.
Proceedings of BigScience Episode #5: Workshop on Challenges & Perspectives in Creating Large Language Models. ed. / Angela Fan; Suzana Ilic; Thomas Wolf; Matthias Gallé. USA: Association for Computational Linguistics, 2022. p. 75-83.

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

TY - GEN

T1 - Entities, Dates, and Languages

T2 - BigScience Episode #5 – Challenges & Perspectives in Creating Large Language Models

AU - De Toni, Francesco

AU - Akiki, Christopher

AU - de la Rosa, Javier

AU - Fourrier, Clémentine

AU - Manjavacas, Enrique

AU - Schweter, Stefan

AU - van Strien, Daniel

PY - 2022/5

Y1 - 2022/5

N2 - In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

AB - In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

KW - named entity recognition

KW - zero-shot learning

KW - large language models

KW - digital humanities

UR - https://aclanthology.org/2022.bigscience-1.0/

UR - https://bigscience.huggingface.co/acl-2022

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:000847249900007

UR - https://www.scopus.com/pages/publications/85137734213

M3 - Conference paper

SN - 9781955917261

SP - 75

EP - 83

BT - Proceedings of BigScience Episode #5

A2 - Fan, Angela

A2 - Ilic, Suzana

A2 - Wolf, Thomas

A2 - Gallé, Matthias

PB - Association for Computational Linguistics

CY - USA

Y2 - 27 May 2022 through 27 May 2022

ER -

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Abstract

Workshop

Access to Document

Other files and links

Fingerprint

Cite this