The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset

Hugo Laurençon; Lucile Saulnier; Thomas Wang; Christopher Akiki; Albert Villanova del Moral; Teven Le Scao; Leandro von Werra; Chenghao Mou; Eduardo González Ponferrada; Huu Nguyen; Jörg Frohberg; Mario Šaško; Quentin Lhoest; Angelina Mcmillan-Major; Gérard Dupont; Stella Biderman; Anna Rogers; Loubna Ben Allal; Francesco de Toni; Giada Pistilli; Olivier Nguyen; Somaieh Nikpoor; Maraim Masoud; Pierre Colombo; Javier de la Rosa; Paulo Villegas; Tristan Thrush; Shayne Longpre; Sebastian Nagel; Leon Weber; Manuel Romero Muñoz; Jian Zhu; Daniel van Strien; Zaid Alyafeai; Khalid Almubarak; Vu Minh Chien; Itziar Gonzalez-Dios; Aitor Soroa; Kyle Lo; Manan Dey; Pedro Ortiz Suarez; Aaron Gokaslan; Shamik Bose; David Ifeoluwa Adelani; Long Phan; Hieu Tran; Ian Yu; Suhas Pai; Jenny Chim; Violette Lepercq; Suzana Ilić; Margaret Mitchell; Sasha Luccioni; Yacine Jernite

The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina Mcmillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco de Toni, Giada PistilliOlivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilić, Margaret Mitchell, Sasha Luccioni, Yacine Jernite

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

Abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 35 (NeurIPS 2022)
Place of Publication	New Orleans, United States
Publisher	Neural Information Processing Systems Foundation, Inc.
ISBN (Electronic)	9781713871088
Publication status	Published - 2022
Event	36th Conference on Neural Information Processing Systems - New Orleans Convention Center & Virtual, New Orleans, United States Duration: 28 Nov 2022 → 8 Dec 2022

Conference

Conference	36th Conference on Neural Information Processing Systems
Abbreviated title	NeurIPS 2022
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 8/12/22

Access to Document

Cite this

Laurençon, H., Saulnier, L., Wang, T., Akiki, C., Villanova del Moral, A., Le Scao, T., von Werra, L., Mou, C., González Ponferrada, E., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., Mcmillan-Major, A., Dupont, G., Biderman, S., Rogers, A., Ben Allal, L., de Toni, F., ... Jernite, Y. (2022). The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Neural Information Processing Systems Foundation, Inc.. https://openreview.net/forum?id=UoEw6KigkUn

@inproceedings{8770bc531e3c4c70a28f88b6e7c017ad,

title = "The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset",

abstract = "As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus. ",

author = "Hugo Lauren{\c c}on and Lucile Saulnier and Thomas Wang and Christopher Akiki and {Villanova del Moral}, Albert and {Le Scao}, Teven and {von Werra}, Leandro and Chenghao Mou and {Gonz{\'a}lez Ponferrada}, Eduardo and Huu Nguyen and J{\"o}rg Frohberg and Mario {\v S}a{\v s}ko and Quentin Lhoest and Angelina Mcmillan-Major and G{\'e}rard Dupont and Stella Biderman and Anna Rogers and {Ben Allal}, Loubna and {de Toni}, Francesco and Giada Pistilli and Olivier Nguyen and Somaieh Nikpoor and Maraim Masoud and Pierre Colombo and {de la Rosa}, Javier and Paulo Villegas and Tristan Thrush and Shayne Longpre and Sebastian Nagel and Leon Weber and {Romero Mu{\~n}oz}, Manuel and Jian Zhu and {van Strien}, Daniel and Zaid Alyafeai and Khalid Almubarak and Chien, {Vu Minh} and Itziar Gonzalez-Dios and Aitor Soroa and Kyle Lo and Manan Dey and {Ortiz Suarez}, Pedro and Aaron Gokaslan and Shamik Bose and Adelani, {David Ifeoluwa} and Long Phan and Hieu Tran and Ian Yu and Suhas Pai and Jenny Chim and Violette Lepercq and Suzana Ili{\'c} and Margaret Mitchell and Sasha Luccioni and Yacine Jernite",

year = "2022",

language = "English",

booktitle = "Advances in Neural Information Processing Systems 35 (NeurIPS 2022)",

publisher = "Neural Information Processing Systems Foundation, Inc.",

address = "Cocos (Keeling) Islands",

note = "36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 08-12-2022",

}

Laurençon, H, Saulnier, L, Wang, T, Akiki, C, Villanova del Moral, A, Le Scao, T, von Werra, L, Mou, C, González Ponferrada, E, Nguyen, H, Frohberg, J, Šaško, M, Lhoest, Q, Mcmillan-Major, A, Dupont, G, Biderman, S, Rogers, A, Ben Allal, L, de Toni, F, Pistilli, G, Nguyen, O, Nikpoor, S, Masoud, M, Colombo, P, de la Rosa, J, Villegas, P, Thrush, T, Longpre, S, Nagel, S, Weber, L, Romero Muñoz, M, Zhu, J, van Strien, D, Alyafeai, Z, Almubarak, K, Chien, VM, Gonzalez-Dios, I, Soroa, A, Lo, K, Dey, M, Ortiz Suarez, P, Gokaslan, A, Bose, S, Adelani, DI, Phan, L, Tran, H, Yu, I, Pai, S, Chim, J, Lepercq, V, Ilić, S, Mitchell, M, Luccioni, S & Jernite, Y 2022, The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset. in Advances in Neural Information Processing Systems 35 (NeurIPS 2022). Neural Information Processing Systems Foundation, Inc., New Orleans, United States, 36th Conference on Neural Information Processing Systems, New Orleans, Louisiana, United States, 28/11/22. <https://openreview.net/forum?id=UoEw6KigkUn>

The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset. / Laurençon, Hugo; Saulnier, Lucile; Wang, Thomas et al.
Advances in Neural Information Processing Systems 35 (NeurIPS 2022). New Orleans, United States: Neural Information Processing Systems Foundation, Inc., 2022.

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

TY - GEN

T1 - The BigScience ROOTS Corpus

T2 - 36th Conference on Neural Information Processing Systems

AU - Laurençon, Hugo

AU - Saulnier, Lucile

AU - Wang, Thomas

AU - Akiki, Christopher

AU - Villanova del Moral, Albert

AU - Le Scao, Teven

AU - von Werra, Leandro

AU - Mou, Chenghao

AU - González Ponferrada, Eduardo

AU - Nguyen, Huu

AU - Frohberg, Jörg

AU - Šaško, Mario

AU - Lhoest, Quentin

AU - Mcmillan-Major, Angelina

AU - Dupont, Gérard

AU - Biderman, Stella

AU - Rogers, Anna

AU - Ben Allal, Loubna

AU - de Toni, Francesco

AU - Pistilli, Giada

AU - Nguyen, Olivier

AU - Nikpoor, Somaieh

AU - Masoud, Maraim

AU - Colombo, Pierre

AU - de la Rosa, Javier

AU - Villegas, Paulo

AU - Thrush, Tristan

AU - Longpre, Shayne

AU - Nagel, Sebastian

AU - Weber, Leon

AU - Romero Muñoz, Manuel

AU - Zhu, Jian

AU - van Strien, Daniel

AU - Alyafeai, Zaid

AU - Almubarak, Khalid

AU - Chien, Vu Minh

AU - Gonzalez-Dios, Itziar

AU - Soroa, Aitor

AU - Lo, Kyle

AU - Dey, Manan

AU - Ortiz Suarez, Pedro

AU - Gokaslan, Aaron

AU - Bose, Shamik

AU - Adelani, David Ifeoluwa

AU - Phan, Long

AU - Tran, Hieu

AU - Yu, Ian

AU - Pai, Suhas

AU - Chim, Jenny

AU - Lepercq, Violette

AU - Ilić, Suzana

AU - Mitchell, Margaret

AU - Luccioni, Sasha

AU - Jernite, Yacine

PY - 2022

Y1 - 2022

N2 - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

AB - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

UR - https://neurips.cc/Conferences/2022/CallForPapers

UR - https://proceedings.neurips.cc/paper_files/paper/2022

M3 - Conference paper

BT - Advances in Neural Information Processing Systems 35 (NeurIPS 2022)

PB - Neural Information Processing Systems Foundation, Inc.

CY - New Orleans, United States

Y2 - 28 November 2022 through 8 December 2022

ER -

The BigScience ROOTS Corpus: A 1.6TB composite multilingual dataset

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this