AdiosStMan: Parallelizing Casacore Table Data System using Adaptive IO System

Ruonan Wang, C. Harris, Andreas Wicenec

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

© 2016 Elsevier B.V.In this paper, we investigate the Casacore Table Data System (CTDS) used in the casacore and CASA libraries, and methods to parallelize it. CTDS provides a storage manager plugin mechanism for third-party developers to design and implement their own CTDS storage managers. Having this in mind, we looked into various storage backend techniques that can possibly enable parallel I/O for CTDS by implementing new storage managers. After carrying on benchmarks showing the excellent parallel I/O throughput of the Adaptive IO System (ADIOS), we implemented an ADIOS based parallel CTDS storage manager. We then applied the CASA MSTransform frequency split task to verify the ADIOS Storage Manager. We also ran a series of performance tests to examine the I/O throughput in a massively parallel scenario.
Original languageEnglish
Pages (from-to)146-154
Number of pages9
JournalAstronomy and Computing
Volume16
DOIs
Publication statusPublished - 2016

Fingerprint

data systems
Adaptive systems
Managers
Throughput
performance tests
photographic developers

Cite this

@article{8444e1a1b3034d008cbd6a6c135145bd,
title = "AdiosStMan: Parallelizing Casacore Table Data System using Adaptive IO System",
abstract = "{\circledC} 2016 Elsevier B.V.In this paper, we investigate the Casacore Table Data System (CTDS) used in the casacore and CASA libraries, and methods to parallelize it. CTDS provides a storage manager plugin mechanism for third-party developers to design and implement their own CTDS storage managers. Having this in mind, we looked into various storage backend techniques that can possibly enable parallel I/O for CTDS by implementing new storage managers. After carrying on benchmarks showing the excellent parallel I/O throughput of the Adaptive IO System (ADIOS), we implemented an ADIOS based parallel CTDS storage manager. We then applied the CASA MSTransform frequency split task to verify the ADIOS Storage Manager. We also ran a series of performance tests to examine the I/O throughput in a massively parallel scenario.",
author = "Ruonan Wang and C. Harris and Andreas Wicenec",
year = "2016",
doi = "10.1016/j.ascom.2016.05.003",
language = "English",
volume = "16",
pages = "146--154",
journal = "Astronomy and Computing",
issn = "2213-1337",
publisher = "Elsevier",

}

AdiosStMan: Parallelizing Casacore Table Data System using Adaptive IO System. / Wang, Ruonan; Harris, C.; Wicenec, Andreas.

In: Astronomy and Computing, Vol. 16, 2016, p. 146-154.

Research output: Contribution to journalArticle

TY - JOUR

T1 - AdiosStMan: Parallelizing Casacore Table Data System using Adaptive IO System

AU - Wang, Ruonan

AU - Harris, C.

AU - Wicenec, Andreas

PY - 2016

Y1 - 2016

N2 - © 2016 Elsevier B.V.In this paper, we investigate the Casacore Table Data System (CTDS) used in the casacore and CASA libraries, and methods to parallelize it. CTDS provides a storage manager plugin mechanism for third-party developers to design and implement their own CTDS storage managers. Having this in mind, we looked into various storage backend techniques that can possibly enable parallel I/O for CTDS by implementing new storage managers. After carrying on benchmarks showing the excellent parallel I/O throughput of the Adaptive IO System (ADIOS), we implemented an ADIOS based parallel CTDS storage manager. We then applied the CASA MSTransform frequency split task to verify the ADIOS Storage Manager. We also ran a series of performance tests to examine the I/O throughput in a massively parallel scenario.

AB - © 2016 Elsevier B.V.In this paper, we investigate the Casacore Table Data System (CTDS) used in the casacore and CASA libraries, and methods to parallelize it. CTDS provides a storage manager plugin mechanism for third-party developers to design and implement their own CTDS storage managers. Having this in mind, we looked into various storage backend techniques that can possibly enable parallel I/O for CTDS by implementing new storage managers. After carrying on benchmarks showing the excellent parallel I/O throughput of the Adaptive IO System (ADIOS), we implemented an ADIOS based parallel CTDS storage manager. We then applied the CASA MSTransform frequency split task to verify the ADIOS Storage Manager. We also ran a series of performance tests to examine the I/O throughput in a massively parallel scenario.

U2 - 10.1016/j.ascom.2016.05.003

DO - 10.1016/j.ascom.2016.05.003

M3 - Article

VL - 16

SP - 146

EP - 154

JO - Astronomy and Computing

JF - Astronomy and Computing

SN - 2213-1337

ER -