Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

Kevin Vinsen; Mark Boulton; Ian Cooper; Richard Dodson; Markus Dolensky; Dave Pallot; Rodrigo Tobar; Andreas Wicenec; Chen Wu

Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

Kevin Vinsen, Mark Boulton, Ian Cooper, Richard Dodson, Markus Dolensky, Dave Pallot, Rodrigo Tobar, Andreas Wicenec, Chen Wu

International Centre for Radio Astronomy Research (ICRAR)

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

Abstract

Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.

The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.

Original language	English
Title of host publication	Astronomical Data Analysis Software and Systems XXVI
Editors	Marco Molinaro, Keith Shortridge, Fabio Pasian
Publisher	Astronomical Society of the Pacific
Pages	628-631
ISBN (Print)	978-1-58381-929-6
Publication status	Published - 1 Oct 2019
Event	26th Annual Conference for Astronomical Data Analysis Software and Systems - Italian National Institute of Astrophysics, Trieste, Italy Duration: 16 Oct 2016 → 20 Oct 2016 Conference number: 26

Publication series

Name	ASP Conference Series
Publisher	Astronomical Society of the Pacific
Volume	521
ISSN (Print)	1050-3390

Conference

Conference	26th Annual Conference for Astronomical Data Analysis Software and Systems
Abbreviated title	ADASS XXVI
Country/Territory	Italy
City	Trieste
Period	16/10/16 → 20/10/16

Cite this

Vinsen, K., Boulton, M., Cooper, I., Dodson, R., Dolensky, M., Pallot, D., Tobar, R., Wicenec, A., & Wu, C. (2019). Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE. In M. Molinaro, K. Shortridge, & F. Pasian (Eds.), Astronomical Data Analysis Software and Systems XXVI (pp. 628-631). (ASP Conference Series; Vol. 521). Astronomical Society of the Pacific.

@inproceedings{ad0036779c93442896c4056000e464a8,

title = "Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE",

abstract = "Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.",

author = "Kevin Vinsen and Mark Boulton and Ian Cooper and Richard Dodson and Markus Dolensky and Dave Pallot and Rodrigo Tobar and Andreas Wicenec and Chen Wu",

year = "2019",

month = oct,

day = "1",

language = "English",

isbn = "978-1-58381-929-6",

series = "ASP Conference Series",

publisher = "Astronomical Society of the Pacific",

pages = "628--631",

editor = "Marco Molinaro and Keith Shortridge and Fabio Pasian",

booktitle = "Astronomical Data Analysis Software and Systems XXVI",

note = "26th Annual Conference for Astronomical Data Analysis Software and Systems, ADASS XXVI ; Conference date: 16-10-2016 Through 20-10-2016",

}

Vinsen, K, Boulton, M, Cooper, I, Dodson, R, Dolensky, M, Pallot, D, Tobar, R, Wicenec, A & Wu, C 2019, Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE. in M Molinaro, K Shortridge & F Pasian (eds), Astronomical Data Analysis Software and Systems XXVI. ASP Conference Series, vol. 521, Astronomical Society of the Pacific, pp. 628-631, 26th Annual Conference for Astronomical Data Analysis Software and Systems, Trieste, Italy, 16/10/16.

Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE. / Vinsen, Kevin; Boulton, Mark; Cooper, Ian et al.
Astronomical Data Analysis Software and Systems XXVI. ed. / Marco Molinaro; Keith Shortridge; Fabio Pasian. Astronomical Society of the Pacific, 2019. p. 628-631 (ASP Conference Series; Vol. 521).

Research output: Chapter in Book/Conference paper › Conference paper › peer-review

TY - GEN

T1 - Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

AU - Vinsen, Kevin

AU - Boulton, Mark

AU - Cooper, Ian

AU - Dodson, Richard

AU - Dolensky, Markus

AU - Pallot, Dave

AU - Tobar, Rodrigo

AU - Wicenec, Andreas

AU - Wu, Chen

N1 - Conference code: 26

PY - 2019/10/1

Y1 - 2019/10/1

N2 - Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.

AB - Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.

M3 - Conference paper

SN - 978-1-58381-929-6

T3 - ASP Conference Series

SP - 628

EP - 631

BT - Astronomical Data Analysis Software and Systems XXVI

A2 - Molinaro, Marco

A2 - Shortridge, Keith

A2 - Pasian, Fabio

PB - Astronomical Society of the Pacific

T2 - 26th Annual Conference for Astronomical Data Analysis Software and Systems

Y2 - 16 October 2016 through 20 October 2016

ER -

Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

Abstract

Publication series

Conference

Fingerprint

Cite this