Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE

Kevin Vinsen, Mark Boulton, Ian Cooper, Richard Dodson, Markus Dolensky, Dave Pallot, Rodrigo Tobar, Andreas Wicenec, Chen Wu

Research output: Chapter in Book/Conference paperConference paper

Abstract

Just one of the Square Kilometre Array (SKA) Phase I science projects will produce data of the order of tens of terabytes per second. The SKA Phase I project will have stringent power constraints in order to limit the operational costs (Dewdney et al. 2013); it is a considerable challenge to manage, process and store such large datasets within these constraints.

The current state-of-the-art astronomy data processing systems are designed to handle data approximately two to three orders of magnitude smaller than the SKA Phase I. To tackle this challenge, we have developed the Data-Activated Flow Graph Engine (DALiuGE), as part of the prototyping effort for the Science Data Processor Consortium of the SKA Phase I design. DALiuGE aims to provide a distributed data management platform and a scalable pipeline execution environment to support continuous, time and power bounded, data-intensive processing for producing SKA science-ready products. In this paper, we provide a brief overview of DALiuGE.
Original languageEnglish
Title of host publicationAstronomical Data Analysis Software and Systems XXVI
EditorsMarco Molinaro, Keith Shortridge, Fabio Pasian
PublisherAstronomical Society of the Pacific
Pages628-631
ISBN (Print)978-1-58381-929-6
Publication statusPublished - 1 Oct 2019
Event26th Annual Conference for Astronomical Data Analysis Software and Systems - Italian National Institute of Astrophysics, Trieste, Italy
Duration: 16 Oct 201620 Oct 2016
Conference number: 26

Publication series

NameASP Conference Series
PublisherAstronomical Society of the Pacific
Volume521
ISSN (Print)1050-3390

Conference

Conference26th Annual Conference for Astronomical Data Analysis Software and Systems
Abbreviated titleADASS XXVI
CountryItaly
CityTrieste
Period16/10/1620/10/16

Fingerprint Dive into the research topics of 'Imaging SKA-Scale Data on Cloud and Supercomputer Infrastructure Using Drops and DALiuGE'. Together they form a unique fingerprint.

Cite this