Imaging SKA-scale data in three different computing environments

    Research output: Contribution to journalArticle

    7 Citations (Scopus)

    Abstract

    © 2015 Elsevier B.V. We present the results of our investigations into options for the computing platform for the imaging pipeline in the chiles project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. chiles pushes the current computing infrastructure to its limits and understanding how to deliver the images from this project is clarifying the Science Data Processing requirements for the SKA. We have tested three platforms: a moderately sized cluster, a massive High Performance Computing (HPC) system, and the Amazon Web Services (AWS) cloud computing platform. We have used well-established tools for data reduction and performance measurement to investigate the behaviour of these platforms for the complicated access patterns of real-life Radio Astronomy data reduction. All of these platforms have strengths and weaknesses and the system tools allow us to identify and evaluate them in a quantitative manner. With the insights from these tests we are able to complete the imaging pipeline processing on both the HPC platform and also on the cloud computing platform, which paves the way for meeting big data challenges in the era of SKA in the field of Radio Astronomy. We discuss the implications that all similar projects will have to consider, in both performance and costs, to make recommendations for the planning of Radio Astronomy imaging workflows.
    Original languageEnglish
    Pages (from-to)8-22
    JournalAstronomy and Computing
    Volume14
    DOIs
    Publication statusPublished - 2016

    Fingerprint

    Radio astronomy
    platforms
    Cloud computing
    Imaging techniques
    Data reduction
    radio astronomy
    Pipelines
    Chile
    data reduction
    Web services
    Planning
    web services
    Processing
    Costs
    recommendations
    planning
    costs
    requirements

    Cite this

    @article{ad29c34ab6ef42e3b85d91630e42a7c2,
    title = "Imaging SKA-scale data in three different computing environments",
    abstract = "{\circledC} 2015 Elsevier B.V. We present the results of our investigations into options for the computing platform for the imaging pipeline in the chiles project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. chiles pushes the current computing infrastructure to its limits and understanding how to deliver the images from this project is clarifying the Science Data Processing requirements for the SKA. We have tested three platforms: a moderately sized cluster, a massive High Performance Computing (HPC) system, and the Amazon Web Services (AWS) cloud computing platform. We have used well-established tools for data reduction and performance measurement to investigate the behaviour of these platforms for the complicated access patterns of real-life Radio Astronomy data reduction. All of these platforms have strengths and weaknesses and the system tools allow us to identify and evaluate them in a quantitative manner. With the insights from these tests we are able to complete the imaging pipeline processing on both the HPC platform and also on the cloud computing platform, which paves the way for meeting big data challenges in the era of SKA in the field of Radio Astronomy. We discuss the implications that all similar projects will have to consider, in both performance and costs, to make recommendations for the planning of Radio Astronomy imaging workflows.",
    author = "Richard Dodson and Kevin Vinsen and Chen Wu and Attila Popping and Martin Meyer and Andreas Wicenec and Peter Quinn and {Van Gorkom}, J. and E. Momjian",
    year = "2016",
    doi = "10.1016/j.ascom.2015.10.007",
    language = "English",
    volume = "14",
    pages = "8--22",
    journal = "Astronomy and Computing",
    issn = "2213-1337",
    publisher = "Elsevier",

    }

    TY - JOUR

    T1 - Imaging SKA-scale data in three different computing environments

    AU - Dodson, Richard

    AU - Vinsen, Kevin

    AU - Wu, Chen

    AU - Popping, Attila

    AU - Meyer, Martin

    AU - Wicenec, Andreas

    AU - Quinn, Peter

    AU - Van Gorkom, J.

    AU - Momjian, E.

    PY - 2016

    Y1 - 2016

    N2 - © 2015 Elsevier B.V. We present the results of our investigations into options for the computing platform for the imaging pipeline in the chiles project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. chiles pushes the current computing infrastructure to its limits and understanding how to deliver the images from this project is clarifying the Science Data Processing requirements for the SKA. We have tested three platforms: a moderately sized cluster, a massive High Performance Computing (HPC) system, and the Amazon Web Services (AWS) cloud computing platform. We have used well-established tools for data reduction and performance measurement to investigate the behaviour of these platforms for the complicated access patterns of real-life Radio Astronomy data reduction. All of these platforms have strengths and weaknesses and the system tools allow us to identify and evaluate them in a quantitative manner. With the insights from these tests we are able to complete the imaging pipeline processing on both the HPC platform and also on the cloud computing platform, which paves the way for meeting big data challenges in the era of SKA in the field of Radio Astronomy. We discuss the implications that all similar projects will have to consider, in both performance and costs, to make recommendations for the planning of Radio Astronomy imaging workflows.

    AB - © 2015 Elsevier B.V. We present the results of our investigations into options for the computing platform for the imaging pipeline in the chiles project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. chiles pushes the current computing infrastructure to its limits and understanding how to deliver the images from this project is clarifying the Science Data Processing requirements for the SKA. We have tested three platforms: a moderately sized cluster, a massive High Performance Computing (HPC) system, and the Amazon Web Services (AWS) cloud computing platform. We have used well-established tools for data reduction and performance measurement to investigate the behaviour of these platforms for the complicated access patterns of real-life Radio Astronomy data reduction. All of these platforms have strengths and weaknesses and the system tools allow us to identify and evaluate them in a quantitative manner. With the insights from these tests we are able to complete the imaging pipeline processing on both the HPC platform and also on the cloud computing platform, which paves the way for meeting big data challenges in the era of SKA in the field of Radio Astronomy. We discuss the implications that all similar projects will have to consider, in both performance and costs, to make recommendations for the planning of Radio Astronomy imaging workflows.

    U2 - 10.1016/j.ascom.2015.10.007

    DO - 10.1016/j.ascom.2015.10.007

    M3 - Article

    VL - 14

    SP - 8

    EP - 22

    JO - Astronomy and Computing

    JF - Astronomy and Computing

    SN - 2213-1337

    ER -