The performance impact of data modelling for large scale astronomical data analysis on distributed and parallel platforms

Geoffrey Duniam

doi:10.26182/yk40-tx91

The performance impact of data modelling for large scale astronomical data analysis on distributed and parallel platforms

Geoffrey Duniam

School of Physics, Mathematics and Computing

Research output: Thesis › Doctoral Thesis

151 Downloads (Pure)

Abstract

The new generation of large-scale telescopes will produce thousands of data products in hundreds of GBs to multiple TBs, increasing in complexity as they increase in size. Large scale data products will present several challenges that will need to be addressed to enable the science outputs from this data. This thesis investigates new techniques to extract, transform, model and load data from large scale raw data products into efficient data structures for storage and retrieval on an open-source parallel storage and processing platform. These approaches are investigated and demonstrated by exploring three separate use cases working with astronomical data.

Original language	English
Qualification	Doctor of Philosophy
Awarding Institution	The University of Western Australia
Supervisors/Advisors	Wu, Chen, Supervisor Kitaeff, Slava, Supervisor Wicenec, Andreas, Supervisor
Thesis sponsors	Australian Government Research Training Program
Award date	4 Oct 2023
DOIs	https://doi.org/10.26182/yk40-tx91
Publication status	Unpublished - 2023

Access to Document

10.26182/yk40-tx91

THESIS - DOCTOR OF PHILOSOPHY - DUNIAM Geoffrey Jack - 2023
This work is protected by Copyright. You may print or download ONE copy of this document for the purpose of your own non-commercial research or study. Any other use requires permission from the copyright owner. The Copyright Act requires you to attribute any copyright works you quote or paraphrase.

Cite this

@phdthesis{bdda08a1c182478592687cdb9bbe0bad,

title = "The performance impact of data modelling for large scale astronomical data analysis on distributed and parallel platforms",

abstract = "The new generation of large-scale telescopes will produce thousands of data products in hundreds of GBs to multiple TBs, increasing in complexity as they increase in size. Large scale data products will present several challenges that will need to be addressed to enable the science outputs from this data. This thesis investigates new techniques to extract, transform, model and load data from large scale raw data products into efficient data structures for storage and retrieval on an open-source parallel storage and processing platform. These approaches are investigated and demonstrated by exploring three separate use cases working with astronomical data.",

keywords = "Big Data, Parallel processing, data modelling",

author = "Geoffrey Duniam",

year = "2023",

doi = "10.26182/yk40-tx91",

language = "English",

school = "The University of Western Australia",

}

TY - BOOK

T1 - The performance impact of data modelling for large scale astronomical data analysis on distributed and parallel platforms

AU - Duniam, Geoffrey

PY - 2023

Y1 - 2023

N2 - The new generation of large-scale telescopes will produce thousands of data products in hundreds of GBs to multiple TBs, increasing in complexity as they increase in size. Large scale data products will present several challenges that will need to be addressed to enable the science outputs from this data. This thesis investigates new techniques to extract, transform, model and load data from large scale raw data products into efficient data structures for storage and retrieval on an open-source parallel storage and processing platform. These approaches are investigated and demonstrated by exploring three separate use cases working with astronomical data.

AB - The new generation of large-scale telescopes will produce thousands of data products in hundreds of GBs to multiple TBs, increasing in complexity as they increase in size. Large scale data products will present several challenges that will need to be addressed to enable the science outputs from this data. This thesis investigates new techniques to extract, transform, model and load data from large scale raw data products into efficient data structures for storage and retrieval on an open-source parallel storage and processing platform. These approaches are investigated and demonstrated by exploring three separate use cases working with astronomical data.

KW - Big Data

KW - Parallel processing

KW - data modelling

U2 - 10.26182/yk40-tx91

DO - 10.26182/yk40-tx91

M3 - Doctoral Thesis

ER -

The performance impact of data modelling for large scale astronomical data analysis on distributed and parallel platforms

Abstract

Access to Document

Fingerprint

Cite this