ESO's archive computing framework

Andreas J. Wicenec, Jens Knudstrup

Research output: Contribution to journalConference articlepeer-review

2 Citations (Scopus)

Abstract

ESO's Science Archive is distributed across four different sites on two continents. With the huge amount of data produced by the various instruments this poses special requirements on the way data is transfered between the sites and distributed to the various subscribers. ESO's latest development, the Next Generation Archive System (NGAS), is based on cheap ATA disks connected to custom PCs running http based servers controlling the archiving process, supporting retrieval and checking the health status of the disks and the data itself. The current deployment of this system covers just a single 8kx8k pixel wide field imager, which is producing about 30 GB of raw data per night. The next generation of wide field telescopes/instruments VISTA/VISTACam and VST/OmegaCam will produce data rates well exceeding 500 GB and 125 GB during a single typical night, respectively. The total data rate of all ESO telescopes/instruments will grow to about 0.75 TB/night once VISTA is operational. The archiving of this data is essential, the next important step is to support not just only retrieval, but also flexible processing schemes of the data within the archive cluster directly. The NGAS already supports fully transparent retrieval of processed data on a single file basis, where the users just specifies a file name and one of the supported process to be executed. The goal is to offer a system where one can apply a predefined processing logic on any number of frames and provide the processing scheme on top to maximize the throughput of such a request. This paper gives an outline of the framework and a brief introduction to the technologies used.

Original languageEnglish
Pages (from-to)137-144
Number of pages8
JournalProceedings of SPIE - The International Society for Optical Engineering
Volume4845
DOIs
Publication statusPublished - 2002
Externally publishedYes
EventAdvanced Global Communications Technologies for Astronomy II - Waikoloa, HI, United States
Duration: 22 Aug 200223 Aug 2002

Fingerprint

Dive into the research topics of 'ESO's archive computing framework'. Together they form a unique fingerprint.

Cite this