A Real Time Processing system for big data in astronomy: Applications to HERA

P La Plante; PKG Williams; M Kolopanis; JS Dillon; AP Beardsley; NS Kern; M Wilensky; ZS Ali; Z Abdurashidova; JE Aguirre; P Alexander; Y Balfour; G Bernardi; TS Billings; JD Bowman; RF Bradley; P Bull; J Burba; S Carey; CL Carilli; C Cheng; DR DeBoer; M Dexter; E de Lera Acedo; J Ely; A Ewall-Wice; N Fagnoni; R Fritz; SR Furlanetto; K Gale-Sides; B Glendenning; D Gorthi; B Greig; J Grobbelaar; Z Halday; BJ Hazelton; JN Hewitt; J Hickish; DC Jacobs; A Julius; J Kerrigan; P Kittiwisit; SA Kohn; A Lanman; T Lekalake; D Lewis; A Liu; D MacMahon; L Malan; C Malgas; M Maree; ZE Martinot; E Matsetela; A Mesinger; M Molewa; MF Morales; T Mosiane; S Murray; AR Neben; B Nikolic; AR Parsons; R Pascua; N Patra; S Pieterse; JC Pober; N Razavi-Ghods; J Ringuette; J Robnett; K Rosie; MG Santos; P Sims; C Smith; A Syce; N Thyagarajan; H Zheng

Journal article

A Real Time Processing system for big data in astronomy: Applications to HERA

P La Plante, PKG Williams, M Kolopanis, JS Dillon, AP Beardsley, NS Kern, M Wilensky, ZS Ali, Z Abdurashidova, JE Aguirre, P Alexander, Y Balfour, G Bernardi, TS Billings, JD Bowman, RF Bradley, P Bull, J Burba, S Carey, CL Carilli Show all

ASTRONOMY AND COMPUTING | ELSEVIER | Published : 2021

DOI: 10.1016/j.ascom.2021.100489

Abstract

As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquir..

View full abstract

University of Melbourne Researchers

Bradley Greig Author Physics

Grants

Awarded by National Science Foundation, United States of America

Awarded by Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D)

Awarded by INAF PRIN-SKA 2017, Italy

Awarded by Ministero degli Affari Esteri della Coorperazione Internazionale -Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, Italy

Awarded by National Research Foundation of South Africa as part of the ISARP RADIOSKY2020 Joint Research Scheme

Awarded by Newton Fund, UK

Awarded by National Research Foundation of South Africa

Awarded by European Research Council (ERC) under the European Union

Awarded by STFC, UK

Awarded by NSF AAPF, United States of America

Funding Acknowledgements

This material is based upon work supported by the National Science Foundation, United States of America under Grant No. 1636646, the Gordon and Betty Moore Foundation, and institutional support from the HERA collaboration partners, United States of America. HERA is hosted by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Parts of this research were supported by the Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D), through project number CE170100013. G. Bernardi acknowledges funding from the INAF PRIN-SKA 2017, Italy project 1.05.01.88.04 (FORECaST), support from the Ministero degli Affari Esteri della Coorperazione Internazionale -Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, Italy ZA18GR02 and the National Research Foundation of South Africa (Grant No. 113121) as part of the ISARP RADIOSKY2020 Joint Research Scheme, from the Royal Society, UK and the Newton Fund, UK under grand NA150184 and from the National Research Foundation of South Africa (Grant No. 103424). P. Bull acknowledges funding for part of this research from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 948764), and from STFC, UK Grant ST/T000341/1. J.S. Dillon gratefully acknowledges the support of the NSF AAPF, United States of America award #1701536. N Kern acknowledges support from the MIT Pappalardo, UK fellowship. A. Liu acknowledges support from the New Frontiers in Research Fund Exporation grant program, Canada, the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program, a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and a Discovery Launch Supplement, the Sloan Research Fellowship, and the William Dawson Scholarship at McGill, United States of America.