The Dutch database technology company MonetDB Solutions and CWI will develop new techniques to facilitate the storage and analysis of the vast data volumes from the Square Kilometre Array (SKA) telescope. The project will be funded by a grant in the AstroCompute in the Cloud program of the SKA Organization (SKAO) and Amazon Web Services (AWS).
The SKA project is an international effort to build the world’s largest radio telescope. The telescope will eventually have over a square kilometre (one million square metres) of collecting area, spread throughout Australia and South Africa. The telescope will generate vast amounts of data at 100’s of gigabits per second, which need to be processed in real-time. Therefore, MonetDB Solutions and CW will develop technology that ensures that the system can sustainably handle growth in terms of both query complexity and data volume. In addition, it will allow for safer addition of new features and performance improvements of the software and algorithms used in the data processing, guaranteeing they work as expected.
The database management system is a critical component in the processing of astronomical data. The team will use MonetDB, an open-source columnar database management system optimized for analytical workloads. This project uses MonetDB’s embedded R and Python capabilities for fast statistical data analysis, and the MonetDB Data Vaults extension for handling external data files.
The AstroCompute in the Cloud program is initiated by AWS and SKAO to accelerate tools and technique development for storage and analysis of the vast data volumes produced by modern telescopes. The complete system will run in the AWS cloud. To this end, Amazon provides 1 petabyte of storage capacity for all grant recipients in the program. The project is expected to be finished at the end of 2016.
Image: skatelescope.org
