Australasian Science: Australia's authority on science since 1938

Big Data for Big Astronomy

Credit: alexovicsattila/iStock

Credit: alexovicsattila/iStock

By Peter Quinn

The Square Kilometre Array will generate huge amounts of data. Can computing capacity keep up?

The full text of this article can be purchased from Informit.

When completed, the SKA will be capable of producing a stream of raw science data that will require computers with processing power beyond the performance of the largest computing systems on Earth today. The resulting scientific data will accumulate at a rate that exceeds 1 Exabyte (one million Terabytes) per year.

These figures represent a scientific endeavour that is Google-scale, and consequently the SKA is attracting considerable international interest and excitement from within the industrial and academic communities. Much of this excitement, and the associated challenges and opportunities, are shared with other research and commercial disciplines under the banner of “Big Data”.

Some of the most significant challenges for the SKA may be addressed by learning from other Exa-scale enterprises to ensure the affordable, effective and global achievement of SKA science goals.

The SKA as a Big Data Project

A lot has been written about the Big Data problem, but what kind of a problem is it? Is there one root issue? Are there many? Are they all technical? And if we can define the parameters of the problem, can we identify where solutions might be found?

Phase 1 of the SKA is expected to be operational in the first half of the next decade. On that time scale, what are the main components of the Big Data challenge that must be faced to...

The full text of this article can be purchased from Informit.