Mapping the universe and other embarrassingly parallel problems (Part of the NZ HPC Applications Workshop)

The Sloan Deep Sky Survey (SDSS) is an American astrophysics project in its tenth year, which has the ultimate goal of mapping the observable universe. In July 2012, this project team released over 60 terabytes of scientific data into the public domain, allowing anyone to download their photographic observations and measurements of millions of stars and galaxies. To date, the project team have photographed and collected data on approximately one quarter of all the observable objects in the night sky.

Using the data this team has collected, it should be possible to create a virtual 3D space in which to accurately plot the relative positions of these stars and galaxies and ‘fly through the universe’. To achieve this goal, the parallel processing capabilities of Whitireia’s High Performance Computing Centre (HPCC) will be used for real-time rendering of millions of astronomical images as a user navigates within this virtual universe.

As a preliminary step towards this goal, several sub-projects are currently underway to tackle a series of embarrassingly parallel problems, including a deep zoom into the Mandelbrot Set and the processing of a large dataset using the K-Means data mining algorithm. These sub-projects will develop our experience in parallel programming and the movement of large amounts of data. They will also assist in refining the configuration of our Rocks cluster and the scheduling and processing of parallelised programming tasks. Parallel programming will be performed in R (an open source statistical programming language) using Rmpi; the Open MPI interface for R. Data will be accessed from an Oracle database using Oracle R Enterprise, which is a component of the Oracle Advanced Analytics Option of the Oracle Database Enterprise Edition.

The presentation will describe the outcomes of these sub-projects and provide a demonstration of the capabilities of Oracle R Enterprise that will involve plotting the relative positions of over 2 million astronomical objects (rendered as points in this proof of concept, rather than images) within a virtual 3D space.

eResearch NZ 2013 session type: 


Submitted by Tim McNamara on