Experiences of Large International Data Transfer

Large scale data transfer was not our primary objective in looking to participate in the 2012 IEEE Large Scale Data Analysis and Visualization Contest. It was however a necessary first step before we could begin work on the contest problem.

We will describe our experiences of transferring 4 TB of data from Oak Ridge National Lab, US to the New Zealand eScience Infrastructure at BlueFern Supercomputing Centre, University of Canterbury.

We will outline the steps we had to go through to gain access to the data and to perform an initial trial using the publically available Data Transfer nodes located at various US sites.

Our discussion will focus on how the KAREN network enabled the transfer and the considerations we had to make to avoid a “tragedy of the commons” situation both with local network access and international connectivity. We will also discuss our experiences of GridFTP, transfer monitoring, credential management and authentication, placing systems under heavy load and how the Science DMZ concept would have altered our experience.

eResearch NZ 2013 session type: 


Submitted by Tim McNamara on