Data Transfer for Education and Training Allocations

The security policies for the Education and Training accounts require a different way to do data movement. As the accounts do not have RSA tokens, the accounts are not able to use the Blue Waters Globus Online public endpoints, nor can they use scp or sftp to "push" data to Blue Waters from a remote host. Education and Training do not have access to the Blue Waters Nearline storage system by default due to the absence of the RSA token.

Below are some suggestions on how to transfer files and data to Blue Waters for Education and Training accounts.

Please contact help+bw@ncsa.illinois.edu if there are questions or issues.

Use of login nodes

Educational allocations do not have access to the published Blue Waters Globus Online endpoints.

Small files ( less than 1 GB or 10 GB total for directories )

You can use rsync to copy filles off the Blue Waters file system, but need to make sure that all data is piped through the bwbay host like so

rsync -e 'ssh traXXX@bwbay.ncsa.illinois.edu ssh' bw.ncsa.illinois.edu:yourfile-on-bw your-file-on-local-machine

If you are using WinSCP or the Unix sftp command then you must set the "SFTP server" setting in the advanced settings or via the -s option to "ssh bw /usr/lib64/ssh/sftp-server". This screenshot shows how it should be done for WinSCP.

Finally you can use sshfs (available for Linux, MacOS and Windows) like so:

mkdir ~/bw
sshfs -o ssh_command="ssh traXXX@bwbay.ncsa.illinois.edu ssh" bw:/ ~/bw

Larger files

  • For large files we recommend using Globus Online by way of Globus Connect. Start a globus connect client from your login session on Blue Waters, and use Globus Online (GO) to transfer data between a login node and off-site GO endpoints. Globus Connect setup is documented on the Blue Waters portal at https://bluewaters-archive.ncsa.illinois.edu/setting-up-globus-connect.  You cannot use this method to transfer between Blue Waters and your laptop, since only one endpoint in a transfer can be a globus connect personal endpoint.

  • You will need to create a GO account if you do not already have one at globus.org .  Supporting software is available here (along with steps to get your Globus Connect endpoint going) by staring with a copy of the Globus Connect software (check with GO and download the latest if a newer version is available).

cd $HOME
tar xzvf /sw/userdoc/gofiles/globusconnect-latest.tgz
cd globusconnect*
./globusconnectpersonal -setup <your_setup_key>
./globusconnectpersonal -start -debug

Now start a browser somewhere and use your new globus connect endpoint on h2ologinN 

Caveats

  • 2 globusconnectpersonal clients cannot transfer data between them, that is you cannot use globuspersonalconnect to transfer between Blue Waters and your laptop using two globuspersonalconnect processes. Instead you need to use rsync or an account on a regular globus site as an intermediary
  • Interactive limits will apply on login nodes; long running transfers (hours) will require special arrangements
  • Use no more than 2 login nodes for moving files.  If load or interactive response becomes an issue, the admins will terminate data transfers without warning.
  • Interactive response for compiling and building is the top priority for login nodes