SFTP file transfer to the cluster¶
It is assumed that you have a basic experience with the linux terminal. Should you require a recap of comman commands, go to this notebook. + Let’s embed a terminal to begin with.
from IPython.display import IFrame your_gitlab_username = "revins" # change this to your username IFrame( "https://hub.besos.uvic.ca/user/" + your_gitlab_username + "/terminals/1", width=1200, height=250, )
Connecting to the cluster¶
cd, navigate to the directory in which you have kept the files to be transferred to cluster. See (
sftp cc_username@cluster_name.computecanada.cain the terminal.
- Enter the password when prompted.
- This should get you the
SFTP commands refresher¶
You will need to execute commands on the remote machine as well as on
the local machine. When inside of
sftp> prompt, all the linux
commands work as if they are executed on remote machine. To execute any
command locally, prefix it with a
l denoting local. Note that all
commands cannot be executed locally inside the
sftp> prompt using
l. To explicitly execute a command locally prefix it with
- On the remote machine (i.e. the cluster):
pwd, cd, ls, mkdir, etc.
- On the local machine (i.e. the BESOS hub):
lpwd, lcd, lls, lmkdir, etc.
General process for transferring files to or from the cluster¶
We want to: 1. Set the directories (between which file transfer must to
take place) on local and remote machine using
respectively. 2. Execute
get filename to copy the file from the
remote to local, or
get -r folder if a folder is to be transferred.
put filename to copy the file from the local to remote,
put -r folder if a folder is to be transferred. 4. Execute
exit to exit the
See the examples below.
Sending files from BESOS to the cluster¶
Here we will send
clusterbatch.sh on the cluster.
Copying files to the cluster¶
When the correct target and source directories have been set, execute
the following from inside of
sftp> prompt: +
[copying cluster.py to remote] +
put parameter_sets.py [copying
parameter_sets.py to remote] +
put clusterbatch.sh [copying
clusterbatch.sh to remote]
Getting files from the cluster back to BESOS¶
Here we will get
op.out from the cluster.
Navigating to the directory in which to transfer files¶
sftp>prompt spawns in the current directory only.
Navigating to the directory from which to transfer files¶
- We now need to go to the directory on cluster (remote) from which we
submitted the job. We called this directory
- Execute the following from inside of
cd /home/cc_username[navigate to home directory]
cd scratch[navigate to scratch]
cd job[navigate to ‘job’]
Copying files from the cluster¶
- When the correct target and source directories are set, execute the
following from inside of
get time.cluster[copying time.cluster from remote]
get op.out[copying op.out from remote]