SFTP file transfer to the cluster
It is assumed that you have a basic experience with the linux terminal. Should you require a recap of comman commands, go to this notebook. + Let’s embed a terminal to begin with.
from IPython.display import IFrame
your_gitlab_username = "revins" # change this to your username
IFrame(
"https://hub.besos.uvic.ca/user/" + your_gitlab_username + "/terminals/1",
width=1200,
height=250,
)
Connecting to the cluster
Using
cd
, navigate to the directory in which you have kept the files to be transferred to cluster. See (`cd
tutorial <TerminalTutorial.ipynb#cd>`__).Execute
sftp cc_username@cluster_name.computecanada.ca
in the terminal.Enter the password when prompted.
This should get you the
sftp>
prompt.
SFTP commands refresher
You will need to execute commands on the remote machine as well as on
the local machine. When inside of sftp>
prompt, all the linux
commands work as if they are executed on remote machine. To execute any
command locally, prefix it with a l
denoting local. Note that all
commands cannot be executed locally inside the sftp>
prompt using
l
. To explicitly execute a command locally prefix it with !
.
On the remote machine (i.e. the cluster):
pwd, cd, ls, mkdir, etc.
On the local machine (i.e. the BESOS hub):
lpwd, lcd, lls, lmkdir, etc.
General process for transferring files to or from the cluster
We want to: 1. Set the directories (between which file transfer must to
take place) on local and remote machine using lcd
and cd
respectively. 2. Execute get filename
to copy the file from the
remote to local, or get -r folder
if a folder is to be transferred.
3. Execute put filename
to copy the file from the local to remote,
or put -r folder
if a folder is to be transferred. 4. Execute
exit
to exit the sftp>
prompt.
See the examples below.
Sending files from BESOS to the cluster
Here we will send cluster.py
, parameter_sets.py
and
clusterbatch.sh
on the cluster.
Copying files to the cluster
When the correct target and source directories have been set, execute
the following from inside of sftp>
prompt: + put cluster.py
[copying cluster.py to remote] + put parameter_sets.py
[copying
parameter_sets.py to remote] + put clusterbatch.sh
[copying
clusterbatch.sh to remote]
Now you can submit the job for execution on the cluster.
Go back to the cluster submission notebook for steps.
Getting files from the cluster back to BESOS
Here we will get time.cluster
and op.out
from the cluster.
Navigating to the directory in which to transfer files
sftp>
prompt spawns in the current directory only.lcd
and lpwd
.Navigating to the directory from which to transfer files
We now need to go to the directory on cluster (remote) from which we submitted the job. We called this directory
job
.Execute the following from inside of
sftp>
prompt:cd /home/cc_username
[navigate to home directory]cd scratch
[navigate to scratch]cd job
[navigate to ‘job’]
Copying files from the cluster
When the correct target and source directories are set, execute the following from inside of
sftp>
prompt:get time.cluster
[copying time.cluster from remote]get op.out
[copying op.out from remote]