SFTP file transfer to the cluster

It is assumed that you have a basic experience with the linux terminal. Should you require a recap of comman commands, go to this notebook. + Let’s embed a terminal to begin with.

from IPython.display import IFrame


your_gitlab_username = "revins"  # change this to your username
IFrame(
    "https://hub.besos.uvic.ca/user/" + your_gitlab_username + "/terminals/1",
    width=1200,
    height=250,
)

Connecting to the cluster

  • Using cd, navigate to the directory in which you have kept the files to be transferred to cluster. See (`cd tutorial <TerminalTutorial.ipynb#cd>`__).
  • Execute sftp cc_username@cluster_name.computecanada.ca in the terminal.
  • Enter the password when prompted.
  • This should get you the sftp> prompt.

SFTP commands refresher

You will need to execute commands on the remote machine as well as on the local machine. When inside of sftp> prompt, all the linux commands work as if they are executed on remote machine. To execute any command locally, prefix it with a l denoting local. Note that all commands cannot be executed locally inside the sftp> prompt using l. To explicitly execute a command locally prefix it with !.

  • On the remote machine (i.e. the cluster): pwd, cd, ls, mkdir, etc.
  • On the local machine (i.e. the BESOS hub): lpwd, lcd, lls, lmkdir, etc.

General process for transferring files to or from the cluster

We want to: 1. Set the directories (between which file transfer must to take place) on local and remote machine using lcd and cd respectively. 2. Execute get filename to copy the file from the remote to local, or get -r folder if a folder is to be transferred. 3. Execute put filename to copy the file from the local to remote, or put -r folder if a folder is to be transferred. 4. Execute exit to exit the sftp> prompt.

See the examples below.

Sending files from BESOS to the cluster

Here we will send cluster.py, parameter_sets.py and clusterbatch.sh on the cluster.

Copying files to the cluster

When the correct target and source directories have been set, execute the following from inside of sftp> prompt: + put cluster.py [copying cluster.py to remote] + put parameter_sets.py [copying parameter_sets.py to remote] + put clusterbatch.sh [copying clusterbatch.sh to remote]

Now you can submit the job for execution on the cluster.

Go back to the cluster submission notebook for steps.

Getting files from the cluster back to BESOS

Here we will get time.cluster and op.out from the cluster.

Navigating to the directory in which to transfer files

We need to go to the directory in which you want your files to be copied from cluster.
We will copy them in the current directory so we need not navigate since sftp> prompt spawns in the current directory only.
If you are not in the directory you want to copy the files to, navigate using lcd and lpwd.

Navigating to the directory from which to transfer files

  • We now need to go to the directory on cluster (remote) from which we submitted the job. We called this directory job.
  • Execute the following from inside of sftp> prompt:
    • cd /home/cc_username [navigate to home directory]
    • cd scratch [navigate to scratch]
    • cd job [navigate to ‘job’]

Copying files from the cluster

  • When the correct target and source directories are set, execute the following from inside of sftp> prompt:
    • get time.cluster [copying time.cluster from remote]
    • get op.out [copying op.out from remote]

Now we can exit the sftp> prompt by typing exit