Skip to content

File transfer using sftp

Learning outcomes

  • Practice using the documentation of your favorite HPC cluster
  • (Optional) Can transfer files using sftp
For teachers

Teaching goals are:

  • Learners have practiced using the documentation of their favorite HPC cluster
  • Learners have transferred files using sftp

Prior questions:

  • Who has heard of sftp?
  • Who has already used sftp?

Introduction

SFTP is an abbreviation of 'SSH File Transfer Protocol' and 'SSH' is an abbreviation of 'Secure Shell protocol'.

  • The difference from rsync and scp is that you login to a SFTP session
    • just like a log in to a server with SSH.
  • You use the terminal to transfer the files.
  • You need the credentials only when establishing the connection.

Pros

  • Secure
  • Use cases:
    • copying large or many files
    • asks if you want to overwrite target files
    • possible to resume file transfer after a broken connection.
    • handle bulk file transfers
    • modify directories
    • and encrypt or decrypt specific files
When not to use
  • Rsync is better for "syncing"

Procedure

  • Run the sftp commands on YOUR computer, since you probably do not have a server address to your computer!
  • In the terminal (from local, not server session)
sftp <username>@<cluster address>

In sftp session, go to right folders and upload/download files to/from the server. You can do this as long as the connection is active!

  • pwd/cd/ls - browse and list files in server file system
  • lpwd/lcd/lls - browse and list files in local file system
  • put <file/folder> - Copy a file/folder from the local computer to the remote host
  • get <file/folder> - Copy a file/folder from the remote host to the local computer

Options for compressing during the transfer

  • Start the sftp session with sftp -C user@host
  • The file(-s) are then also decompressed on the destination.
Example for Tetralith

Connect to SFTP session

In the terminal, run sftp to connect to Tetralith by doing:

sftp x_nisse@tetralith.nsc.liu.se

If asked, give your center's password. You can get rid of this prompt if you have setup SSH keys

Upload/download files to/from Tetralith

In sftp session, upload/download files/folders to/from the server.

  • put localfile.txt - Copy a file from the local computer to the remote host
  • get serverfile.txt - Copy a file from the remote host to the local computer
Cheat sheet in the SFTP session
  • ?/help - List of all commands or explain a command, eg ? get

Remote

  • put - Copy a file from the local computer to the remote host
  • cd - Change the active directory path on the remote host
  • dir/ls - List the contents of the current directory on the remote host
  • pwd - Show the present working directory path on the remote host
  • mkdir - Create a directory on the remote host
  • rename - Rename a file on the remote host
  • rm - Delete a file on the remote host
  • rmdir - Remove an empty directory on the remote host
  • exit/quit - Close the connection and leave SFTP

Locally

  • get - Copy a file from the remote host to the local computer
  • lcd - Change the active directory on the local system
  • lls - List the contents of the current directory on the local computer
  • lmkdir - Create a directory on the local computer
  • lpwd - Show the present working directory on the local computer

Exercises

Exercise 0: Use the documentation of your HPC cluster
  • Search for how to transfer files to/from your HPC cluster using scp. Which URL is it described?
  • Tip: not all HPC centers have documented this, so you should give up searching after a while.
  • If the center maintaining you HPC cluster has not documented how to use sftp, follow the Rackham documentation.
Where is that documentation?
HPC Cluster Documentation
Alvis None yet.
Berzelius Documentation
Bianca Documentation, through transit
COSMOS Documentation
Dardel Documentation about the similar psftp
Kebnekaise Documentation
LUMI Apparently not supported
Pelle Documentation
Rackham Documentation
Tetralith Documentation
Vera None yet.
Dardel documentation uses psftp, what is that?
  • It is just a SFTP component of the client PuTTY.
  • Same commands as for SFTP.
Exercise 1: Establish the sftp connection and find the relevant folders locally and remotely

Tips

  • Connect from your local terminal
  • Orient yourself remotely and locally with ls, cd,pwd,lls etc...
  • Where are you remotely?
  • Go to the course folder where you have files
  • Where are you locally?
  • Go to your local folder used for the course
Answer (Tetralith example)

Locally

  • Establish connection
$ sftp sm_bcarl@tetralith.nsc.liu.se
(sm_bcarl@tetralith.nsc.liu.se) Password:
(sm_bcarl@tetralith.nsc.liu.se) Verification code:
Connected to tetralith.nsc.liu.se.
  • Use pwd to find where you are remotely
  • Use cd to reach relevant remote folder
  • Use lpwd to find where you are locally

  • Use lcd to reach relevant local folder

  • Video for Tetralith

Exercise 2: Upload

Tips

  • (If you want to create a file in local terminal: $ touch local_file_sftp)
  • Send it to an existing folder (e.g. transfer) on Tetralith
  • Check on server (in sftp session) that it is there
Answer (Tetralith example)

Locally

  • (If you want to create a file in local terminal: $ touch local_file_sftp)
  • use the put command: > put local_file_sftp

Check on server (in sftp session) that it is there

Exercise 3: Download

Tips

  • (If you want to create a file in remote terminal: $ touch remote_file_sftp)
  • Send it to an existing local folder
  • Check locally (in sftp session) that it is there
Answer (Tetralith example)

On Server

  • use the get command: > get remote_file_sftp

Check locally (in sftp session) that it is there

(Optional): Exercise 3: Test the difference between transferring one or several files (using SFTP)

Tips

In an SSH session (not SFTP) with REMOTE/server

  • To not interfer with last exercise make a new folder by creating 3000 files REMOTELY in a directory with name many_files
    • $ mkdir many_files
    • $ cd many_files_sftp
    • $ touch file_{1..3000}.txt
    • Check content: $ ls for checking
    • Leave directory to be able to perform next step: $ cd ..
  • Also archive the many_files_sftp folder to many_files_sftp.tar
    • The original directory is still there! Check!

Establish the SFTP session (Exercise 1 in SFT session)

  • Download (to local) the directory and note the time needed (not shown in numbers so count the seconds!)
  • Download (to local) the .tar file and note the time needed
  • Was there a significant difference?
Answer (Example with Tetralith)

Archiving and compressions step REMOTELY

  • tar -cvf many_files_sftp.tar many_files

Establish SFTP connection

  • $ sftp sm_bcarl@tetralith.nsc.liu.se

Download

  • > get -r many_files_sftp (we need the recursive command -r)

  • > get many_files_sftp.tar.gz

  • Video for Tetralith