File transfer using rsync¶
Learning outcomes
- Practice using the documentation of your favorite HPC cluster
- Can transfer files using
rsync
For teachers
Teaching goals are:
- Learners have practiced using the documentation of their favorite HPC cluster
- Learners have transferred files using
rsync
Prior questions:
- How can one do a file transfer from the terminal?
- Can you name some programs that can do file transfer from the terminal?
- Who has heard of
rsync? - Who has already used
rsync?
Notes:
- Bianca users will have the most complex procedure by far
Feedback:
- When would you recommend a colleague to use
rsync, and when not?
Why?¶
Fairly sure you'll want to upload or download files to your favorite HPC cluster. Here we do so.
We use a free and open-source command-line tool to do this,
called rsync.
It works under Linux, Mac and Windows.
Exercises¶
Need a video?
| HPC Cluster | YouTube video |
|---|---|
| Alvis | YouTube video |
| Berzelius | Cannot: is not a NAISS cluster |
| Bianca | YouTube video |
| COSMOS | YouTube video |
| Dardel | YouTube video |
| Kebnekaise | Cannot: is not a NAISS cluster |
| LUMI | YouTube video |
| Pelle | YouTube video |
| Rackham | YouTube video |
| Tetralith | YouTube video |
| Vera | Cannot: is not a NAISS cluster |
How difficult will this be?
This depends on:
- operating system
- your HPC cluster
Per operating system, it differs how easy it is to install rsync:
| Operating system | Difficulty |
|---|---|
| Linux | Easy |
| Mac | Unsure |
| Windows | Unsure |
| HPC Cluster | Difficulty | Reason |
|---|---|---|
| Alvis | Medium | No specialized documentation |
| Berzelius | Easy | Has specialized documentation |
| Bianca | Hard | Has specialized documentation, complex procedure |
| COSMOS | Medium | No specialized documentation |
| Dardel | Medium | No specialized documentation |
| Kebnekaise | Easy | Has specialized documentation |
| LUMI | Medium | CSC refuses to document this, but UPPMAX has specialized documentation |
| Pelle | Easy | Has specialized documentation |
| Rackham | Easy | Has specialized documentation |
| Tetralith | Easy | Has specialized documentation |
| Vera | Easy | Has specialized documentation |
Exercise 1: transfer a file¶
Use the documentation of your HPC cluster.
- For maximally 5 minutes, search for how to transfer files
to/from your HPC cluster
using
rsync. Which URL is it described? Take a look at the answer if you cannot find it: sometimes there is no documentation
Where is that documentation?
| HPC Cluster | Documentation |
|---|---|
| Alvis | Documentation that is closest. For now, use the UPPMAX documentation |
| Berzelius | Documentation |
| Bianca | Documentation |
| COSMOS | None. Documentation that is closest. Use the scp documentation and replace scp by rsync |
| Dardel | Documentation that is closest |
| Kebnekaise | Documentation |
| LUMI | CSC refuses to document this, but UPPMAX has specialized documentation |
| Pelle | Documentation |
| Rackham | Documentation |
| Sigma | Documentation that is closest. For now, use the UPPMAX documentation |
| Tetralith | Documentation |
| Vera | Documentation that is closest. For now, use the UPPMAX documentation |
If the center maintaining you HPC cluster has not documented how to use
rsync,
follow the Rackham documentation.
- Copy a file from your local computer to your HPC cluster's home folder. Verify that this worked.
- Copy a file from your HPC cluster's home folder to your local computer. Verify that this worked.
(optional) Exercise 2: transfer a compressed file¶
One of the biggest bottlenecks for file transfer speed is file size.
In this optional exercise, we'll compress the file before transferring,
and decompress it at its destination. rsync can do this,
by adding the --compress flag, e.g.
-
Download an example file from
examplefile.com, that has the size you are interested in, for example this 100 MB text file -
Let
rsynctransfer the file with and without using the--compressflag. Does the compress and decompress make the operation faster as a whole?
Answer
This depends.
The slower the network/internet connection is, the likelier it is that
--compress will make the transfer faster.