Using packages

Learning outcomes

Practice using the documentation of your HPC cluster
Can find and load a Python package module
Can determine if a Python package is installed

Why Python packages are important

Python packages are pieces of tested Python code. Prefer using a Python package over writing your own code.

Why software modules are important

Software modules allows users of any HPC cluster to activate their favorite software of any version. This helps to assure reproducible research.

Exercises

Want to see the answers as a video?

Some HPC clusters have multiple remote desktops. We recommend:

HPC cluster	YouTube video
Alvis	Here
Bianca	Here
COSMOS	Here
Dardel	Here
Kebnekaise	Here
LUMI	Here
Rackham	Here
Tetralith	Here

Exercise 0: pick a Zoom room

We will use breakout rooms, as this is best for learning. Ideally, each breakout room has 2 learners for the same HPC cluster. As it cannot be predicted how many learners show up per cluster, we do this dynamically:

Go to the main breakout room of your HPC cluster

HPC cluster name	Main breakout room
Alvis	Room 1
Bianca	Room 2
COSMOS	Room 3
Dardel	Room 4
Kebnekaise	Room 5
LUMI	Room 6
Rackham	Room 7
Tetralith	Room 8

When the main breakout room reaches 4 learners, decide upon 2 to move to a free breakout room. Repeat until the group size is below 4.
When you are the only one in your main breakout room, the teachers will let you decide to either work in silence or to join another group. You will be taken care of well :-)

Exercise 1: using Python packages

login to your HPC cluster

Forgot how to do this?

Answer can be found at day 1

load the Python module of the version below

HPC cluster	Python version
Alvis	`3.12.3`
Bianca	`3.12.7`
COSMOS	`3.11.5`
Dardel	`3.11.4`
Kebnekaise	`3.11.3`
LUMI	`3.11.7`
Rackham	`3.12.7`
Tetralith	`3.11.5` (bare)

Forgot how to do this?

HPC cluster	Python version
Alvis	`module load Python/3.12.3-GCCcore-13.3.0`
Bianca	`module load python/3.12.7`
COSMOS	`module load GCCcore/13.2.0 Python/3.11.5`
Dardel	`module load bioinfo-tools python/3.11.4`
Kebnekaise	`module load GCC/12.3.0 Python/3.11.3`
LUMI	`module load cray-python/3.11.7`
Rackham	`module load python/3.12.7`
Tetralith	`module load Python/3.11.5-bare-hpc1-gcc-2023b-eb`

Confirm that the Python package, indicated in the table below, is absent. You can use any way to do so.

HPC cluster	Python package
Alvis	`scipy`
Bianca	`tensorflow` (CPU version)
COSMOS	`scipy`
Dardel	`tensorflow`
Kebnekaise	`scipy`
LUMI	`matplotlib`
Rackham	`tensorflow` (CPU version)
Tetralith	`scipy`

Answer

From the terminal, use the command below to confirm that the package is not available yet:

HPC cluster	Command
Alvis	`pip list`
Bianca	`pip list`
COSMOS	`pip list`
Dardel	`pip list`
Kebnekaise	`pip list`
LUMI	`pip list`
Rackham	`pip list`
Tetralith	`pip list`

In all cases, the package is not yet installed, as that is what we’ll be doing next :-)

Find the software module to load the package. Use either the documentation of the HPC center, or use the module system

Answer: where is this documented?

HPC cluster	URL to documentation
Alvis	Here
Bianca	Here
COSMOS	Here
Dardel	Here, but it is irrelevant
Kebnekaise	Here
LUMI	Has no software modules
LUMI	Use the thanard/matplotlib container
Rackham	Here
Tetralith	Here

Answer: how to use the module system?

In the terminal, type the command as shown below to get a decent hint. There are many possible terms to use with module spider: whatever works for you is good too :-)

HPC cluster	Command
Alvis	`module spider SciPy`
Bianca	`module spider packages`
COSMOS	`module spider SciPy`
Dardel	`module spider package`
Kebnekaise	`module spider SciPy`
LUMI	Has no module system, use a container instead.
LUMI	`singularity pull docker://thanard/matplotlib`
Rackham	`module spider packages`
Tetralith	`module spider Python`

Load the software module

Answer

In the terminal, type the following command:

HPC cluster	Command
Alvis	`module load SciPy-bundle/2024.05-gfbf-2024a`
Bianca	`module load python_ML_packages/3.9.5-cpu`. You will be asked to do a `module unload python` first. Do so :-)
COSMOS	`module load module load GCC/13.3.0 SciPy-bundle/2024.05`
Dardel	`module load python_ML_packages/3.11.8-cpu`. You will be asked to do a `module unload python` first. Do so :-)
Kebnekaise	`module load GCC/13.3.0 SciPy-bundle/2024.05`
LUMI	Not applicable: we are using a container
Rackham	`module load python_ML_packages/3.11.8-cpu`. You will be asked to do a `module unload python` first. Do so :-)
Tetralith	`module load buildtool-easybuild/4.9.4-hpc71cbb0050 GCC/13.2.0 SciPy-bundle/2023.11`
Tetralith	Alternatively: `module load Python/3.11.5` (which happens to be a Python version with `scipy` installed)

See the package is now present

Answer

From the terminal, use the command below to confirm that the package is now available:

HPC cluster	Command
Alvis	`pip list`
Bianca	`pip list`
COSMOS	`pip list`
Dardel	`pip list`
Kebnekaise	`pip list`
LUMI	`./matplotlib pip list`
Rackham	`pip list`
Tetralith	`pip list`

In all cases, the package is now installed. Well done!

Done?

When done, and if you haven’t done so yet, do Use the tarball with exercises.

After that, read what the next session is about.

You can easily navigate there by pressing the ‘Next’ arrow at the bottom of this page, at the right-hand side