Using packages

Learning outcomes

Practice using the documentation of your HPC cluster
Can find and load a Python package module
Can determine if a Python package is installed

Why Python packages are important

Python packages are pieces of tested Python code. Prefer using a Python package over writing your own code.

Why software modules are important

Software modules allows users of any HPC cluster to activate their favorite software of any version. This helps to assure reproducible research.

How to see which Python packages are installed

There are two ways to determine which Python packages are installed:

Where	Command to run	The package is present when …
On the command-line	`pip list`	It shows up in the list
In the Python interpreter	`import [package_name]`, e.g. `import scipy`	There is no error

Exercises

Want to see the answers as a video?

Some HPC clusters have multiple remote desktops. We recommend:

HPC cluster	YouTube video
Alvis	Here
Bianca	Here
COSMOS	Here
Dardel	Here
Kebnekaise	Here
LUMI	Here
Pelle	Here
Rackham	Here
Tetralith	Here

Exercise 1: using Python packages

login to your HPC cluster

Forgot how to do this?

Answer can be found at day 1

load the Python module of the version below

HPC cluster	Python version
Alvis	`3.12.3`
Bianca	`3.12.7`
COSMOS	`3.11.5`
Dardel	`3.11.4`
Kebnekaise	`3.11.3`
LUMI	`3.11.7`
Pelle	`3.12.3`
Tetralith	`3.11.5` (bare)

Forgot how to do this?

HPC cluster	Python version
Alvis	`module load Python/3.12.3-GCCcore-13.3.0`
Bianca	`module load python/3.12.7`
COSMOS	`module load GCCcore/13.2.0 Python/3.11.5`
Dardel	`module load bioinfo-tools python/3.11.4`
Kebnekaise	`module load GCC/12.3.0 Python/3.11.3`
LUMI	`module load cray-python/3.11.7`
Pelle	`module load Python/3.12.3-GCCcore-13.3.0`
Tetralith	`module load Python/3.11.5-bare-hpc1-gcc-2023b-eb`

Confirm that the Python package, indicated in the table below, is absent. You can use any way to do so.

HPC cluster	Python package
Alvis	`scipy`
Bianca	`tensorflow` (CPU version)
COSMOS	`scipy`
Dardel	`tensorflow`
Kebnekaise	`scipy`
LUMI	`matplotlib`
Pelle	`torch`
Tetralith	`scipy`

Answer

From the terminal, use the command below to confirm that the package is not available yet:

HPC cluster	Command
Alvis	`pip list`
Bianca	`pip list`
COSMOS	`pip list`
Dardel	`pip list`
Kebnekaise	`pip list`
LUMI	`pip list`
Pelle	`pip list`
Tetralith	`pip list`

In all cases, the package is not yet installed, as that is what we’ll be doing next :-)

Find the software module to load the package. Use either the documentation of the HPC center, or use the module system

Answer: where is this documented?

HPC cluster	URL to documentation
Alvis	Here
Bianca	Here
COSMOS	Here
Dardel	Here, but it is irrelevant
Kebnekaise	Here
LUMI	Has no software modules
LUMI	Use the thanard/matplotlib container
Pelle	Python bundles
Tetralith	Here

Answer: how to use the module system?

In the terminal, type the command as shown below to get a decent hint. There are many possible terms to use with module spider: whatever works for you is good too :-)

HPC cluster	Command
Alvis	`module spider SciPy`
Bianca	`module spider <package>`
COSMOS	`module spider SciPy`
Dardel	`module spider <package>`
Kebnekaise	`module spider SciPy`
LUMI	Has no module system, use a container instead.
LUMI	`singularity pull docker://thanard/matplotlib`
Pelle	`module spider torch`
Tetralith	`module spider Python`

Load the software module

Answer

In the terminal, type the following command:

HPC cluster	Command
Alvis	`module load SciPy-bundle/2024.05-gfbf-2024a`
Bianca	`module load python_ML_packages/3.9.5-cpu`. You will be asked to do a `module unload python` first. Do so :-)
COSMOS	`module load module load GCC/13.3.0 SciPy-bundle/2024.05`
Dardel	`module load python_ML_packages/3.11.8-cpu`. You will be asked to do a `module unload python` first. Do so :-)
Kebnekaise	`module load GCC/13.3.0 SciPy-bundle/2024.05`
LUMI	Not applicable: we are using a container
Pelle	`module load PyTorch/2.6.0-foss-2024a`
Tetralith	`module load buildtool-easybuild/4.9.4-hpc71cbb0050 GCC/13.2.0 SciPy-bundle/2023.11`
Tetralith	Alternatively: `module load Python/3.11.5` (which happens to be a Python version with `scipy` installed)

See the package is now present

Answer

From the terminal, use the command below to confirm that the package is now available:

HPC cluster	Command
Alvis	`pip list`
Bianca	`pip list`
COSMOS	`pip list`
Dardel	`pip list`
Kebnekaise	`pip list`
LUMI	`./matplotlib pip list`
Pelle	`pip list`
Tetralith	`pip list`

In all cases, the package is now installed. Well done!

Done?

When done, and if you haven’t done so yet, do Use the tarball with exercises.

After that, read what the next session is about.

You can easily navigate there by pressing the ‘Next’ arrow at the bottom of this page, at the right-hand side