Use isolated environments

!!! info “Learning objectives”

• Practice using the documentation of your HPC cluster

• Find out which isolated environment tool to use on your HPC cluster

• Work (create, activate, work, deactivate) with isolated environments in the way recommended by your HPC cluster

• (optional) work (create, activate, work, deactivate) with isolated environments in the other way (if any) possible on your HPC cluster

• (optional) export and import a virtual environment

<!–

HPC cluster| Who creates answer ———–|———————————————- Alvis | Find out which, BC takes Conda, RB takes venv Bianca | conda, hence BC COSMOS | Find out which, BC takes Conda, RB takes venv Dardel | Find out which, BC takes Conda, RB takes venv Kebnekaise | venv, hence RB LUMI | RB Rackham | venv, hence RB Tetralith | Find out which, BC takes Conda, RB takes venv

–>

HPC cluster	Conda vs venv
Alvis	venv, conda in container
Bianca	conda/latest, venv via wharf
COSMOS	Anaconda3/2024.02-1
Dardel	miniconda3/24.7.1-0-cpeGNU-23.12
Kebnekaise	venv
LUMI	?
Rackham	venv, conda/latest
Tetralith	Anaconda3/2024.02-1

<!– Exercise 1: which system to pick –>

<!– Exercise 2:

• Confirm package is absent

• Create environment

• Activate environment

• Confirm package is absent

• Install package in isolated environment

• Confirm package is now present

• Deactivate environment

• Confirm package is now absent again

–>

<!– (optional) Exercise 3: like 2, but for other tool –>

<!– (optional) Exercise 4:

?export and import a virtual environment

–>

<!– below is still old –>

Objectives

Learners can

• work (create, activate, work, deactivate) with virtual environments

• install a python package

• export and import a virtual environment

Isolated environments

As an example, maybe you have been using TensorFlow 1.x.x for your project and now you need to install a package that requires TensorFlow 2.x.x but you will still be needing the old version of TensorFlow for another package, for instance. This is easily solved with isolated environments.

Note

Isolated/virtual environments solve a couple of problems:

• You can install specific, also older, versions into them.

• You can create one for each project and no problem if the two projects require different versions.

• You can remove the environment and create a new one, if not needed or with errors.

• Isolated environments lets you create separate workspaces for different versions of Python and/or different versions of packages.

• You can activate and deactivate them one at a time, and work as if the other workspace does not exist.

The tools

• venv UPPMAX+HPC2N+LUNARC+NSC

• virtualenv UPPMAX+HPC2N+LUNARC+NSC

• Conda LUNARC + UPPMAX (recommended only for Bianca cluster)

Warning

About Conda on HPC systems

• Conda is good in many ways but can interact negatively when trying to use the pytrhon modules in the HPC systems.

• LUNARC seems to have working solutions

• At UPPMAX Conda is installed but we have many users that get into problems.

  • However, on Bianca this is the most straight-forward way to install packages (no ordinary internet)

Conda in HPC

• Anaconda at LUNARC

• Conda at UPPMAX

  • Conda on Bianca

Virtual environment - venv & virtualenv

Workflow

1. You load the Python module you will be using, as well as any site-installed package modules (requires the --system-site-packages option later)

2. You create the isolated environment with something like venv, virtualenv (use the --system-site-packages to include all “non-base” packages)

3. You activate the environment

4. You install (or update) the environment with the packages you need

5. You work in the isolated environment

6. You deactivate the environment after use

venv vs. virtualenv

• These are almost completely interchangeable

• The difference being that virtualenv supports older python versions and has a few more minor unique features, while venv is in the standard library.

• Step 1:

  • Virtualenv: virtualenv --system-site-packages Example

  • venv: python -m venv --system-site-packages Example2

• Next steps are identical and involves “activating” and pip installs

• We recommend venv in the course. Then we are just needing the Python module itself!

Keypoints

• With a virtual environment you can tailor an environment with specific versions for Python and packages, not interfering with other installed python versions and packages.

• Make it for each project you have for reproducibility.

• There are different tools to create virtual environments.

  • conda, only recommended for personal use and at some clusters

  • virtualenv, may require to load extra python bundle modules.

  • venv, most straight-forward and available at all HPC centers. Recommended

• More details to follow!

Example

Tip

Do not type along!

Create a venv. First load the python version you want to base your virtual environment on:

UPPMAXHPC2NLUNARCNSC

$ module load python/3.11.8
$ python -m venv --system-site-packages Example2

“Example2” is the name of the virtual environment. The directory “Example2” is created in the present working directory. The -m flag makes sure that you use the libraries from the python version you are using.

Note

To save space, you should load any other Python modules you will need that are system installed before installing your own packages! Remember to choose ones that are compatible with the Python version you picked! --system-site-packages includes the packages already installed in the loaded python module.

At HPC2N, NSC and LUNARC, you often have to load SciPy-bundle. This is how you could create a venv (Example3) with a SciPy-bundle included which is compatible with Python/3.11.3:

$ module load GCC/12.3.0 Python/3.11.3 SciPy-bundle/2023.07 # for HPC2N and LUNAR
$ module load buildtool-easybuild/4.8.0-hpce082752a2 GCC/13.2.0 Python/3.11.5 SciPy-bundle/2023.11 # for NSC
$ python -m venv --system-site-packages Example3

NOTE: since it may take up a bit of space if you are installing many Python packages to your virtual environment, we strongly recommend you place it in your project storage!

NOTE: if you need to for instance working with both Python 2 and 3, then you can of course create more than one virtual environment, just name them so you can easily remember which one has what.

If you want your virtual environment in a certain place…

• Example for course project location and $USER being you user name.

  • If your directory in the project has another name, replace $USER with that one!

• UPPMAX:

  • Create: python -m venv /proj/hpc-python-fall/$USER/Example

  • Activate: source /proj/hpc-python-fall/<user-dir>/Example/bin/activate

• HPC2N:

  • Create: python -m venv /proj/nobackup/hpc-python-fall-hpc2n/$USER/Example

  • Activate: source /proj/nobackup/hpc-python-fall-hpc2n/<user-dir>/Example/bin/activate

• LUNARC:

  • Create: python -m venv /lunarc/nobackup/projects/lu2024-17-44/$USER/Example

  • Activate: source /lunarc/nobackup/projects/lu2024-17-44/<user-dir>/Example/bin/activate

• NSC:

  • Create: python -m venv /proj/hpc-python-fall-nsc/$USER/Example

  • Activate: source /proj/hpc-python-fall-nsc/<user-dir>/Example/bin/activate

Note that your prompt is changing to start with (Example) to show that you are within an environment.

Note

• source can most often be replaced by ., like in . Example/bin/activate. Note the important <space> after .

• For clarity we use the source style here.

Install packages to the virtual environment with pip

Tip

Do not type along!

Install your packages with pip. While not always needed, it is often a good idea to give the correct versions you want, to ensure compatibility with other packages you use. This example assumes your venv is activated:

(Example) $ pip install --no-cache-dir --no-build-isolation numpy matplotlib

The --no-cache-dir" option is required to avoid it from reusing earlier installations from the same user in a different environment. The --no-build-isolation is to make sure that it uses the loaded modules from the module system when building any Cython libraries.

Deactivate the venv.

(Example) $ deactivate

Everytime you need the tools available in the virtual environment you activate it as above (after also loading the modules).

source /proj/<your-project-id>/<your-dir>/Example/bin/activate

Note

• You can use “pip list” on the command line (after loading the python module) to see which packages are available and which versions.

• Some packaegs may be inhereted from the moduels yopu have loaded

• You can do pip list --local to see what is instaleld by you in the environment.

• Some IDE:s like Spyder may only find those “local” packages

Working with virtual environments defined from files

Creator/developer

• First _create_ and _activate_ an environment (see above)

• Install packages with pip

• Create file from present virtual environment:

$ pip freeze > requirements.txt

• That includes also the system site packages if you included them with --system-site-packages

• Test that everything works by running use cases scripts within the environment

• You can list packages specific for the virtualenv by pip list --local

• So, creating a file from just the local environment:

$ pip freeze --local > requirements.txt

Note

requirements.txt (used by the virtual environment) is a simple text file which looks similar to this:

numpy
matplotlib
pandas
scipy

requirements.txt with versions that could look like this:

numpy==1.20.2
matplotlib==3.2.2
pandas==1.1.2
scipy==1.6.2

• Deactivate

User

• Create an environment based on dependencies given in an environment file

• This can be done in new virtual environment or as a genera installtion locally (not activating any environment

pip install -r requirements.txt

• Check

pip list

More on dependencies

• Dependency management from course Python for Scientific computing

Python packages in HPC and ML

It is difficult to give an exhaustive list of useful packages for Python in HPC, but this list contains some of the more popular ones:

Popular packages

Package	Module to load, UPPMAX	Module to load, HPC2N	Brief description
Dask	python	dask	An open-source Python library for parallel computing.
Keras	python_ML_packages	Keras	An open-source library that provides a Python interface for artificial neural networks. Keras acts as an interface for both the TensorFlow and the Theano libraries.
Matplotlib	python or matplotlib	matplotlib	A plotting library for the Python programming language and its numerical mathematics extension NumPy.
Mpi4Py	Not installed	SciPy-bundle	MPI for Python package. The library provides Python bindings for the Message Passing Interface (MPI) standard.
Numba	python	numba	An Open Source NumPy-aware JIT optimizing compiler for Python. It translates a subset of Python and NumPy into fast machine code using LLVM. It offers a range of options for parallelising Python code for CPUs and GPUs.
NumPy	python	SciPy-bundle	A library that adds support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.
Pandas	python	SciPy-bundle	Built on top of NumPy. Responsible for preparing high-level data sets for machine learning and training.
PyTorch/Torch	PyTorch or python_ML_packages	PyTorch	PyTorch is an ML library based on the C programming language framework, Torch. Mainly used for natural language processing or computer vision.
SciPy	python	SciPy-bundle	Open-source library for data science. Extensively used for scientific and technical computations, because it extends NumPy (data manipulation, visualization, image processing, differential equations solver).
Seaborn	python	Not installed	Based on Matplotlib, but features Pandas’ data structures. Often used in ML because it can generate plots of learning data.
Sklearn/SciKit-Learn	scikit-learn	scikit-learn	Built on NumPy and SciPy. Supports most of the classic supervised and unsupervised learning algorithms, and it can also be used for data mining, modeling, and analysis.
StarPU	Not installed	StarPU	A task programming library for hybrid architectures. C/C++/Fortran/Python API, or OpenMP pragmas.
TensorFlow	TensorFlow	TensorFlow	Used in both DL and ML. Specializes in differentiable programming, meaning it can automatically compute a function’s derivatives within high-level language.
Theano	Not installed	Theano	For numerical computation designed for DL and ML applications. It allows users to define, optimise, and gauge mathematical expressions, which includes multi-dimensional arrays.

Remember, in order to find out how to load one of the modules, which prerequisites needs to be loaded, as well as which versions are available, use module spider <module> and module spider <module>/<version>.

Often, you also need to load a python module, except in the cases where it is included in python or python_ML_packages at UPPMAX or with SciPy-bundle at HPC2N.

NOTE that not all versions of Python will have all the above packages installed!

Summary of workflow

In addition to loading Python, you will also often need to load site-installed modules for Python packages, or use own-installed Python packages. The work-flow would be something like this:

1. Load Python and prerequisites: module load <pre-reqs> Python/<version>

2. Load site-installed Python packages (optional): module load <pre-reqs> <python-package>/<version>

3. Create the virtual environment: python -m venv [PATH]/Example

4. Activate your virtual environment: source <path-to-virt-env>/Example/bin/activate

5. Install any extra Python packages: pip install --no-cache-dir --no-build-isolation <python-package>

6. Start Python or run python script: python

7. Do your work

8. Deactivate

• Installed Python modules (modules and own-installed) can be accessed within Python with import <package> as usual.

• The command pip list given within Python will list the available modules to import.

• More about packages and virtual/isolated environment to follow in later sections of the course!

Exercises

1. Make a test environment

1. make a virtual environment with the name venv1. Do not include packages from the the loaded module(s)

2. activate

3. install matplotlib

4. make a requirements file of the content

5. deactivate

6. make another virtual environment with the name venv2

7. activate that

8. install with the aid of the requirements file

9. check the content

10. open python shell from command line and try to import

11. exit python

12. deactivate

Solution

• First load the required Python module(s) if not already done so in earlier lessons. Remember that this steps differ between the HPC centers

1. make the first environment

$ python -m venv venv1

2. Activate it.

$ source venv1/bin/activate

- Note that your prompt is changing to start with ``(venv1)`` to show that you are within an environment.

3. install matplotlib

pip install matplotlib

4. make a requirements file of the content

pip freeze --local > requirements.txt

5. deactivate

deactivate

6. make another virtual environment with the name venv2

python -m venv venv2

7. activate that

source venv2/bin/activate

8. install with the aid of the requirements file

pip install -r requirements.txt

9. check the content

pip list

10. open python shell from command line and try to import

python

import matplotlib

11. exit python

exit()

12. deactivate

deactivate

Prepare fore the course environments

Note

• All centers has had different approaches in what is included in the module system and not.

• Therefore the solution to complete the necessary packages needed for the course lessons, different approaches has to be made.

• This is left as exercise for you

We will need to install the LightGBM Python package for one of the examples in the ML section.

Tip

Follow the track where you are working right now

Create a virtual environment called vpyenv. First load the python version you want to base your virtual environment on, as well as the site-installed ML packages.

NSCLUNARCUPPMAXHPC2N

If you do not have matplotlib already outside any virtual environment

• Install matplotlib in your .local folder, not in a virtual environment.

• Do:

ml buildtool-easybuild/4.8.0-hpce082752a2 GCC/13.2.0 Python/3.11.5
pip install --user matplotlib

• Check that matplotlib is there by pip list

Check were to find environments needed for the lessons in the afternoon tomorrow

• browse /proj/hpc-python-fall-nsc/ to see the available environments.

• their names are

  • venvNSC-TF

  • venvNSC-torch

  • venvNSC-numba

  • venv-spyder-only

Note

• To use self-installed Python packages in a batch script, you also need to load the above mentioned modules and activate the environment. An example of this will follow later in the course.

• To see which Python packages you, yourself, have installed, you can use pip list --user while the environment you have installed the packages in are active. To see all packages, use pip list.

See also

• UPPMAX’s documentation pages about installing Python packages and virtual environments: http://docs.uppmax.uu.se/software/python/#installing-python-packages

• HPC2N’s documentation pages about installing Python packages and virtual environments: https://www.hpc2n.umu.se/resources/software/user_installed/python

Keypoints

• With a virtual environment you can tailor an environment with specific versions for Python and packages, not interfering with other installed python versions and packages.

• Make it for each project you have for reproducibility.

• There are different tools to create virtual environemnts.

  • UPPMAX has conda and venv and virtualenv

  • HPC2N has venv and virtualenv