Isolated environments

  • remember there are multiple virtual environment managers

  • practice to create, activate, work in and deactive a virtual environment

  • practice to export and import a virtual environment

Compute allocations in this workshop

  • Rackham: naiss2024-22-107

  • Kebnekaise: hpc2n2024-025

Storage space for this workshop

  • Rackham: /proj/r-py-jl

  • Kebnekaise: /proj/nobackup/hpc2n2024-025

Introduction

Different experiments may need different versions of Python and/or Python packages. Virtual environments allow one to work with multiple sets of (potentially incompatible) packages, where each set is independent and isolated.

Additionally, you may want to have a reproducible computational environment, so that others can reproduce your computational experiments. Virtual environments can be exported and imported to provide for better reproducible computational environments.

In this session, we create, activate, use, deactivate, export and import some virtual environments.

Virtual environment managers

flowchart TD python[Python] package_versions[combination of package versions] virtual_environment_manager[virtual environment manager] create_isolated_environments[create isolated environments] venv[venv\nvirtualenv] conda[conda] uppmax[UPPMAX] hpc2n[HPC2N] python -->|has| package_versions package_versions -->|managed by|virtual_environment_manager virtual_environment_manager --> |has goal|create_isolated_environments package_versions -.- create_isolated_environments virtual_environment_manager --> |among others|conda virtual_environment_manager --> |among others|venv conda -->|works on|uppmax venv -->|works on|uppmax venv -->|works on|hpc2n

In this course, we will look at the following environment managers:

Manager

HPC2N

UPPMAX

Scope

conda

Avoid

Recommended

Language agnostic

venv

Recommended

OK

Python only

Both centers have their documentation on virtual environment managers:

In this session, we use venv, as it works for both UPPMAX and HPC2N.

General workflow

flowchart TD create[Create] activate[Activate] use[Use] deactivate[Deactivate] create --> activate activate --> use use --> deactivate deactivate --> activate

Whatever environment manager you use, this is the workflow:

  • You create the isolated environment

  • You activate the environment

  • You work in the isolated environment. Here you install (or update) the environment with the packages you need

  • You deactivate the environment after use

A virtual environment can be created in multiple ways, for example, from scratch. However, there are more efficient ways, which we will use.

Exercises

In these exercises, we create multiple virtual environments.

For exercises 2 and 3, there is considerate time needed to install all the Python packages:

Exercise

Virtual environment name

Time to install packages (minutes)

1

vpyenv

1

2

Example-gpu

6

3

analysis

13

Tip for UPPMAX users: do exercise 2 and 3 in parallel by logging in twice.

Exercise 0: remove the Python packages installed in the home folder

In the previous session, we have installed Python packages in the home folder. This will interfere with our virtual environments.

To make sure your virtual environments work, ruthlessly delete the Python packages in your home folder:

rm -Ir ~/.local/lib/python3.11

You will be asked to confirm.

Exercise 1: work with vpyenv

  • Create a Python virtual environment from a step-by-step instruction

In this exercise, we create the course environment vpyenv in a step-by-step fashion:

flowchart TD load_modules[1. Load modules] create[2. Create] activate[3. Activate] install_libraries[4. Install Python libraries] check[5. Check installed Python libraries] use[6. Use] deactivate[7. Deactivate] load_modules --> create create --> activate activate --> install_libraries install_libraries --> check check --> use use --> deactivate deactivate --> activate

We create the virtual environment needed for this course, called vpyenv. As virtual environments can take up a lot of disc space, we create it in the course project folder.

Exercise 1.1: load the modules needed

module load python/3.11.8

Exercise 1.2: create the virtual environment

Create the virtual environment called vpyenv as such:

$ python -m venv --system-site-packages /proj/r-py-jl/[username]/python/vpyenv

where [username] is your UPPMAX username, for example python -m venv --system-site-packages /proj/r-py-jl/sven/python/vpyenv.

Exercise 1.3: activate the virtual environment

Activate the virtual environment called vpyenv as such:

source /proj/r-py-jl/[username]/python/vpyenv/bin/activate

where [username] is your UPPMAX username, for example python -m venv --system-site-packages /proj/r-py-jl/sven/python/vpyenv.

Exercise 1.4: install Python packages

Install the spacy and seaborn packages

pip install --user spacy seaborn

Exercise 1.5: check if the Python packages are installed

To see all installed Python packages:

pip list

To see which Python packages you have installed yourself (i.e. not loaded from a module), use:

pip list --user

Exercise 1.6: use the virtual environment

Start Python and import a Python package:

(vpyenv) $ python
Python 3.11.8 (main, Feb  8 2024, 11:48:52) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
>>>

Exercise 1.7: deactivate the virtual environment

deactivate

Well done, you’ve just created a virtual environment called vpyenv!

Exercise 2: work with Example-gpu

  • Rehearse creating a Python virtual environment from a step-by-step instruction

This exercise if for UPPMAX users only.

In this exercise, we create another environment Example-gpu in the same step-by-step fashion as done in exercise 1:

flowchart TD load_modules[1. Load modules] create[2. Create] activate[3. Activate] install_libraries[4. Install Python libraries] check[5. Check installed Python libraries] use[6. Use] deactivate[7. Deactivate] load_modules --> create create --> activate activate --> install_libraries install_libraries --> check check --> use use --> deactivate deactivate --> activate

This virtual environment called Example-gpu is used for examples where the use of GPUs is demonstrated, by using the numba and PyTorch Python packages.

Because the structure is the same as Exercise 1, see exercise 1 for details.

Exercise 2.1: load the modules needed

Here we need an older Python module, as that is what available on the Snowy computer cluster:

module load python/3.9.5

Exercise 2.2: create the virtual environment

python -m venv --system-site-packages /proj/r-py-jl/<user>/python/Example-gpu

where [username] is your UPPMAX username, for example python -m venv --system-site-packages /proj/r-py-jl/sven/python/Example-gpu.

Exercise 2.3: activate the virtual environment

source /proj/r-py-jl/<user>/python/Example-gpu/bin/activate

where [username] is your UPPMAX username, for example source /proj/r-py-jl/sven/python/Example-gpu/bin/activate.

Exercise 2.4: install Python packages

Installing these packages takes around 6 minutes.

pip install --upgrade numpy scipy numba torch

Exercise 2.5: check if the Python packages are installed

pip list

Exercise 2.6: use the virtual environment

Not now :-)

Exercise 2.7: deactivate the virtual environment

deactivate

Exercise 3: export and import a virtual environment

  • Rehearse creating a Python virtual environment

  • Export a virtual environment from a step-by-step instruction

  • Import a virtual environment from a step-by-step instruction

In this exercise, we export and import a virtual environment. Additionally, you get to rehearse to create a virtual environment: as we have done this earlier, the answers will now be hidden :-)

flowchart TD load_modules[1. Load modules] create[2. Create] activate[3. Activate] install_libraries[4. Install Python libraries] check[5. Check installed Python libraries] use[6. Use] deactivate[7. Deactivate] load_modules --> create create --> activate activate --> install_libraries install_libraries --> check check --> use use --> deactivate deactivate --> activate

Exercise 3.1: load the modules needed

Load the modules for Python 3.11.x.

Exercise 3.2: create the virtual environment

Create a virtual environment with the name analysis.

Exercise 3.3: activate the virtual environment

Activate the virtual environment.

Exercise 3.4: install Python packages

Installing these packages takes around 13 minutes.

Create a file called requirements.txt, with the following content:

numpy==1.22.3
matplotlib==3.5.2
pandas==1.4.2

Install packages by using the requirements.txt file:

pip install -r requirements.txt

Exercise 3.5: check if the Python packages are installed

Check that the packages were installed.

Exercise 3.6: use the virtual environment

Export the Python packages our virtual environment uses:

pip freeze > requirements.txt

View the file requirements.txt.

Exercise 3.7: deactivate the virtual environment

Deactivate the virtual environment

Conclusion

Keypoints

You have:

  • heard that virtual environments allows one for independent and isolated set of Python packages

  • heard that there are multiple virtual environments managers:

    • UPPMAX: Conda and venv

    • HPC2N has venv

  • created, activated, used and deactivated virtual environments

  • installed Python packages by using a requirements.txt file

  • exported the Python packages of a virtual environment

You may:

  • consider to create a virtual environment per project, to provide for better reproducibility