Using isolated environments

Learning objectives

Practice using the documentation of your HPC cluster
Find out which isolated environment tool to use on your HPC cluster
Work (create, activate, work, deactivate) with isolated environments in the way recommended by your HPC cluster
(optional) work (create, activate, work, deactivate) with isolated environments in the other way (if any) possible on your HPC cluster
(optional) export and import a virtual environment

For teachers

Introduction 5 m
venv 5 m
Conda 5
Exercises 30 m
Wrap-up

Why isolated environments are important

Isolated/virtual environments solve a couple of problems:

You can install specific, also older, package versions into them.
You can create one for each project and no problem if the two projects require different versions.
You can remove the environment and create a new one, if not needed or with errors.
Good for reproducibility!

When?

When installed modules are not enough for you.
Examples:
- you continue a project on your computer on the HPC cluster
- you continue someone else’s project on the HPC cluster

Discussion

What experience have you had?

What is an isolated environments

Isolated environments let you create separate workspaces for different versions of Python and/or different versions of packages.
You can activate and deactivate them one at a time, and work as if the other workspace does not exist.

Principles

create environment (choose Python version as well)
activate the environment
- now you work isolated from the rest of the system, e.g. Python packages
install packages
- these are now reached only from the activated project
do your research
deactivate

The tools

Python’s built-in venv module: uses pip
virtualenv (in bundle module or can be installed): uses pip
conda/forge: uses conda and mamba

Tool	Software module	Own Python	Can use python packages from outside
venv	Python-	Uses the one it was activated from	`--system-site-packages`
conda	conda/forge	Yes	No

Other tools perhaps covered in the future

pixi: package management tool for developers
- It allows the developer to install libraries and applications in a reproducible way. Use pixi cross-platform, on Windows, Mac and Linux.
- could replace conda/mamba
uv: An extremely fast Python package and project manager, written in Rust.
- A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more

What happens at activation?

Python version is defined by the environment.
- Check with which python, should show at path to the environment.
Packages are defined by the environment.
- Check with pip list
- conda can only see what you installed for it.
- venv and virtualenv also see other packages if you allowed for that when creating the environment (--system-site-packages).
You can work in a Python shell or IDE (coming session)
You can run scripts dependent on packages now installed in your environment.

Warning

About Conda on HPC systems

Conda is good in many ways but can interact negatively when
- using the python modules (module load) at the same time
- having base environment always active
Not recommended at HPC2N
At the other clusters, handle with care!

HPC cluster	Conda vs venv
Alvis	venv, conda in container
Bianca	conda/latest, venv via wharf
COSMOS	Anaconda3/2024.02-1
Dardel	miniconda3/25.3.1-1-cpeGNU-24.11
Kebnekaise	venv only
LUMI	venv, conda-containerize
Pelle	venv, Miniforge3/24.11.3-0
Tetralith	Anaconda3/2024.02-1
LUMI	conda-containerize

Tip

Try with venv first
If very troublesome, try with conda
To use self-installed Python packages in a batch script, you also need to load the above mentioned modules and activate the environment. An example of this will follow later in the course.
To see which Python packages you have installed in an environment, you can use pip list --local while the environment you have installed the packages in is active. To see all packages, use pip list.
- Note that --user must be omitted: else the package will be installed in the global user folder.

pip list documentation

--local: If in a virtualenv that has global access, do not list globally-installed packages.
--user: Only output packages installed in user-site.
documentation

Virtual environment - venv & virtualenv

With this tool you can download and install with pip from the PyPI repository

venv vs. virtualenv

These are almost completely interchangeable
The difference being that virtualenv supports older python versions and has a few more minor unique features, while venv is in the standard library.
Step 1:
- Virtualenv: virtualenv Example
- venv: python -m venv Example2
Next steps are identical and involves “activating” and pip install
We recommend venv in the course. Then we are just needing the Python module itself!

Tip for Tetralith

load a “bare” python, like Python/3.10.4-bare-hpc1-gcc-2022a-eb
in an environment install setuptools and wheel: pip3 install --upgrade pip setuptools wheel

Example NSC

      ml buildtool-easybuild/4.8.0-hpce082752a2 GCC/13.2.0 Python/3.11.5
      which python
      python -V
      cd /proj/courses-fall-2025/users/<username>
      python -m venv env-matplotlib
      source env-matplotlib/bin/activate
      #note that the prompt has ``(env-matplotlib)``
      #install some extra pip related tools
      #pip3 install --upgrade pip setuptools wheel
      pip install matplotlib
      # do some work
      python

    >>> import matplotlib

When work is done, deactivate the environment with

   deactivate

Conda

Conda is an installer of packages but also bigger toolkits and is useful also for R packages and C/C++ installations.
Conda creates isolated environments not clashing with other installations of python and other versions of packages.
Conda environment requires that you install all packages needed by yourself.
- That is, you cannot load the Python module and use the packages therein inside you Conda environment.

Conda vs mamba etc…

what-is-the-difference-with-conda-mamba-poetry-pip

One can choose from different repos, or channels

Conda channels

bioconda
biocore
conda-forge
dranew
free
~main~ # not available at the HPC-clusters
pro
qiime2
r
r2018.11
scilifelab-lts

You reach them all by loading the conda module. You don't have to state the specific channel when using UPPMAX. Otherwise you do with ``conda -c <channel> ...``

Miniconda and Miniforge are the common software modules at the centres.

Warning

Drawbacks

Conda cannot use already installed packages from the Python modules and libraries, and hence installs them anyway
Conda is therefore known for creating many small files. Your disk space is not only limited in GB, but also in number of files (typically 300000 in $HOME).
Check your disk usage and quota limit
- Do a conda clean -a once in a while to remove unused and unnecessary files

Tip

The conda environments including many small files are by default stored in ~/.conda folder that is in your $HOME directory with limited storage.
Use the commands

export CONDA_ENVS_PATH="path/to/your/project/(subdir)"
export CONDA_PKG_DIRS="path/to/your/project/(subdir)"
mamba create --prefix=$CONDA_ENVS_PATH/<conda env name>

Example NSC

module load Miniforge/24.7.1-2-hpc1
export CONDA_PKG_DIRS=/proj/spring-courses-naiss/users/$USER
export CONDA_ENVS_PATH=/proj/spring-courses-naiss/users/$USER
mamba create --prefix=$CONDA_ENVS_PATH/numpy-proj-39 python=3.9.5 -c conda-forge
mamba activate nsc-example
# A prompt "(/path-to/nsc-example/)" should show up
# double-check we are using python from the Conda environment!
which python  # should point to the conda environment!
python -V     # should give python version 3.9.5
conda install numpy

>>> import numpy

Note, when pinning with conda, use single = instead of double (as used by pip)

Conda base env

When conda is loaded you will by default be in the base environment, which works in the same way as other conda environments.
It includes a Python installation and some core system libraries and dependencies of Conda.
It is a “best practice” to avoid installing additional packages into your base software environment.

Conda cheat sheet

List packages in present environment: conda list
List all environments: conda info -e or conda env list
Install a package: conda install somepackage
Install from certain channel (conda-forge): conda install -c conda-forge somepackage
Install a specific version: conda install somepackage=1.2.3
Create a new environment: conda create --name myenvironment
Create a new environment from requirements.txt: conda create --name myenvironment --file requirements.txt
On e.g. HPC systems where you don’t have write access to central installation directory: conda create --prefix /some/path/to/env
Activate a specific environment: conda activate myenvironment
Deactivate current environment: conda deactivate

Install from file

This is handy when you want to move your Python environment somewhere else.
Also, when giving your code to someone else (in research group or to a community).
Good for reproducibility

Principle for both venv & conda

Activate the environment and make sure it works
Save a environment file
Distribute the file
Receiver installs the environment from the file

venv

Make a requirements file:

pip freeze --local > requirements.txt

How does it look like?

   contourpy==1.3.3
   cycler==0.12.1
   fonttools==4.60.1
   kiwisolver==1.4.9
   matplotlib==3.10.7
   numpy==2.3.5
   pillow==12.0.0
   pyparsing==3.2.5
   python-dateutil==2.9.0.post0
   six==1.17.0

Install packages from a file

   pip install -r requirements.txt

conda/forge

Make environment file:

    conda env export > environment.yml

How does it look like?

   name: /lunarc/nobackup/projects/lu2025-17-52/bjornc/example
   channels:
     - conda-forge
   dependencies:
     - _libgcc_mutex=0.1=conda_forge
     - _openmp_mutex=4.5=2_gnu
     - alsa-lib=1.2.14=hb9d3cd8_0
     - brotli=1.2.0=hed03a55_1
     - brotli-bin=1.2.0=hb03c661_1
     - bzip2=1.0.8=hda65f42_8
     - ca-certificates=2025.11.12=hbd8a1cb_0
     - cairo=1.18.4=h3394656_0
     - contourpy=1.3.3=py312hd9148b4_3
     - cycler=0.12.1=pyhd8ed1ab_1
     - cyrus-sasl=2.1.28=hd9c7081_0
     - dbus=1.16.2=h3c4dab8_0
     - double-conversion=3.3.1=h5888daf_0
     - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
     - font-ttf-inconsolata=3.000=h77eed37_0
     - font-ttf-source-code-pro=2.038=h77eed37_0
     - font-ttf-ubuntu=0.83=h77eed37_3
     - fontconfig=2.15.0=h7e30c49_1
     - fonts-conda-ecosystem=1=0
     - fonts-conda-forge=1=hc364b38_1
     - fonttools=4.60.1=py312h8a5da7c_0
     - freetype=2.14.1=ha770c72_0
     - graphite2=1.3.14=hecca717_2
     - harfbuzz=12.2.0=h15599e2_0
     - icu=75.1=he02047a_0
     - keyutils=1.6.3=hb9d3cd8_0
     - kiwisolver=1.4.9=py312h0a2e395_2
     - krb5=1.21.3=h659f571_0
     - lcms2=2.17=h717163a_0
     - ld_impl_linux-64=2.45=default_hbd61a6d_104
     - lerc=4.0.0=h0aef613_1
     - libblas=3.11.0=2_h4a7cf45_openblas
     ...
     this was just like 30%

Create an environment from a file. Do this on another computer or rename.

   conda env create -f environment.yml
   # if renaming is necessary
   conda env create -f environment.yml -p /path/renamed

Discussion

Did you note the difference in file length for venv & conda?
What does it mean?

Summary

Keypoints

With a virtual environment you can tailor an environment with specific versions for Python and packages, not interfering with other installed python versions and packages.
Make it for each project you have for reproducibility.
There are different tools to create virtual environments.
- venv, most straight-forward and available at all HPC centers. Recommended
  - only supports Python packages
- conda, only recommended at some clusters
  - supports more and is a bit more reliable
  - do not use together with Python modules
  - install in project folder due to many files.

Summary of the workflows

Venv
Conda

Documentation at the centres

NSC:

PDC:

LUNARC

https://lunarc-documentation.readthedocs.io/en/latest/guides/applications/Python/#anaconda-distributions

UPPMAX

HPC2N

https://docs.hpc2n.umu.se/software/userinstalls/#venv

LUMI

https://docs.lumi-supercomputer.eu/software/installing/container-wrapper/#examples-of-using-the-lumi-container-wrapper>

Using isolated environments

Why isolated environments are important

What is an isolated environments

What happens at activation?

Virtual environment - venv & virtualenv

Example NSC

Conda

Example NSC

Install from file

venv

conda/forge

Exercises

Summary

Summary of the workflows