Skip to content

Package and software installation on Bianca

Learning Objectives

Learners

  • understand principles how to install software and packages yourself
  • can install packages of one or several types
    • Python packages using conda
    • Python packages using pip
    • R packages
  • understand what containers are

  • learners have explored the UPPMAX documentation

  • can install software using a container

  • (optional) can build software from source
Notes for teachers

Teaching goals:

  • See above

Lesson plan:

  • Intro 20
    • module system 5
    • content 5
    • packages and libraries 10
  • Exercise 20 min
  • Install software yourself 15
    • Containers
    • Build from source
  • Exercise 10 min
  • Summary 5 min

Total time: 70 m

The module system

Forgot how to use the module system?

See the basic Bianca course page 'Using the module system'.

  • Lots of programs and tools installed as modules on Bianca.
    • You can request installations but that may take several days or weeks to be handled by the application experts at UPPMAX.
    • Workflow: Application expert installs on Rackham and it is synced over to Bianca within a day.
  • Install yourself.
    • Workflow: use the wharf to transfer installation files and packages to Bianca from other place.

Warning

Procedures may change for Maja system, but probably to the simpler way.

Content

  • Principles of installing packages on Bianca
  • Exercise: test yourself in EITHER
    • R
    • Conda
    • Python/pip
  • Feedback
  • Principles of software installation on Bianca
    • From binary
    • Containers
    • (From source)
  • Development and Git on Bianca

Packages and libraries to scripting programs

  • Python, R (and Julia) all have some centrally installed packages that are available from the modules.
    • R has a special module called R_packages,
    • Python has many packages already in the Python module
      • Machine Learning python packages are included in the python_ml_packages module.
  • If not found there you can try to install those by yourself.

Check packages

R

$ ml R_packages/4.3.1

Then within R, try loading the package you want, like glmnet:

library(glmnet)

Python

  • Check python versions: ml avail python

  • Check python packages/modules

    1. help output from: ml help python/3.12.7

    2. In a loaded python

      • Load a python version, like: ml python/3.11.8
      • from Python shell with the import command
      • from BASH shell with the pip list command
Julia
  • At UPPMAX there is a central library with installed packages.
  • This is good, especially when working on Bianca, since you then do not need to install via the wharf.

  • It is often better to install your own, see below, or ask the support to install centrally.

  • Check julia versions: ml avail julia

  • Check julia packages/modules:

    1. help output from: ml help julia/1.9.3

    2. In a loaded julia

      • Load a python version, like: ml julia/1.8.5
      • from julia shell with the using command

Install packages, principles

Installation principle

  • Get the package without installing it to a computer with internet.
    • transit works well because you can reach R and Python modules from there
  • Sync to wharf.
  • Move the files on Bianca to a place in the path used for packages of R, Python (pip) or Julia.
  • Install there.
R
  • Typical place to put R packages: ~/R
  • Otherwise you may have to update your R_LIBS_USER="<path>"

Links:

pip (PyPI)
  • Typical place to put python packages: ~/.local/lib/python<version>/site-packages/
  • Otherwise you may have to update PYTHONPATH="<path>"

Links:

Tip Python packages

  • Try Conda first directly on Bianca. We have mirrored all major Conda repositories directly on Bianca. These are updated every third day.
  • If you want to keep number of files down, use PyPI (pip), but then you need to use the wharf.
Conda
  • We have mirrored the non-proprietary Conda repositories (not main, anaconda and r) directly on Bianca. These are updated every third day.
  • Good to change CONDA_ENVS_PATH to project folder, because of many small files.
  • Example: CONDA_ENVS_PATH=/proj/sens2025560/bjornc/conda

Links:

Julia

Exercise ~20 min

Pick one of the following topics!

Discussion

  • Did it work out well?
  • Any questions?
  • Any input?

Install software yourself

  • If not available on Bianca already (like Conda repositories) --> use the wharf to install your tools.

Typical workflow for installation

  • Download the

    • source code or
    What is source code?
    • These are files written in a programming language that needs to be compiled before run.
    • Typically they come as compressed archives with the file endingsd .tar.gz
    • binary (Linux on x86 and 64-bit)
    What is a binary?
    • This is a ready-compiled program.
    • You need to choose one that states it is linux and x86_64 system.

  • Transfer to the wharf

  • Move file(s) to either
    • $HOME directory.
      • Handy for personal needs and low numbers of files — i.e. not Conda.
    • Usually better to install in project directory.
      • This way the project contains both data and software
      • Good for reproducibility, collaboration, and everyone's general sanity.
  • Then, either:
    • Binaries for Linux on x86 and 64-bit should be able to be run directly as they are.
    • Install program from source code, following instructions from documentation of the software!

Containers

  • Containers let you install programs without needing to think about the computer environment, like
    • operative system
    • dependencies (libraries and other programs) with correct versions

Containerization

Info

  • Everything is included

  • Workflow:

    • Download on a computer with internet
    • Transfer to Bianca
    • Move to from wharf to any place in your working folders on Bianca

  • Draw-backs

    • you will install also things that may be already installed
    • therefore, probably more disk space is needed

More info

Build from source (C/C++ and Fortran)

Attention

We are not covering this more than superficially.

Still interested?
  • Get the source files to Bianca via the Wharf
  • To build from source use a compiler module
  • We have several compiler versions from GNU and INTEL
  • make is installed on the system
    • ⚠ It could happen that the "Makefile" contains web fetching, which will not work from Bianca.
    • Usually it is not a problem to build on Rackham and move to Bianca.

  • cmake is available as module

  • Extra material: Build from source

Exercise ~10 min

Learn and try out containers
  • Read the introduction with a demo and use it to solve the exercise in the end.

  • "Containers" contains less material but may take time to install.

  • Containers

  • Help each-other!

Discussion

  • Did it work out well?
  • Any questions?
  • Any input?

Own development and Git

Keypoints

  • You have got an overview of the procedures to install packages/libraries and tools on Bianca through the wharf
  • If you feel uncomfortable or think that many users would benefit from the software, ask the support to install it.