Introduction to running R, Python, and Julia in HPC

Content

• This course aims to give a brief, but comprehensive introduction to using Python, Julia and R in an HPC environment.

• You will learn how to

  • start the interpreters of Python, Julia and R by the HPC module system

  • find site installed packages/libraries

  • install packages/libraries yourself

  • use virtual environments

  • use the computation nodes

    • write batch scripts

    • work interactivly

• This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

• We aim to give this course in spring and fall every year.

Cluster-specific approaches

• The course is a cooperation between UPPMAX (Rackham, Snowy, Bianca) and HPC2N (Kebnekaise) and will focus on the compute systems at both centres.

• Although there are differences we will only have few seperate sessions.

• Most participants will use UPPMAX’s systems for the course, as Kebnekaise is only for local (UmU, IRF, MIUN, SLU, LTU) users.

• The general information given in the course will be true for all/most HPC centres in Sweden.

  • The examples will often have specific information, like module names and versions, which may vary. What you learn here should help you to make any changes needed for the other centres.

  • When present, links to the Python/Julia/R documentation at other NAISS centres are given in the corresponding session.

Schedule Spring 2024

Day	Language
Tuesday 12 March	Python
Wednesday 13 March	Julia
Thursday 14 March	R

Some practicals

Code of Conduct

We strive to follow the Contributor Covenant Code of Conduct to foster an inclusive and welcoming environment for everyone.

In short:

• Use welcoming and inclusive language

• Be respectful of different viewpoints and experiences

• Gracefully accept constructive criticism

• Focus on what is best for the community

• Show courtesy and respect towards other community members

Furthermore, as this is a hands-on, interactive workshop:

• Be kind to each other and help each other as best you can.

• If you can’t help someone or there is some problem, let someone know.

Finally, if you notice something that prevents you from learning as well as you can, let us know and don’t suffer silently, even the “little things”:

• Volume too low?

• Font size too small?

• Generally confusing instructor?

• Not enough breaks?

Zoom

• The course is run over Zoom. You should have gotten an email with the links

• There will be a zoom for the lectures

• When you join the Zoom meeting, use your REAL NAME.

• The lectures and demos will be recorded, but NOT the exercises.

  • If you ask questions during the lectures, you may thus be recorded.

  • If you do not wish to be recorded, then please keep your microphone muted and your camera off during lectures and write your questions in the Q/A document (see below about Microsoft-365 collaboration document).

• Please MUTE your microphone when you are not speaking and use the “Raise hand” functionality under the “Participants” window during the lecture. Please do not clutter the Zoom chat. Behave politely!

• There will be breakout rooms for the exercises.

  • You may enter there and you will get personal help

  • There will be a “silent” room for those who just want to sit and work

Collaboration document (Microsoft-365)

• Q/A collaboration document

• Use this page for the workshop with your questions.

• It becomes part of the course and some Q/A:s may be more relevant to you at a later occasion!!

• Depending on how many helpers there are we’ll see how fast there are answers.

  • Some answers may come after the workshop day.

Hint

• Main project on UPPMAX:

  • Project ID: naiss2024-22-107

  • Directory name on rackham: /proj/r-py-jl

  • Please create a suitably named subdirectory below /proj/r-py-jl, for your own exercises.

• Local project on HPC2N:

  • Project ID: hpc2n2024-025

  • Directory name on Kebnekaise: /proj/nobackup/hpc2n2024-025

  • Please create a suitably named subdirectory below /proj/nobackup/hpc2n2024-025, for your own exercises.

Example of screen arrangement

If you only have one screen to work with for the course, try to fit:

╔════════════╗ ╔══════════════╗
║            ║ ║  TERMINAL    ║
║            ║ ║   WINDOW     ║
║   ZOOM     ║ ╚══════════════╝
║  WINDOW    ║ ╔══════════════╗
║            ║ ║   BROWSER    ║
║            ║ ║  W/MATERIAL  ║
╚════════════╝ ╚══════════════╝

Warning

• Please be sure that you have gone through the `pre-requirements <https://uppmax.github.io/R-python-julia-HPC/prereqs.html>`_

• It mentions the familiarity with the LINUX command line.

• The applications to connect to the clusters

  • terminals

  • remote graphical desktop ThinLinc

Material for improving your programming skills

• This course does not aim to improve your coding skills.

• Rather you will learn to understand the ecosystems and navigations for the the different languages on a HPC cluster.

See the pre-requirements, section Coding

Briefly about the cluster hardware and system at UPPMAX and HPC2N

What is a cluster?

• Login nodes and calculations/computation nodes

• A network of computers, each computer working as a node.

• Each node contains several processor cores and RAM and a local disk called scratch.

• The user logs in to login nodes via Internet through ssh or Thinlinc.

  • Here the file management and lighter data analysis can be performed.

• The calculation nodes have to be used for intense computing.

The two HPC centers UPPMAX and HPC2N

Two HPC centers

• There are many similarities:

  • Login vs. calculation/compute nodes

  • Environmental module system with software hidden until loaded with module load

  • Slurm batch job and scheduling system

  • pip install procedure

• … and small differences:

  • commands to load Python, Python packages, R, Julia

  • slightly different flags to Slurm

• … and some bigger differences:

  • UPPMAX has three different clusters

    • Rackham for general purpose computing on CPUs only

    • Snowy available for local projects and suits long jobs (< 1 month) and has GPUs

    • Bianca for sensitive data and has GPUs

• HPC2N has Kebnekaise with GPUs

• Conda is recommended only for UPPMAX users

Warning

• At both HPC2N and UPPMAX we call the applications available via the module system modules.

• https://www.uppmax.uu.se/resources/software/module-system/

• https://www.hpc2n.umu.se/documentation/environment/lmod

To distinguish these modules from the python modules that work as libraries we refer to the later ones as packages.

See also

• Details about the systems - https://uppmax.github.io/R-python-julia-HPC/prereqs.html#common-features - and below

Prepare your environment now!

Type-Along

• Please log in to Rackham, Kebnekaise or other cluster that you are using.

Use Thinlinc or terminal?

• It is up to you!

• Graphics come easier with Thinlinc

• For this course, when having many windows open, it may be better to run in terminal, for screen space issues.

UPPMAXHPC2N

1. Log in to Rackham!

• Terminal: ssh -X <user>@rackham.uppmax.uu.se

• ThinLinc app: <user>@rackham-gui.uppmax.uu.se

• ThinLinc in web browser: https://rackham-gui.uppmax.uu.se

2. If not already: create a working directory where you can code along.

• We recommend creating it under the course project storage directory

3. Example. If your username is “mrspock” and you are at UPPMAX, then we recommend you create this folder:

   $ mkdir /proj/r-py-jl/mrspock/
   $ mkdir /proj/naiss2024-22-107/mrspock/
   

Content of the course

Pre-requirements:

• Pre-requirements
  • Technical on local computer
  • Text editors on the Clusters
  • Coding
  • Understanding clusters
  • The two HPC centers UPPMAX and HPC2N

Python Lessons:

• Introduction Python
  • Preliminary schedule
  • What is python?
  • Other NAISS centres
• Load and run python
  • Check for Python versions
  • Load a Python module
  • Run
• Packages
  • Introduction
  • Finding packages
  • Python package installers
  • Exercises
  • Exercise 1
  • Exercise 2
  • Exercise 3
  • Conclusion
• Isolated environments
  • Introduction
  • Virtual environment managers
  • General workflow
  • Exercises
  • Exercise 0: remove the Python packages installed in the home folder
  • Exercise 1: work with vpyenv
  • Exercise 2: work with Example-gpu
  • Exercise 3: export and import a virtual environment
  • Conclusion
  • Links
• Running Python in batch mode
  • Useful commands to the batch system
  • Example Python batch scripts
  • Exercises
• Using GPUs with Python
  • GPUs on UPPMAX and HPC2N systems
  • Numba example
  • Exercises
  • Additional information
• Interactive work on the compute nodes
  • Introduction
  • The different way HPC2N and UPPMAX provide for an interactive session
  • Start an interactive session
  • Check to be in an interactive session
  • Check to have booked the expected amount of cores
  • Running a Python script in an interactive session
  • End the interactive session
  • Exercises
  • Exercise 0: be able to use the Python scripts
  • Exercise 1
  • Exercise 2
  • Conclusion
  • Links
• Jupyter on compute nodes
  • Introduction
  • UPPMAX procedure
  • Kebnekaise
  • Links
• Parallel and multithreaded functions
  • What is parallel programming?
  • Why is parallel programming needed?
  • Common parallel programming paradigms
  • Exercises
• Conda at UPPMAX
  • Using Conda
  • Install with conda (UPPMAX)
  • First steps
  • Working with Conda environments defined by files
  • Exercises
  • Links
• Exercises
  • Isolated
  • Conda (UPPMAX)
  • Batch
• Summary

Julia Lessons:

• Introduction Julia
  • Preliminary schedule
  • What is Julia?
  • Workflow in Julia (DEMO)
  • Features of Julia
  • Shortcomings of Julia
  • More on Julia?
  • Other NAISS centres
• Load and run Julia
  • Check for Julia versions
  • Load a Julia module
  • Run
  • Exercises
• Packages and isolated environments
  • Create a project environment
  • Create a package environment
  • Customizing the set of visible environments
  • Environment stacks
  • Exercises
• Running Julia in batch mode
  • Useful commands to the batch system
  • Examples of batch scripts for Julia
  • Exercises
• Parallel and multithreaded functions
  • What is parallel programming?
  • Why is parallel programming needed?
  • Common parallel programming paradigms
  • Exercises
• Interactive work on compute nodes
  • General
  • Julia “interactively” on the compute nodes
  • Running IJulia and Jupyter notebooks
• Exercises
  • Introduction
  • Load and run
  • Packages and isolated environments
  • Batch mode
• Summary

R Lessons:

• Introduction R
  • First overview of R
  • Preliminary schedule
• Load and run R
  • Introduction
  • 1. Find an R module
  • 2. Load an R module
  • 3. Use the R interpreter
  • 4. Run an R script
  • Exercises
  • Exercise 1: find an R module
  • Exercise 2: load an R module
  • Exercise 3: use the R interpreter
  • Exercise 4: run an R script
  • Conclusions
• Packages
  • R packages: A short Primer
  • Preinstalled package libraries
  • Installing your own packages
  • Install own packages on Bianca
  • Exercises
• Isolated environments with renv
  • General procedures
  • Workflow
  • Conda (UPPMAX)
• Running R in batch mode
  • Overview of the UPPMAX systems
  • Overview of the HPC2N system
• Parallel and multithreaded functions
  • What is parallel programming?
  • Why is parallel programming needed?
  • Common parallel programming paradigms
  • Exercises
• ML with R
  • Running your code
  • Exercises
• Interactive work on the compute nodes
  • Overview of the UPPMAX systems
  • Overview of the HPC2N system
  • Example Type along
  • Exit
• Using RStudio
  • On UPPMAX
  • On HPC2N
• Exercises
  • Load and run
  • Packages
  • Batch mode
• Summary

COMMON:

• Parallel and multithreaded functions
  • What is parallel programming?
  • Why is parallel programming needed?
  • Common parallel programming paradigms
  • Exercises