Welcome to HPC python course material

This material

Here you will find the content of the workshop Using Python in an HPC environment.

• Documentation at the HPC centres UPPMAx and HPC2N

  • UPPMAX: https://www.uppmax.uu.se/support/user-guides/python-user-guide/

  • HPC2N: https://www.hpc2n.umu.se/resources/software/user_installed/python

Content

• This course aims to give a brief, but comprehensive introduction to using Python in an HPC environment.

• You will learn how to

  • use modules to load Python

  • find site installed Python packages

  • install packages yourself

  • use virtual environments,

  • write a batch script for running Python

  • use Python in parallel

  • how to use Python for ML and on GPUs.

• This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

Cluster-specific approaches

• The course is a cooperation between UPPMAX (Rackham, Snowy, Bianca) and HPC2N (Kebnekaise) and will focus on the compute systems at both centres.

• For the site-specific part of the course you will be divided into groups depending on which center you will be running your code, as the approach is somewhat different.

Preliminary schedule

Preliminary schedule

Time	Topic	Activity
9:00	Syllabus
	Introduction	Lecture
	Loading modules and running Python codes	Lecture+code along
	Dealing with packages	Lecture+code along
	Coffee
	Isolated environments (general)	Lecture+code along
	SLURM Batch scripts for Python jobs	Lecture+code along + exercise
	Interactive	Lecture+code along
	Short leg stretch
	Separated sessions 1 for UPPMAX/HPC2N:isolated environments	Lecture
12:00	LUNCH
13:00	Parallelising a simple Python code	Lecture+code along + exercise
	Deploying GPU:s for Python	Lecture+code along + exercise
	Short leg stretch
	Using Python for Machine Learning jobs	Lecture+code along
	Separated sessions 2 for UPPMAX/HPC2N: Bianca/ML exercises	Lecture
	Coffee
	Summary
	Q&A on-demand
	Additional exercises
16.00	END

Lessons:

• Introduction
  • Some practicals
  • What is python?
  • More python?
  • The two HPC centers UPPMAX and HPC2N
  • Briefly about the cluster hardware and system at UPPMAX and HPC2N
• Load and run python
  • Check for Python versions
  • Load a Python module
  • Run
• Packages
  • Check current available packages
  • Install with pip
• Isolated environments
  • General procedures
  • Virtual environment - venv (UPPMAX)
  • Virtual environment - virtualenv (HPC2N)
• Running Python in batch mode
  • Useful commands to the batch system
  • Example Python batch scripts
  • Exercises
• Interactive work on the compute nodes
  • General
  • Python “interactively” on the compute nodes
• Parallel computing with Python
  • Threads
  • Distributed
  • MPI
  • Monitoring resources’ usage
  • Additional information
• Using GPUs with Python
  • Numba example
  • Numba example 2
  • Additional information
• Using Python for Machine Learning jobs
  • PyTorch
  • TensorFlow
  • General
• Summary

UPPMAX Sessions:

• On UPPMAX clusters
  • UPPMAX clusters
  • UPPMAX packages
• Isolated environments at UPPMAX
  • Virtual environment - venv
  • Prepare the course environment
  • Working with virtual environments defined from files
  • pyenv
• Conda
  • Install with conda (UPPMAX)
  • First steps
  • Using Conda
  • Working with Conda environments defined by files
• Running Jupyter on compute nodes
  • On the compute nodes
  • 1. Start an interactive session from the login node
  • 2. Start jupyter notebook from the interactive session (when it gets allocated)
  • 3. Connect to the running notebook
• On Bianca cluster

HPC2N Sessions:

• On Kebnekaise cluster
  • Accessing Kebnekaise
  • Loading Python
  • Starting Python and running Python scripts
• Isolated environments at HPC2N
  • Virtual environment - virtualenv
• Using Python for Machine Learning jobs
  • PyTorch
  • TensorFlow
  • General
  • Horovod

Exercises:

• Exercises and demos
  • Examples
  • TensorFlow
  • Numba example 2
  • Exercises